[{"content":" Introduction # So about two months back, I moved to another city for work. My home PC — a fairly bulky Fedora machine that I\u0026rsquo;ve been calling titan — wasn\u0026rsquo;t something I was going to drag along in a bag. I left it behind at my hometown, plugged in and running on my home network. Before I left, I already had Tailscale configured on it, so at least I knew I could poke at it remotely if I needed to.\nFast forward to recently. I\u0026rsquo;ve been wanting to run LLMs locally without burning through money on cloud GPU services. Downloading llama3.1 and pointing Ollama at it on titan made sense — that machine has decent specs for it. But there\u0026rsquo;s a small catch. I\u0026rsquo;m not sitting in front of titan. I\u0026rsquo;m in another city with just my Ubuntu laptop.\nSo how do you talk to Ollama on a machine that\u0026rsquo;s physically in another place? Tailscale. It builds a private, encrypted tunnel between your machines. No port forwarding. No opening firewall ports to the public internet. No messing with your router. It just works, and I already had it set up.\nThis post is essentially me documenting what I did. If your situation is similar — a beefy home machine and a lighter laptop elsewhere — this might work for you too.\nWhat you need before starting # I\u0026rsquo;ll be honest, this won\u0026rsquo;t be a from-scratch tutorial. You\u0026rsquo;ll need a few things already in place:\nTailscale installed and running on both machines — the home PC and whatever you\u0026rsquo;re using remotely Both machines on the same tailnet — meaning same Tailscale account, so they can see each other Ollama already installed on the home machine — with at least one model pulled. I\u0026rsquo;m using llama3.1 If you haven\u0026rsquo;t set up Tailscale yet, I wrote about it in my homelab post. Start there.\nStep 1: Get your PC\u0026rsquo;s Tailscale IP # First thing is to find out what IP Tailscale assigned to your home machine. On the Fedora machine, run:\n1 tailscale ip -4 You\u0026rsquo;ll get back something in the 100.x.x.x range. That\u0026rsquo;s your VPN IP and it doesn\u0026rsquo;t change even if your home network IP does. Write it down somewhere — you\u0026rsquo;ll need it in every step after this.\nStep 2: Tell Ollama to listen on the Tailscale IP # Here\u0026rsquo;s something I didn\u0026rsquo;t know at first: by default, Ollama only listens on 127.0.0.1:11434. That means it only takes connections from localhost. Makes sense as a default, but not useful for our case.\nWe need to tell Ollama to also listen on the Tailscale IP. Since Ollama runs as a systemd service on Fedora, we\u0026rsquo;re going to use systemctl edit to drop in an override:\n1 sudo systemctl edit ollama This opens an editor (probably nano or whatever your default is). Add these lines:\n1 2 [Service] Environment=\u0026#34;OLLAMA_HOST=100.x.x.x:11434\u0026#34; Swap 100.x.x.x for the actual IP you found in Step 1.\nAfter saving, reload the daemon and restart Ollama:\n1 sudo systemctl daemon-reload \u0026amp;\u0026amp; sudo systemctl restart ollama Now let\u0026rsquo;s confirm it actually worked:\n1 ss -tlnp | grep 11434 If you see output with your Tailscale IP on port 11434, you\u0026rsquo;re good. If it\u0026rsquo;s still showing 127.0.0.1, something didn\u0026rsquo;t take — double check the override file.\nStep 3: Poke a hole in the firewall (just for Tailscale) # Fedora uses firewalld by default, not ufw like Ubuntu. I keep forgetting this every time I switch machines, so if you\u0026rsquo;re also coming from Ubuntu, heads up.\nWe need to allow port 11434, but specifically only on the Tailscale interface. No point exposing this to the rest of the internet.\n1 2 3 sudo firewall-cmd --zone=trusted --add-interface=tailscale0 --permanent sudo firewall-cmd --zone=trusted --add-port=11434/tcp --permanent sudo firewall-cmd --reload What this does is put the Tailscale interface in the trusted zone and allow port 11434 on it. Your port never shows up on your regular network interface — only reachable over the Tailscale tunnel. I like this approach because it\u0026rsquo;s quite surgical. You\u0026rsquo;re not blowing open a port for everyone, just for the VPN.\nStep 4: Connect from the other machine # Now on your laptop — in my case my Ubuntu machine sitting in another city — set the OLLAMA_HOST environment variable:\n1 export OLLAMA_HOST=http://100.x.x.x:11434 Again, replace with your actual IP. Now try listing the models:\n1 ollama list If that returns something, the connection is working. If it hangs or times out, go back and check the firewall step. In my case the first time I ran this, it just hung — I\u0026rsquo;d forgotten to reload firewalld.\nYou can also run a model directly:\n1 ollama run llama3.1 And if you want a quick sanity check without opening a full REPL, curl works:\n1 curl http://100.x.x.x:11434/api/tags You should get back a JSON blob listing your models. If that works, everything is wired up correctly.\nStep 5: Using the Python client # Now that Ollama is accessible over the network, you can also talk to it programmatically. I wanted to hook it into some scripts I was working on, so I tried the Python client:\n1 2 3 4 5 6 7 8 import ollama client = ollama.Client(host=\u0026#34;http://100.x.x.x:11434\u0026#34;) response = client.chat( model=\u0026#34;llama3.1\u0026#34;, messages=[{\u0026#34;role\u0026#34;: \u0026#34;user\u0026#34;, \u0026#34;content\u0026#34;: \u0026#34;Hello from another city!\u0026#34;}], ) print(response[\u0026#34;message\u0026#34;][\u0026#34;content\u0026#34;]) Swap out the IP and you\u0026rsquo;re talking to the model on your home machine from wherever you are. No API keys, no usage limits, no paying per token. I found this really useful for automating some repetitive tasks without touching any cloud services.\nThe Python ollama library is quite minimal so it\u0026rsquo;s easy to build on top of. You can also use the raw HTTP API if you prefer — it\u0026rsquo;s the same endpoint curl was hitting above.\nA few things I ran into # While this mostly worked, a couple of things caught me off guard:\nThe OLLAMA_HOST env variable needs to be set in every shell session, or you add it to your .bashrc/.zshrc. Easy to forget this when you open a new terminal and suddenly Ollama isn\u0026rsquo;t connecting. If your home machine reboots, the Tailscale IP stays the same, which is nice. But the systemctl edit override should survive reboots too since it\u0026rsquo;s persisted. I had this working even after a power cut. Large models are slow to respond over a VPN tunnel depending on your internet speeds at home. llama3.1 at 8B parameters is manageable. Bigger ones might test your patience. I haven\u0026rsquo;t tried anything larger than that over this setup yet. Conclusion # That\u0026rsquo;s pretty much it. You\u0026rsquo;ve got your home machine running Ollama, listening on its Tailscale IP, with a firewall that only lets through traffic from the VPN. And from wherever you are, as long as Tailscale is connected, you can reach it.\nWhat I like about this setup is that there\u0026rsquo;s nothing exotic going on. Tailscale handles the hard parts — the key exchange, encryption, NAT traversal — and we\u0026rsquo;re just telling Ollama where to listen. If Tailscale can already reach your machine, adding Ollama access on top of that is only a few commands.\nIf you have something similar running or tried a different approach, I\u0026rsquo;d be curious to hear about it in the comments. And if you want to see more of this kind of homelab stuff, you can subscribe to my YouTube channel — I\u0026rsquo;ll probably make a video on this at some point too.\n","date":"24 May 2026","externalUrl":null,"permalink":"/posts/2026/accessing-ollama-remotely-with-tailscale/","section":"Posts","summary":"Introduction # So about two months back, I moved to another city for work. My home PC — a fairly bulky Fedora machine that I’ve been calling titan — wasn’t something I was going to drag along in a bag. I left it behind at my hometown, plugged in and running on my home network. Before I left, I already had Tailscale configured on it, so at least I knew I could poke at it remotely if I needed to.\n","title":"Accessing Ollama Remotely with Tailscale","type":"posts"},{"content":"","date":"24 May 2026","externalUrl":null,"permalink":"/tags/fedora/","section":"Tags","summary":"","title":"Fedora","type":"tags"},{"content":"","date":"24 May 2026","externalUrl":null,"permalink":"/","section":"Fullstack with Santosh","summary":"","title":"Fullstack with Santosh","type":"page"},{"content":"","date":"24 May 2026","externalUrl":null,"permalink":"/tags/homelab/","section":"Tags","summary":"","title":"Homelab","type":"tags"},{"content":"","date":"24 May 2026","externalUrl":null,"permalink":"/tags/linux/","section":"Tags","summary":"","title":"Linux","type":"tags"},{"content":"","date":"24 May 2026","externalUrl":null,"permalink":"/tags/ollama/","section":"Tags","summary":"","title":"Ollama","type":"tags"},{"content":"","date":"24 May 2026","externalUrl":null,"permalink":"/posts/","section":"Posts","summary":"","title":"Posts","type":"posts"},{"content":"","date":"24 May 2026","externalUrl":null,"permalink":"/tags/self-hosted/","section":"Tags","summary":"","title":"Self-Hosted","type":"tags"},{"content":"","date":"24 May 2026","externalUrl":null,"permalink":"/tags/","section":"Tags","summary":"","title":"Tags","type":"tags"},{"content":"","date":"24 May 2026","externalUrl":null,"permalink":"/tags/tailscale/","section":"Tags","summary":"","title":"Tailscale","type":"tags"},{"content":" Introduction # It\u0026rsquo;s been quite a while since I started homelabbing explicitly. I have already posted about Monitoring My HomeLab With Prometheus and Grafana, but I haven\u0026rsquo;t posted any video on my channel yet.\nSo today I want to take some time to talk about what I\u0026rsquo;m already up to. And for this, I\u0026rsquo;m going to divided this post into the following sections:\nHow I got started and the machines I had What services have I tried out and what I have running Future plans How it all started # This started actually at work when I was talking to my senior about chroot. The topic extended and we were talking about raspberry now. After the conversation, I was provoked to take back out my raspberry which was kept in my closet for the last couple of months.\nThe raspberry was kept in there in the first place because it was not really getting used after I did some initial experimentation with it. I played with some sensors and basic stuff before keeping it in there. I bought it for IoT and even though it takes really low energy, I thought keeping it shutdown was a better option because I was not really using it.\nBut this time, along the same timeline, I was drawn towards r/homelab and r/selfhosted on Reddit. It was a concept that got me excited. That was the thing I was looking for. An enterprise-like test environment where there will be multiple servers with services running on them. And I don\u0026rsquo;t have to pay for it because I\u0026rsquo;m not running it on the cloud, which I used to do previously and was bound to experiment in the limits of my finances.\nAnother factor is my data won\u0026rsquo;t live on any other entity\u0026rsquo;s machine, and will not be prone to auditing, theft et cetra.\nWith raspi already back in service, I was also building a PC. With the new PC as my primary workstation. I can make my existing laptop a node of my cluster. So that I will have 2 devices that can run 24×7.\nSo in total, I have 3 machines, two for servers:\nRaspberry Pi 4B HP Gaming Laptop (1060 GPU) Custom PC (intermediate level) Setting up the cluster # At first, I didn\u0026rsquo;t have any concept of cluster. It was just my raspberry pi. The first service I ever hosted at the start of my homelab journey was Planka. Sadly Planka has been replaced by Vikunja in my current setup. Both are alternatives to Trello, and I use them for the Kanban board. It\u0026rsquo;s a good way to plan pending work I have, and when I plan to do them.\nVikunja use Postgres for data layer. I made one decision at the start of my journey to keep one central database instance across my services. It helps make backups simpler. I might be wrong, but that\u0026rsquo;s what I did. What is your opinion on this one?\nI was using Docker Compose for both of the services I mentioned above. But when I learned about this new thing, it changed the way I was thinking. This thing was Docker Swarm. If you haven\u0026rsquo;t heard about Docker Swarm, but still know about Kubernetes, they both are somewhat similar in what they do. Kubernetes provides more flexibility and thus is best suited for enterprise environments. But Docker Swarm is very simple to set up. And in my opinion enough for a homelab environment. I know home labs are all about learning, and I would definitely set up a Kubernetes cluster in the future. But at the time of writing this, I\u0026rsquo;m happy with Swarm.\nBut how did Docker Swarm change the way I was thinking?\nIt gives the ability to add more node to the cluster. So if I add my laptop to the cluster, I can run two instances of services, one on raspi and the other on the laptop to achieve resiliency. Good learning ground for load balancing and reverse proxy. No matter which node the service is running on, every node exposes the port if the ports are exposed from the container and mapped host. This made the reverse proxy easier. I could now only point to the master node no matter which host is running the service. I\u0026rsquo;ll talk about reverse proxy in later stages. I can pin services to a specific node. Use case? I like my Postgres to stay on one node, all data stays on one node of the cluster. Of course, I\u0026rsquo;m going to do something for distributed storage in the cluster. But in a later post. Docker Swarm can be coupled with a UI such as Portainer. Portainer is like a missing component for Docker. It basically shows every aspect of Docker in a UI. You can see containers, volumes, secrets, and images all in one place. Overall it provides better visibility of the Docker system. There are other projects which facilitate the same; example would be Swarmpit and Docker Desktop. Though I have not used any of them very extensively. One best thing about Docker Swarm for someone who is starting with a HomeLab journey is the normal docker-compose.yaml file just works out of the box. To add more control, you can add swarm specific properties to the compose files. I will talk about it in the next sections.\nHow to set up the Docker Swarm Cluster? # Well, once again it will be best to consult documentation for Swarm. But the installation is very basic. There will be 2 kinds of nodes. manager and worker. The manager also works as a worker, meaning that it will schedule deployment on itself when scheduling. Going ahead, make sure that all the hosts you are planning to run inside the swarm cluster have docker already installed. For this I would highly recommend official docker installation, and not from the package managers. E.g. docker.io on Ubuntu is way behind in terms of version.\nFinally, on the manager node, you\u0026rsquo;re going to run this command:\ndocker swarm init --advertise-addr \u0026lt;MANAGER-IP\u0026gt; You will have to use the actual IP of the manager node right there. If you have the ip command available on your system, you can use the following oneliner to get your internal IP:\nip route get 1 | awk '{print $(NF-2);exit}' After you have run that, you\u0026rsquo;ll be able to see output from the command docker node ls (this command only runs on master node). Output could be something like this:\n1 2 3 $ docker node ls ID HOSTNAME STATUS AVAILABILITY MANAGER STATUS ENGINE VERSION lkm62ip2tdiz7dpx8s3qoqfst * pioneer Ready Active Leader 27.0.3 On this manager node, you\u0026rsquo;re going to input a command that will give a command to run on worker nodes. The command is:\ndocker swarm join-token worker Input the command generated by above command in the worker node and you are done. Unless you have a firewall enabled on either of the machines. You need to find out what firewall you are using and allow the ports used by Swarm. Once again, you can find them in the Docker Swarm setup documentation.\nYou can make as many node node as manager/leader. In fact, you can add only manager nodes, or promote existing one to manager. Community suggests having even number of master nodes to give better high availability and consensus for scheduling.\nServices outside Swarm # We have set up our orchestration with docker, now let\u0026rsquo;s see what I have in my arsenal.\nMost of my services are inside the docker swarm. But there are a few things I wanted to keep out of it. It\u0026rsquo;s not that they can\u0026rsquo;t be dockerized. It was just my initial setup and I never bothered modifying it.\nOne of the things that motivates me to keep out these services of docker subsystem is\u0026hellip; well, it depends on docker. These things are kinda more primitive and other hosts are dependent on it.\nMaybe someday something will motivate me to dockerize it. But for now, it\u0026rsquo;s the way it is.\nPi-hole - ad blocking and custom domain # Pi-hole is one of my hero services I consider in my lab. I learned about Pi-hole as a network-wide ad blocker. Before this, I used OpenVPN-based ad blocking. But setting up an EC2 just for the sake of OpenVPN is not a very smart choice for me right now. And that solution was more of a VPN service than an adblocker. What I was looking for was a solution that is enabled across my home network. And here it was, Pi-hole.\nAs the name hinted, I initially thought Pi-hole was a Raspberry Pi thing. However, after installation, I learned that it can be installed on any machine. Just that it needs to be running 24x7; and for me, my Raspi was the best candidate for this.\nPi-hole uses DNS-based blocking, meaning when a page requests an advertisement site, the DNS lookup for that site is handled by Pi-hole and is blocked when certain sites are requested. For this entire setup to work, you need to set the custom DNS host in the DHCP setting of your router to the host where you installed the Pi-hole. You most probably know your router better than me, so I leave it on to you to set up.\nThere are various types of ad-list provided by various vendors. And I would encourage you to try at least a few of them because not all sites are blocked by all of the adlist. On top of that, you can add or remove custom sites as well. Handy when we want to see if your DNS based blocking is enabled. What I mean is, I have a site which I don\u0026rsquo;t visit much, and I have set it as blocked. When I have to check the adblock status, I just go to that website. If I can access that site, the blocking is not actually work.\nBut adblocking was not the most interesting part in Pi-hole. Pi-hole can provide a few more functionalities, such as custom sites, and DHCP. I never used the latter one, but I have entries mentioned below that map a domain name to certain IPs/hosts in my network.\n1 2 3 4 5 6 7 192.168.10.10 pioneer.santoshk.dev 192.168.10.35 voyager.santoshk.dev 192.168.10.30 titan.santoshk.dev 192.168.10.10 pihole.santoshk.dev 192.168.10.10 vikunja.santoshk.dev 192.168.10.10 portainer.santoshk.dev 192.168.10.10 vaultwarden.santoshk.dev The first 3 entries are my different hosts. pioneer, voyager, and titan are my raspi, laptop, and desktop respectively. The rest of them will make more sense in the next section where we talk about reverse proxy.\nBut for time\u0026rsquo;s sake, know that it has some relation to the host pioneer. This custom domain feature is my favorite in Pi-hole.\nBut creating entires here is just one part. It says that when someone in local network accesses that site, they will be pointed to that give IP. But on the the host site, it is not configured to handle the request. And that is what we are going to see in the next section.\nNginx and certbot - reverse proxy and HTTPS on local domain # You will wonder why I have my nginx outside my swarm if something like Nginx Proxy Manager exists. Although Nginx Proxy Manager is very easy to use and pretty automatic. When I tried Nginx Proxy Manager, I found out that only the services inside docker containers can be used with custom domains. And it is not very useful when you have services such as pihole, and in the next section openmediavault outside the containers. I might be wrong here, and let me know if I am mistken because I didn\u0026rsquo;t go really in-depth.\nWith nginx on the pioneer host itself, I can even reverse proxy the request to other hosts on my network. And that was the reason I chose to have it on host OS itself.\nGoing back to the previous section, if we look at the last 5 entries on the list, you\u0026rsquo;ll see that those domains are tied to one single host. This is because pioneer is my reverse proxy.\nMy /etc/nginx/nginx.conf is pretty stock, and it sources other configuration files in a modular manner. Here is a chunk of code from the http block:\n1 2 include /etc/nginx/conf.d/*.conf; include /etc/nginx/sites-enabled/*; By convention, you have to keep your domain-specific things in the /etc/nginx/sites-available/ directory. Then create a symbolic link to /etc/nginx/sites-enabled/. I have a file called homelab.santoshk.dev.conf inside sites-available and a symbolic link to sites-enabled.\nHere are some selected specific chunks of my homelab site configuration:\nPortainer:\n1 2 3 4 5 6 7 8 9 10 11 server { listen 443 ssl; server_name portainer.santoshk.dev; ssl_certificate /etc/letsencrypt/live/santoshk.dev/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/santoshk.dev/privkey.pem; location / { proxy_pass https://192.168.10.10:9443; } } This is one of the simplest forms of nginx config I have on my reverse proxy instance. And it goes like this:\nYou have a service running at https://192.168.10.10:9443. You want to use a custom subdomain provisioned by Pi-hole. And you want to use HTTPS to access it.\nOn the listen line we say we are using TLS (although conventionally it\u0026rsquo;s written ssl) and would listen on port 443 only (this means you need to access the site with https://). On the server_name line hold the value of the subdomain you want to use. The request coming from different subdomains goes to different server blocks. The next two ssl_certificate* lines tell nginx the keys to use to enable https on the site. We\u0026rsquo;ll see in the next subsection how to get certificates itself, but for now, keep this a placeholder if you don\u0026rsquo;t have anything on that path. The location / { line starts a new block. This line says that when a request arrives at / (basically portainer.santoshk.dev/), do the following. In the proxy_pass line, I pass in the IP:PORT combo where the service is exposed in the network. The best thing about having nginx outside the docker subsystem is that I can proxy traffic to services that are not limited to docker. I can have a VM running on one of my nodes, and I can proxy the traffic to it. My next service is one such service that is out of any docker subsystem.\nPi-hole:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 server { listen 443 ssl; server_name pihole.santoshk.dev; ssl_certificate /etc/letsencrypt/live/santoshk.dev/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/santoshk.dev/privkey.pem; root /var/www/html; autoindex off; index pihole/index.php index.php index.html index.htm; location / { expires max; try_files $uri $uri/ =404; } location ~ \\.php$ { include fastcgi_params; fastcgi_param SCRIPT_FILENAME $document_root/$fastcgi_script_name; fastcgi_pass unix:/run/php/php8.2-fpm.sock; fastcgi_param FQDN true; } location /*.js { index pihole/index.js; } location /admin { root /var/www/html; index index.php index.html index.htm; } location ~ /\\.ht { deny all; } } In this configuration, the first few lines are the same. Then it diverges to some PHP-specific configuration which I\u0026rsquo;m not very familiar with. After that, it has more rules specific to different paths on the host. I don\u0026rsquo;t have in-depth knowledge about the configuration, but I have found them on the Pi-hole documentation site.\nYou might have to install some prerequisites if you are going to have a similar setup. But I will also let you know that the Pi-hole itself can run on a container in host network mode.\nLet\u0026rsquo;s move to the next service I have in the cluster.\nVikunja:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 server { listen 80; server_name vikunja.santoshk.dev; location / { proxy_pass http://192.168.10.10:4444; } } server { listen 443 ssl; server_name vikunja.santoshk.dev; ssl_certificate /etc/letsencrypt/live/santoshk.dev/fullchain.pem; ssl_certificate_key /etc/letsencrypt/live/santoshk.dev/privkey.pem; location / { proxy_pass http://192.168.10.10:4444; } } Configuration for Vikunja is very similar to what we have seen for Portainer. But in this case, I\u0026rsquo;m also exposing port 80. Meaning that I\u0026rsquo;m allowing an insecure connection to the site.\nWell, the insecure one is not needed. But I have kept it here to show you different configs we can have in nginx.\nFinally, although I have more config, you got an idea. You can always refer to the documentation of the service you are installing. They usually have the best config you can have for the functioning of their service at the best. Next we are going to see how can we enable HTTPS on our domains.\nCertbot # I can\u0026rsquo;t end the nginx section without talking about the certbot, as they both work in conjunction. We are using directives such as ssl_certificate and ssl_certificate_key with a path that does not exist. Let\u0026rsquo;s see how are they made.\nAs you can see, I have custom domains inside my local network, e.g. portainer.santoshk.dev. DNS for these domains is being resolved on a local level by Pi-hole. This means you can\u0026rsquo;t access this website outside your network the same way e.g. from your mobile data/network.\nThe thing with certbot (Let\u0026rsquo;s Encrypt) is that you need to prove ownership of whatever domain you want to use. I have santoshk.dev where you are reading this blog post. And I wanted to use the same for my services on the local network.\nThere are multiple ways you can prove the ownership. I prefer the DNS method myself. I have Cloudflare as my DNS provider for my domain. Good thing for me, there is a plugin for certbot for Cloudflare. I simply have to get an API key from Cloudflare and use it to provide API keys to certbot for verification.\nPeople don\u0026rsquo;t always have similar set-up as mine. In that case, the certbot documentation on getting certificates is the savior.\nIf everything is correct, this command will do the work:\nsudo certbot certonly --dns-cloudflare --dns-cloudflare-credentials /path/to/cloudflare.ini -d \u0026quot;*.santoshk.dev\u0026quot; The certificates and the related files are stored in specific locations. The path I have been using inside ssl_certificate etc is automatically generated based on domains I passed to certbot in above command.\nAt this point, certbot will also take care of renewing the certs by registering a cron job. Your domains should have https by now.\nIf you want a more detailed video on this process, please let me know in the comments.\nOpenMediaVault # When you delve into self-hosting, you won\u0026rsquo;t take much time until you need a NAS or a Network Attached Storage. Basically, this program runs on a host and exposes a persistent filesystem over the network.\nThe benefit is, you can use that storage from any host in your network. I have been using my NAS with my phone, laptop, and desktop altogether. This NAS can also be used in conjunction with services running in the cluster.\nBut how to get it set up? I won\u0026rsquo;t take much time on this one, as I have followed a tutorial and would recommend watching that video directly. Here is the link: https://www.youtube.com/watch?v=gyMpI8csWis\nSo in a nutshell, I have a hard drive attached to my Raspberry Pi via USB. I have used OpenMediaVault to expose that drive over the network.\nWith that said, it is the end of this section where we discussed about service running outside docker. These were some basic services that I have kept on the host system.\nServices inside Swarm # Most of my services are being run on a docker subsystem called Swarm. Any docker container that can run with a docker-compose file can also run in a swarm. We get the added benefit of scaling and learning. This knowledge will be transferrable when we are learning Kubernetes.\nWith Docker Swarm setup, which is very easy as we have already seen in previous sections. The first service I installed was Portainer. I already talked about it in the last section. The installation steps are pretty simple. You need to run the following command on the master swarm node:\n1 2 docker volume create portainer_data docker run -d -p 8000:8000 -p 9443:9443 --name portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce:latest This will automatically install a portainer-agent on the master node as well as other nodes you add to the cluster in future.\nRunning this container will expose port 9443, and can be visited from a browser using :9443.\nAs I have already configured nginx for the reverse proxy to portainer, I can access my portainer UI at https://portainer.santoshk.dev.\nAs an anecdote, when I first installed portainer, I was exposed to things in Docker I didn\u0026rsquo;t know existed. Hope you are going to have the same experience and use Docker more efficiently. I think I already mentioned this; there are alternatives to Portainer, such as Swarmit.\nPostgres and pgadmin # I already talked about Postgres. But let\u0026rsquo;s have a look at the compose file to see what additional things we have from a conventional compose file. Please note that this distinction makes a compose file into a stack file.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 version: \u0026#39;3.9\u0026#39; services: master: image: postgres:16 restart: always environment: POSTGRES_DB: example POSTGRES_USER: example POSTGRES_PASSWORD: example ports: - \u0026#34;5432:5432\u0026#34; volumes: - postgres_data:/var/lib/postgresql/data deploy: placement: constraints: - node.hostname == pioneer pgadmin: container_name: pgadmin image: dpage/pgadmin4 depends_on: - postgres ports: - \u0026#34;5050:80\u0026#34; environment: POSTGRES_USER: santosh PGADMIN_DEFAULT_EMAIL: you@domain.com PGADMIN_DEFAULT_PASSWORD: example restart: unless-stopped volumes: postgres_data: What is different?\n1 2 3 4 deploy: placement: constraints: - node.hostname == pioneer This pins the master service with the postgres container on a host called pioneer. If you don\u0026rsquo;t do this, in conjunction with having multiple worker nodes. If postgres somehow crashes, it will start back on any node. Not good, as the container will create another volume on that node. The data is simply lost from the perspective of postgres service. There are more ways to deal with this than pinning. But that\u0026rsquo;s for some other post in the future.\nHosts can crash in multiple ways. For me, it is shutting down of host in most cases.\nI have also used volumes so that even if the postgres crashes, it does not lose any data. Volumes are there even after the container is rebuilt.\nThe second service on this stack is a web UI for postgres. And for this container, I don\u0026rsquo;t care which host it got deployed on. It\u0026rsquo;s not situation-critical and will connect to the same postgres host even if run on another host.\nPrometheus and Grafana - infrastructure monitoring # When you have a lot of machines (for me 3 was enough), you want to know about their health. I mean a htop usually works, but getting to know all of the hosts at the same place is always desirable.\nThere are different use cases of Prometheus and Grafana. You can use them to monitor services running your swarm cluster itself. As well as services in k8s. I also use it for speedtest to keep an I if ISP is up to SLA. But these are out of the scope of this article. What I\u0026rsquo;m using them is to monitor hosts.\nWhat I\u0026rsquo;m doing is known as infrastructure monitoring, and what is described in the previous sentence is application monitoring.\nAnd with that said, let\u0026rsquo;s see how I had it set up.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 version: \u0026#39;3.8\u0026#39; services: prometheus: image: prom/prometheus ports: - \u0026#34;9090:9090\u0026#34; volumes: - /etc/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml - prometheus-data:/prometheus deploy: placement: constraints: - node.hostname == pioneer grafana: image: grafana/grafana container_name: grafana ports: - \u0026#34;3000:3000\u0026#34; volumes: - grafana-data:/var/lib/grafana deploy: placement: constraints: - node.hostname == pioneer volumes: prometheus-data: grafana-data: Some points I want to emphasize:\nI had to pin both services because of the volumes. They both need to interact with the filesystem which is provided through volumes. I need to find a way to store volumes on NAS or something. I\u0026rsquo;ll note this and work on this as I progress. I have decided to manage /etc/prometheus/prometheus.yml from the host machine. This is not necessary, and there are other ways you can set this up. If you are feeling excited, learn about docker configs. It has a great interface on Portainer. Here is a stripped-down version of the config I am using:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 global: scrape_interval: 1m scrape_configs: - job_name: prometheus static_configs: - targets: [\u0026#39;10.100.200.10:9090\u0026#39;] - job_name: titan static_configs: - targets: [\u0026#39;10.100.200.30:9100\u0026#39;] - job_name: voyager static_configs: - targets: [\u0026#39;10.100.200.35:9100\u0026#39;] - job_name: pioneer static_configs: - targets: [\u0026#39;10.100.200.10:9100\u0026#39;] - job_name: speedtest scrape_interval: 60m scrape_timeout: 60s static_configs: - targets: [\u0026#39;10.100.200.35:9090\u0026#39;] You can see 2 top-level blocks here. The first one is self-explanatory.\nThe scrape_configs might require some introduction.\njob_name is how it\u0026rsquo;s going to look in the UI such as Grafana. targets in static_config declares where to reach for data scraping. Although targets is a list, for our use case, there is going to be only one host in it. You may also notice that titan, voyager, and pioneer have something in common; their port. And what is happening is I have installed node-exporter on those nodes. You may want to consult their documentation. They are easily available in the package repository of many distros.\nAlthough I have installed it this way. It initially worked for a few weeks until it started showing partial data.\nAt this moment my setup is the same, but I also have a grafana instance hosted on the cloud. Writing the process down will make this post a lot longer. It involves adding a top-level block called remote_write. I\u0026rsquo;ll leave this task to you. But I hope the issue I am having does not happen to you in the first place.\nVaultwarden - password management # Passwords are the next thing I\u0026rsquo;m self-hosting. I don\u0026rsquo;t want to store my password with some other entity. I would rather store it with me and make accessing it hard.\nI have tried 1Password and BitWarden before. I\u0026rsquo;m not sure if I found a self-hosted solution for 1Password, but I ended up installing vaultwarden.\nVaultwarden is a drop-in replacement for bitwarden server. The latter one is official from the original creators, and the former one is community-developed. Both of them are compatible with the extensions that are installed on the browser. I have been using it with Firefox as well as Brave/Chromium.\nI will present a basic swarm stack definition because I don\u0026rsquo;t want to get into the technical details of security.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 version: \u0026#39;3.8\u0026#39; services: server: image: vaultwarden/server:latest container_name: vaultwarden restart: unless-stopped ports: - \u0026#34;8088:80\u0026#34; volumes: - vw-data:/data deploy: placement: constraints: - node.hostname == pioneer volumes: vw-data: Not so complicated definition file if you compare it to the other two. You should also protect your data at rest in case someone gets unauthorized access to your machine.\nVikunja - task management # I don\u0026rsquo;t need to tell you about this one. Especially if you are a programmer, you know the importance of so-called \u0026ldquo;issues\u0026rdquo; and \u0026ldquo;kanban\u0026rdquo; boards on a project. I use Vikunja for the same interface.\nVikunja was the most satisfying self-hosted project of this kind. Before this, I tried Planka but found Vikunja more feature-rich.\nLet\u0026rsquo;s see the deployment.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 version: \u0026#39;3.9\u0026#39; services: vikunja: image: vikunja/vikunja environment: VIKUNJA_SERVICE_PUBLICURL: https://vikunja.santoshk.dev VIKUNJA_DATABASE_HOST: pioneer.santoshk.dev VIKUNJA_DATABASE_PASSWORD: example VIKUNJA_DATABASE_TYPE: postgres VIKUNJA_DATABASE_USER: example VIKUNJA_DATABASE_DATABASE: vikunja VIKUNJA_SERVICE_JWTSECRET: a-super-secure-random-secret ports: - 4444:3456 volumes: - vikunja_files:/app/vikunja/files restart: unless-stopped volumes: vikunja_files: This deployment is a little bit interesting. Although Vikunja requires Postgres, we don\u0026rsquo;t have any postgres service running as suggested in the official documentation.\nThis is because we are using the postgres installation that I told you I use for anything and everything else on my homelab.\nWhat I have found while working in this manner is database is not created when it is first required. For this, I have to docker exec into the running container, then psql to connect to the server, and then CREATE DATABASE vikunja; manually.\nAlso, note that I have put in dummy credentials and a user. Please use secure credentials.\nAlso, this way of putting credentials in a compose file is risky. You should be using environment variables or secrets. Both are docker-compose things, and you should learn about them.\nI have not delved into that part right now. But when I do, I\u0026rsquo;ll let you know. So please subscribe to Fullstack with Santosh.\nOther services # There are more services that I have tried, but I don\u0026rsquo;t use them regularly. Either it does not solve any problem I have. Is not on this swarm cluster, or interferes with apps that I already have.\nNextCloud - Google Workspace alternative # Initially installed on my PC itself for testing. Then moved to swarm after understanding the configuration.\nmongo - a very popular NoSQL database # Not a lot of services are using it.\nntfy - a producer/subscriber-based notification system # ntfy integrates with many CI-based applications in the community. You can simply publish a message to a certain topic using as primitive as curl. Clients subscribed to that topic will receive those messages.\nfreshrss - an RSS reader # Gave it a try, but not really an RSS reader guy.\nConclusion and Future Plans # This post was mostly aimed toward the services and configuration rather than hardware. That\u0026rsquo;s because I didn\u0026rsquo;t have much hardware when I started writing this post. But as I\u0026rsquo;m progressing towards completion of this post, I have already got some pieces of hardware.\nThis includes:\nDell Optiplex 7040 - Got this mini PC as an alternative to Raspberry Pi. I wanted to add more nodes as a worker to my swarm and Kubernetes cluster. I have already got this and have set up Proxmox for virtualization. It has been going smoothly till now. TP-Link SG1016PE - It is a 16-port switch with 8 port PoE and 50 meters of CAT6 cables. Got this because I wanted to have all devices that are capable of having a wired internet connection. Got PoE one because I want to try out PoE gadgets. This switch and cable are accompanied by a crimping toolkit. Learning to make ethernet cables was very satisfying. TP-Link VIGI C300HP - First candidate to utilize the PoE port on my switch. NVIDIA RTX 4060 Ti 16GB - This dude is responsible for doubling the electricity consumption of my lab. I got this one to experiment with AI developments going on. Mercusys MR70X - Got this router because it was OpenWrt compatible. Finally, I would once again ask you to subscribe to my channel, and have a nice day.\n","date":"12 November 2024","externalUrl":null,"permalink":"/posts/2024/introducing-my-homelab-and-services-running-within/","section":"Posts","summary":"Introduction # It’s been quite a while since I started homelabbing explicitly. I have already posted about Monitoring My HomeLab With Prometheus and Grafana, but I haven’t posted any video on my channel yet.\nSo today I want to take some time to talk about what I’m already up to. And for this, I’m going to divided this post into the following sections:\n","title":"Introducing my Homelab and services running within","type":"posts"},{"content":"","date":"12 November 2024","externalUrl":null,"permalink":"/tags/self-hosting/","section":"Tags","summary":"","title":"Self-Hosting","type":"tags"},{"content":"","date":"14 October 2024","externalUrl":null,"permalink":"/tags/containers/","section":"Tags","summary":"","title":"Containers","type":"tags"},{"content":" Introduction # Previously I have talked about how we can use Vagrant to make managing virtual machines easier. You can read about this article here: Let\u0026rsquo;s Use Vagrant to make Virtualbox less Tedious.\nWhile there are many types of virtualization and containerization techniques and tools available when writing this post, I\u0026rsquo;m specifically talking about virtualization and not containerization.\nVirtualization vs Containerization # Newcomers might get lost when they learn about these things and don\u0026rsquo;t know when to use what. These buzzwords might include Docker Containers, LXC (Linux Containers), Virtual Machines, Vagrant, VirtualBox, VMWare, etc.\nTo let you be more clear, let\u0026rsquo;s divide all of the things into 2 categories:\nContainerization Virtualization Talking about their similarities, they both provide some kind of isolation from the host system. This gives us the ability to mess around the guest OS more confidently knowing that we can also spin up a new one if something goes wrong.\nContainerization # What has made containers so popular today is its low overhead. What I mean by that is, that every containerization engine, say LXC or Docker, etc. uses the host systems\u0026rsquo; kernel. It is half the work/overhead needed from virtualization. If what I\u0026rsquo;m saying does not make sense, this article about User and Kernel space will help a bit.\nIn terms of use cases, containers are good for:\nQuick test or development work. Not that you can\u0026rsquo;t do this on virtual machines, but this one is lighter. Deployment using Docker, Kubernetes or other orchestrator This article is not about containers, so I\u0026rsquo;ll stop blabbering about it here.\nVirtualization # Virtualization has been longer than containerization if I am correct. The first thing I did as a consumer was to run VirtualBox on my Windows machine.\nVirtualization goes a bit deeper than containers in terms of isolation. It does not share kernel space with all its virtual machines. It can affect things in positive and negative ways. Negatively because the startup time is longer than containers. Positively because both the host and the guest system are more isolated from each other. Meaning that there is less chance of malicious programs lurking in your host system if things aren\u0026rsquo;t handled tightly.\nWell, I\u0026rsquo;ll not go deep into the explanation. Various articles and videos explain that. I\u0026rsquo;ll link a few here:\nVirtualization vs. Containerization in System Design https://www.geeksforgeeks.org/virtualization-vs-containerization/ Containers vs VMs: What\u0026rsquo;s the difference? https://www.youtube.com/watch?v=cjXI-yxqGTI This article is more about my personal experience of switching from one virtual machine manager to another, but still using Vagrantfiles to manage the VMs maintaining the infrastructure as code.\nKernel-based Virtual Machine # Kernel-based Virtual Machine (KVM) has been as long as 20 years from now. And I have only recently known about it. Both KVM and VirtualBox can work in conjunction. But it is out of the scope of this article.\nKVM is kind of an API and we need a daemon (libvirtd), and a client (virt-manager) to use it. This video: https://www.youtube.com/watch?v=BgZHbCDFODk explains how to set up and manage VMs using the GUI. Although the instructor uses an Ubuntu based system. There should be no issue in following your distro\u0026rsquo;s manual.\nRunning Linux Distro on KVM with Vagrant # You don\u0026rsquo;t need virt-manager to work with the VMs as it is similar to VirtualBox\u0026rsquo;s GUI. But it\u0026rsquo;s good to have if you feel lost.\nThe important part for Vagrant to work with KVM is the libvirtd. It\u0026rsquo;s the daemon we talked about in the last section. It\u0026rsquo;s like the containerd we have in the Kubernetes/Docker world.\nNext, let\u0026rsquo;s see how can we convert our VirtualBox-based Vagrantfile to KVM.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 Vagrant.configure(\u0026#34;2\u0026#34;) do |config| # The most common configuration options are documented and commented below. # For a complete reference, please see the online documentation at # https://docs.vagrantup.com. config.vm.box = \u0026#34;debian/bookworm64\u0026#34; config.vm.box_check_update = false config.vm.network :public_network, bridge: \u0026#34;enp4s0\u0026#34;, ip: 192.168.122.11 config.vm.define \u0026#34;proteus\u0026#34; do |node| # the string provided here appears in vagrant status node.vm.hostname = \u0026#34;proteus\u0026#34; # This appears in your router\u0026#39;s host listing, and in the VM itself # Provider-specific configuration so you can fine-tune various # backing providers for Vagrant. These expose provider-specific options. node.vm.provider \u0026#34;virtualbox\u0026#34; do |vb| vb.name = \u0026#34;proteus\u0026#34; # This appears in the VirtualBox UI vb.memory = 4096 vb.cpus = 2 end end # Enable provisioning with a shell script. Additional provisioners such as # Ansible, Chef, Docker, Puppet and Salt are also available. Please see the # documentation for more information about their specific syntax and use. config.vm.provision \u0026#34;shell\u0026#34;, inline: \u0026lt;\u0026lt;-SHELL apt-get update apt-get install -y build-essential tree bat git curl su - vagrant -c \u0026#34;git clone https://github.com/santosh/.dotfiles.git \u0026amp;\u0026amp; cd .dotfiles \u0026amp;\u0026amp; make install\u0026#34; SHELL end As you may have figured it out already, there is only one thing we need to modify i.e. the node.vm.provider. Here is a diff to the file after the modification:\n1 2 3 4 5 6 7 8 9 @@ -23,8 +23,8 @@ Vagrant.configure(\u0026#34;2\u0026#34;) do |config| - node.vm.provider \u0026#34;virtualbox\u0026#34; do |vb| - vb.name = \u0026#34;proteus\u0026#34; # This appears in the VirtualBox UI - vb.memory = \u0026#34;2048\u0026#34; - vb.cpus = 1 + node.vm.provider \u0026#34;libvirt\u0026#34; do |vb| + vb.memory = 4096 + vb.cpus = 2 end One more important change is to remove the vb.name line. This is because the name of the VM is picked up from the directory name. libvirt will throw an error if you kept the name field around. You can see the name of the VM by running virsh list --all in the terminal. virsh -c qemu:///system list --all if that does not work.\nAdditionally, you need to install the vagrant-libvirt plugin. You can do this by running the following command:\n1 vagrant plugin install vagrant-libvirt After you have done this, everything about Vagrant stays the same. vagrant ssh to the VM, or use the IP and private key to connect to the VM. Everything is the same. Now you know you can get rid of VirtualBox now.\nSomething still feels off to me. Like I\u0026rsquo;m prompted for a password when I run vagrant up. I suppose this is related to some NFS stuff going on. I\u0026rsquo;ll try to figure that out as I\u0026rsquo;m still new to KVM. If you happen to know the solution, please let me know in the comments below.\nConclusion # In this post, we have seen how we can use Vagrant to manage VMs on KVM. We have also seen how we can convert our VirtualBox-based Vagrantfile to KVM-based Vagrantfile. This is a step forward to ditch VirtualBox and use something more native to Linux.\nI hope you have learned something new today. If you have any questions, feel free to ask in the comments below. I\u0026rsquo;ll try to answer them as soon as possible.\n","date":"14 October 2024","externalUrl":null,"permalink":"/posts/2024/ditching-virtualbox-for-linux-own-kvm/","section":"Posts","summary":"Introduction # Previously I have talked about how we can use Vagrant to make managing virtual machines easier. You can read about this article here: Let’s Use Vagrant to make Virtualbox less Tedious.\nWhile there are many types of virtualization and containerization techniques and tools available when writing this post, I’m specifically talking about virtualization and not containerization.\n","title":"I'm Ditching VirtualBox for Something More Linux Native","type":"posts"},{"content":"","date":"14 October 2024","externalUrl":null,"permalink":"/tags/kvm/","section":"Tags","summary":"","title":"Kvm","type":"tags"},{"content":"","date":"14 October 2024","externalUrl":null,"permalink":"/tags/vagrant/","section":"Tags","summary":"","title":"Vagrant","type":"tags"},{"content":"","date":"14 October 2024","externalUrl":null,"permalink":"/tags/virtualization/","section":"Tags","summary":"","title":"Virtualization","type":"tags"},{"content":"","date":"8 July 2024","externalUrl":null,"permalink":"/tags/audacity/","section":"Tags","summary":"","title":"Audacity","type":"tags"},{"content":"","date":"8 July 2024","externalUrl":null,"permalink":"/tags/audio/","section":"Tags","summary":"","title":"Audio","type":"tags"},{"content":" Introduction # I have been recording screencasts for a while now. And as simple as it sounds and looks, there is always some pre and postprocessing required to make the screencast look and sound as it does. If you are reading this on my blog, I have a channel called Fullstack with Santosh where I post videos about tech. Go, get subscribed if you are wondering what what content I create.\nIn this post, I wanted to take time and share my workflow. Maybe it will become useful to some person who is looking to get started. It\u0026rsquo;s also a way for me to write down steps for future reference. So without further ado, let\u0026rsquo;s get started.\nNoise Cancellation at the time of Recording # So most of my programming videos are shot using OBS. OBS does a pretty good job of stacking different sources in a single video. By stacking sources, I mean that I can have a screensharing, a webcam, and a sound source all recording at once. I can even adjust the position and scale of the source on the screen so that I can be on the corner of the screenshare. OBS has been my go-to application for recording screen recordings.\nAs this post is about audio, when you add an audio source to OBS it might not have a really good noise-to-signal ratio by default. The OBS is going to capture whatever signal you get from the mic. OBS provides a way to alter these signals and enhance them by removing the noise. You can enable this by going to the audio source, and right-clicking on it. Then add Noise Suppression. There are two options shown here. Select one with more CPU consumption, RNNoise. Play around with recording and see if things changed for you.\nRNNoise is a noise suppression library based on a recurrent neural network, thus the name RNN-Noise. It is a very popular open source library to reduce background noise from audio streams. As I mentioned it is open source, it is not only limited to OBS Studio. I have gone one step ahead, and I enable noise cancellation at the system level whenever I log in. The same RNNoise algorithm is used. But it creates an input audio device with noise removed, which then I can use in applications like Zoom or others.\nTo achieve this at the system level, I use EasyEffects. EasyEffects does more than just RNNoise processing. But I only use it for noise reduction. You may want to explore it if your work is audio-related.\nAfter I am done with my recording. I don\u0026rsquo;t send them to Audacity at this time. I first align them, do the cuts, then export the audio. Then I process them as described below. After that, I replaced the original audio is the video editing software to have the enhanced version of the audio.\nAudacity Post Processing # These are the filters I have been using. These filters are supposed to be applied sequentially.\n0. Noise reduction # Role: to reduce/remove background noise.\nThis is numbered 0 because you shouldn\u0026rsquo;t be doing this in Audacity. This should be somewhere in the beginning of the pipeline while recording. RNNoise is one of the popular AI-based algorithms as I already mentioned in the previous section.\n1. Compressor # Role: Normalizes high peaks\nIt will reduce high peaks of sound frequencies to normalize it. Leave this at default settings.\n2. Limiter # Role: Normalize low trenches\nWorks in conjunction with Amplifier to increase the low trenches. I personally don\u0026rsquo;t find it using this more often. As I have a unidirectional mic, I prefer staying at a constant distance whenever I speak in the microphone.\n3. EQ Bass and Treble Boost # Role: To pump bass and Treble into the clip\nThese two are separate filters, have placed them together as they are very similar.\nThe effect of this one is more dramatic than any other filter. These are the filters I call them audio enhancers. Compressor and Limiter are the utility filters.\n4. Amplifier # Role: Increase the volume in general.\nIn general, I mean quieter parts as well as louder parts are made louder in a similar manner. No special treatment is given.\nYou can use Macro and set those parameters for the specific filters if you have more than one file. But I rarely find it doing for myself. I prefer doing it manually.\nBonus # As part of my entire audio processing workflow, there is one more thing I add in some of my videos, mostly the travel videos it is essential. This thing is copyright-free (or attribution-required) music.\nHere are a couple of libraries where you can find such music:\nYouTube Music Library SoundCloud BenSound Incompetech Free Music Archive My current preference is BenSound. But it\u0026rsquo;s only until I run out of relevant free music on their site.\nConclusion # This has been my audio processing workflow. I would like to plug my programming channel as well as travel channel below:\nFullstack with Santosh Backpack with Santosh Get subscribed to know more about me. Thanks and have a nice day.\n","date":"8 July 2024","externalUrl":null,"permalink":"/posts/2024/my-audio-editing-workflow-feat-rnnosie-and-audacity/","section":"Posts","summary":"Introduction # I have been recording screencasts for a while now. And as simple as it sounds and looks, there is always some pre and postprocessing required to make the screencast look and sound as it does. If you are reading this on my blog, I have a channel called Fullstack with Santosh where I post videos about tech. Go, get subscribed if you are wondering what what content I create.\n","title":"My Audio Editing Workflow feat. RNNoise and Audacity","type":"posts"},{"content":"","date":"8 July 2024","externalUrl":null,"permalink":"/tags/screencast/","section":"Tags","summary":"","title":"Screencast","type":"tags"},{"content":"","date":"12 May 2024","externalUrl":null,"permalink":"/tags/archlinux/","section":"Tags","summary":"","title":"Archlinux","type":"tags"},{"content":"","date":"12 May 2024","externalUrl":null,"permalink":"/tags/brave/","section":"Tags","summary":"","title":"Brave","type":"tags"},{"content":"","date":"12 May 2024","externalUrl":null,"permalink":"/tags/kwallet/","section":"Tags","summary":"","title":"Kwallet","type":"tags"},{"content":" Introduction # This post is about my recent experience with Brave browser on Arch Linux. I was getting logged out which was very annoying. After a series of hits and trials, Brave finally does not log me out. In this post, I\u0026rsquo;m going to share my journey and the decisions I made to achieve here.\nFirst pain with Arch Backstory # For those who don\u0026rsquo;t know, I recently built a PC for the first time in my life (the first time I was doing every connection) because of my aging laptop.\nFun fact: This is the 4th computer I\u0026rsquo;m owning, and 3rd PC. My last one was a laptop.\nWhen everything was working properly, I decided to declare this machine as an experiment ground. By that, I meant I\u0026rsquo;d carry important work-related stuff on my laptop itself.\nAs it was experimental, I decided to install Arch on it. The reason you ask? Hyprland was gaining a lot of popularity on r/unixporn. Hyprland is a tiling window manager. If this machine was going to be an experiment, why not try new things?\nI followed my friend Gabriel Chamon\u0026rsquo;s crash course about setting up Hyprland and the ecosystem. You can find them at: https://xd1.dev/2024/03/hyprland-crash-course\nIssue with Brave and explored options # My setup was fun. I was euphoric. The distro was not giving me any issues regardless of what I had heard of Arch is a rolling release and breaking frequently.\nBut one day, after the system update, it was not hard to notice that my Brave was not starting which was configured to start at session startup.\nI tried launching it from the terminal to diagnose the issue by looking at its log. It seems like Brave requires certain things that were not installed. It caught me by surprise because the dependency should have been installed by the package manager itself.\nInstalling kwallet # Seems like, by default, Brave requires kwallet to store the credentials of users. These are the credentials used for logging in to different websites.\nI installed kwallet. And finally my brave was back to normal. Just an annoyance that whenever I started Brave, I had to unlock the wallet by putting in the password.\nJust so that you know, this prompt to enter a password has a timeout. And in a situation where Brave is auto-starting on a session, if you don\u0026rsquo;t enter the password quickly, you are logged out from Brave on all sites for the login session.\nI lived with this annoyance for a couple of weeks. But it was a pain point to survive on Arch. So I started to look at solutions.\nInstalling kwallet-pam and \u0026hellip; # I heard a new concept of PAM. I don\u0026rsquo;t know too deep about it, but this is something that can be used to unlock the kwallet automatically. This is exactly what I wanted.\nAs Brave looks for kwallet, Arch has a kwallet-pam package which can work for me. But a twist with using such PAM plugins is that the wallet password and the login password must be the same. But wait, the wallet was already created. How do I change the password now?\nMeet kwalletmanager. It\u0026rsquo;s a UI for the kwallet backend. As the name implies, you can manage every aspect of the wallet, from creating, and deleting, to changing the password.\nYou\u0026rsquo;ll find a lot of confusion in configuring kwallet-pam if you are doing it for the first time. In the documentation, they\u0026rsquo;ve mentioned that keeping the name of the wallet to the default for many applications to work. The default name they mention is kwallet (not to be confused with the package name).\nBut when you go ahead to create a wallet without specifying the name. It will create a wallet with the name Default keyring. To add more to the confusion, I was convinced that Default keyring is the default name because I was familiar with this name on Ubuntu.\nI tried both, but Brave on the other hand was getting logged out on every reboot no matter what.\nWhat finally worked # What I\u0026rsquo;m going to describe is not the most secure option. And please, if you find a secure setup that is working for you; I\u0026rsquo;m looking forward to it. Please let me know in the comments.\nThere is a flag in Brave, which is basically inherited from Chromium. It\u0026rsquo;s called --password-store. By default, it is kwallet (at least on my Arch install) if you don\u0026rsquo;t pass it anything. There is an option to change it to basic.\nNot sure how to secure my password is, because I never save my password to the browser. I always select the stay signed in option. And whether or not I\u0026rsquo;m logged in or out of a site should be related to cookies, not a password manager. But still, I find this a gray area.\nIf you know more about this situation. Or have already passed through this experience, I\u0026rsquo;d love to hear back from you. Until then, have a good time.\n","date":"12 May 2024","externalUrl":null,"permalink":"/posts/2024/workaround-brave-getting-logout-after-every-reboot-on-arch-linux/","section":"Posts","summary":"Introduction # This post is about my recent experience with Brave browser on Arch Linux. I was getting logged out which was very annoying. After a series of hits and trials, Brave finally does not log me out. In this post, I’m going to share my journey and the decisions I made to achieve here.\nFirst pain with Arch Backstory # For those who don’t know, I recently built a PC for the first time in my life (the first time I was doing every connection) because of my aging laptop.\n","title":"Workaround: Brave Getting Logout After Every Reboot on Arch Linux","type":"posts"},{"content":" Introduction # I had known about Vagrant from before. However, it was not until I was doing a Kubernetes course that I encountered it again. The instructor was spinning up and shutting down VMs like a piece of cake. This made me rethink about virtualization again.\nI have been configuring Virtualbox till now with GUI, and it\u0026rsquo;s pretty cumbersome to download ISOs, configure network, storage, and whatnot. While it is good for beginners to see what\u0026rsquo;s going on, it created a drive in me to steer away from it. But the time has changed now. Introducing Vagrant.\nWhy I liked Vagrant # The reason I am compelled to learn Vagrant is because it bootstraps everything up until the level that you can ssh into it. That too without opening up any GUI.\nI\u0026rsquo;m taking time to learn Vagrant and will keep updating this post as I go.\nInstallation # Well, no project owner suggests this, but I installed Vagrant using my package manager. Project owners suggest this because distro package managers are generally behind, e.g. Ubuntu generally has 6 months schedule to update their repository.\nThis is done to keep the system stable by not getting every update from upstream. Btw I use Arch. This is a rolling release distro, meaning that it can break more often because it has pretty much the current version of all the packages in its repository.\nIf you are also on Arch, Arch Wiki has a page related to Vagrant. If you are not on Arch, head over to Vagrant download page to add Vagrant\u0026rsquo;s repository and get started.\nNot to mention, Virtualbox is dependency to Vagrant. I mean there are other providers available, but I\u0026rsquo;m going to use Virtualbox today.\nVagrant commands to get started # Vagrant works similar to Docker. In Docker, we can use CLI to launch a container, but a Dockerfile makes work easier.\nVagrant follows the same philosophy of specifying the spec. But with the added benefit that the Vagrant file is a valid Ruby file. So you can have your own logic there.\nNever seen a Vagrantfile? No problem. I had also not seen one except the one I saw on the course. That was already written by someone, but actually, you can generate your own:\nvagrant init ubuntu/jammy64\nThis command is going to present you with a Vagrant file inside the current working directory. So make sure you are in the directory where you want to keep all your vagrant files. This is not a requirement, but I like to keep things organized. So I already created ~/workspace/vagrant-vms/ubuntu-jammy before I ran that command.\nThe generated file is going to have mostly comments except for these two lines:\n1 2 Vagrant.configure(\u0026#34;2\u0026#34;) do |config| config.vm.box = \u0026#34;ubuntu/jammy64\u0026#34; This configures the VM to use a Ubuntu distro, specifically Jammy 64-bit. Jammy is the codename for 22.04.\nThis is not the configuration you\u0026rsquo;d like for a regular VM, but good enough to get started.\nI will cover details restated to Vagrantfile in a separate post. Let\u0026rsquo;s keep the scope of this post to make creating VMs fun.\nYou can use the following command to spawn a VM with this spec:\n1 vagrant up I hope you don\u0026rsquo;t get any errors at this point because I didn\u0026rsquo;t. If you did, most probably you need to find vagrant documentation that talks about prerequisites.\nIf the vagrant up command is successful, you can check what VMs are running by issuing this command:\n1 2 3 4 5 6 7 8 9 $ vagrant status Current machine states: default running (virtualbox) The VM is running. To stop this VM, you can run `vagrant halt` to shut it down forcefully, or you can run `vagrant suspend` to simply suspend the virtual machine. In either case, to restart it again, simply run `vagrant up`. As you can see, it lists the name of the VM we started (default), the state of the VM (running), and the provider (VirtualBox).\nWe\u0026rsquo;ll see how to change the default name in another post, but for now, let\u0026rsquo;s SSH into it.\n1 vagrant ssh default You should now be inside your machine.\nCongrats! See how easy it is to spin up a VM with SSH access?\nWhat more you can do: Play around. Check the network interfaces and what IP has been assigned to you, ping other servers, etc if you are a nerd.\nThat\u0026rsquo;s enough of a demo for me today. Let\u0026rsquo;s exit from the SSH login, and get back to the host machine.\nBefore we wrap up this post with a promise to create another post with a focus on Vagrantfile, let\u0026rsquo;s learn how to get rid of our VMs when we are not using them.\nThis part has 2 relevant commands. One is halt and another is suspend.\nI prefer halt. But it might not make sense for other use cases, so it depends upon you.\n1 vagrant halt default A quick note on the difference between both commands mentioned above stolen from this StackOverflow post:\nUse vagrant halt when you want to power off your machine, and use vagrant suspend when you want to hibernate your machine. A suspend effectively saves the exact point-in-time state of the machine, so that when you resume it later, it begins running immediately from that point, rather than doing a full boot.\nIf you really wanna get rid of the VirtualBox created. And this is equivalent to removing entries from the VirtualBox UI, there is another command you might be interested in:\n1 vagrant destroy default And this will wrap everything up.\nConclusion # I won\u0026rsquo;t make this post lengthy like some of my last posts by including everything in the same post. I will create another article with details on Vagrantfile, and what my regular Vagrantfile looks like. Until then, keep reading my other posts, and subscribe to my new YouTube channel: Fullstack with Santosh\n","date":"5 May 2024","externalUrl":null,"permalink":"/posts/2024/lets-use-vagrant-to-make-virtualbox-less-tedious/","section":"Posts","summary":"Introduction # I had known about Vagrant from before. However, it was not until I was doing a Kubernetes course that I encountered it again. The instructor was spinning up and shutting down VMs like a piece of cake. This made me rethink about virtualization again.\nI have been configuring Virtualbox till now with GUI, and it’s pretty cumbersome to download ISOs, configure network, storage, and whatnot. While it is good for beginners to see what’s going on, it created a drive in me to steer away from it. But the time has changed now. Introducing Vagrant.\n","title":"Let's Use Vagrant to make Virtualbox less Tedious","type":"posts"},{"content":"","date":"5 May 2024","externalUrl":null,"permalink":"/tags/virtualbox/","section":"Tags","summary":"","title":"Virtualbox","type":"tags"},{"content":"","date":"5 May 2024","externalUrl":null,"permalink":"/tags/vm/","section":"Tags","summary":"","title":"Vm","type":"tags"},{"content":"","date":"25 April 2024","externalUrl":null,"permalink":"/tags/alerting/","section":"Tags","summary":"","title":"Alerting","type":"tags"},{"content":"","date":"25 April 2024","externalUrl":null,"permalink":"/tags/devops/","section":"Tags","summary":"","title":"Devops","type":"tags"},{"content":"","date":"25 April 2024","externalUrl":null,"permalink":"/tags/grafana/","section":"Tags","summary":"","title":"Grafana","type":"tags"},{"content":"","date":"25 April 2024","externalUrl":null,"permalink":"/tags/metrics/","section":"Tags","summary":"","title":"Metrics","type":"tags"},{"content":"","date":"25 April 2024","externalUrl":null,"permalink":"/tags/monitoring/","section":"Tags","summary":"","title":"Monitoring","type":"tags"},{"content":" Introduction # I have enough machines in my house that I love to call the setup a home lab. Though I\u0026rsquo;m not saying my setup is as cool as the one you find on r/homelab. But I have my laptop which I have been using for 3 years now. I also bought a Raspberry Pi 4 back at the end of 2022. Recently I\u0026rsquo;ve built a PC for which I\u0026rsquo;m yet to upload video.\nThat\u0026rsquo;s enough number of devices that I want to know more about at a single place. The thing in my favor is Raspi, which can be up 24x7 without taking much power. This is going to be my server for telemetry.\nIf you are following along with this setup, you need to have at least 2 hosts. I have my raspi and Arch PC with me. Make sure you take note of the IP addresses of these hosts. It will be better if you configure a static IP for each of those devices so they don\u0026rsquo;t create problems.\nI prefer to set static IP from my router. How you set it depends on the make and model of your router. You can consult your product\u0026rsquo;s documentation.\nStep 1: Prometheus # There is this project which I came across which is lightweight and serves the purpose.\nI headed over to the Getting Started with Prometheus page and followed the steps to source data. Once everything is running, you should be able to see Prometheus UI at localhost:9090.\nWhile above mentioned method is sufficient for testing, it requires some plumbing to set up the systemd files which enables Prometheus to start the collector daemon as soon as the system starts. We can actually write your own systemd files for the binary you downloaded, there are a lot of online resources available for that. But if you are lazy, keep reading.\nMy final target to install Prometheus was my Raspi host. It has Raspberry Pi OS, which is basically Debian bookworm. If you have the same setup, I\u0026rsquo;d recommend installing it from the system package manager.\n1 sudo apt install prometheus Don\u0026rsquo;t forget to enable the Prometheus service to start at system boot.\n1 2 sudo systemctl enable prometheus sudo systemctl start prometheus At this point, you should be able to access the Prometheus UI at \u0026lt;raspi-ip:9090\u0026gt;. Here is what it looks like to me. I have actually gone a step further and looked up a sample query to run and plot the graph on.\nPrometheus Query with Graph Adding scrape targets # So Prometheus is doing its work. To start with, I\u0026rsquo;m going to erase everything and have this config in my /etc/prometheus/prometheus.yml:\n1 2 3 4 5 6 7 8 global: scrape_interval: 15s scrape_configs: - job_name: \u0026#34;prometheus\u0026#34; static_configs: - targets: - localhost:9090 Pay attention to the job_name and targets. We\u0026rsquo;ll be referencing them in future sections. Going forward, I\u0026rsquo;ll be using each hostname as a value for job_name. And targets will be the IP of that node. But let\u0026rsquo;s not worry about it right now.\nDon\u0026rsquo;t forget to reload the service after changing the configuration:\n1 sudo systemctl reload prometheus The configuration above is scraping data from the Prometheus instance itself. And for each job_name you add here, you should see the respective entry on the Targets page http://\u0026lt;raspi-ip\u0026gt;:9090/classic/targets:\nPrometheus Targets We are going to add more targets for each host we want to keep track of. But are we going to install Prometheus on all the hosts? Not actually!\nStep 2: node-exporter # node-exporter is a kind of exporter; the exporter exports certain data from the host to Prometheus. exporters can different kinds of collectors. collectors are not limited to hosts/machines because there are various integrations possible. But today, we are working with node-exporter.\nnode-exporter exposes information about a system in general. Information such as CPU, network, disk, and memory usage. This is what I want as a starter kit.\nInstallation methods are again the same for raspi:\n1 2 3 sudo apt install prometheus-node-exporter sudo systemctl enable prometheus-node-exporter sudo systemctl start prometheus-node-exporter We don\u0026rsquo;t need any fancy configuration for node exporter. I\u0026rsquo;m also going to configure node exporter on other machines.\nOn Arch:\n1 2 3 yay -S prometheus-node-exporter sudo systemctl enable prometheus-node-exporter sudo systemctl start prometheus-node-exporter I\u0026rsquo;m silently going to follow similar steps on my Kubuntu machine.\nNow the time is to update the Prometheus config on the raspi:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 global: scrape_interval: 15s scrape_configs: - job_name: \u0026#34;prometheus\u0026#34; static_configs: - targets: - localhost:9090 - job_name: \u0026#34;archlinux\u0026#34; static_configs: - targets: - 192.168.0.104:9100 - job_name: \u0026#34;raspberrypi\u0026#34; static_configs: - targets: - localhost:9100 job_name is used to identify the host in Grafana. targets is the IP. The node-exporter listens on port 9100.\nAt this point, the prometheus job can be removed, because we already installed node-exporter on it, and Prometheus data is not relevant to us.\nThe yaml is stripped down to this:\n1 2 3 4 5 6 7 8 9 10 11 12 13 global: scrape_interval: 15s scrape_configs: - job_name: \u0026#34;archlinux\u0026#34; static_configs: - targets: - 192.168.0.104:9100 - job_name: \u0026#34;raspberrypi\u0026#34; static_configs: - targets: - localhost:9100 Let\u0026rsquo;s move on to the most interesting part that we were waiting for, which is visualization.\nStep 3: Grafana # We are about to see the most beautiful part, the visualization of collected system information over time.\nAgain, on Debian, I followed these steps: Install Grafana on Debian or Ubuntu\nDon\u0026rsquo;t forget to start the systemd service:\n1 2 3 4 sudo apt-get install grafana sudo systemctl enable grafana-server sudo systemctl start grafana-server sudo systemctl status grafana-server Login to the grafana dashboard. For me, it was 192.168.0.100:3000. Use admin as username and password. The installer will ask to change the password after first use.\nAdd Source # Head over to the Data sources section e.g. http://192.168.0.100:3000/connections/datasources and add a Prometheus instance.\nGrafana Add Datasource In the screen that appears afterward, only the server URL field is required. But I change the name for organizational purposes. I have left every other setting are their default.\nAdd Dashboard # Head over to add the dashboard section. Look for a button saying \u0026ldquo;New\u0026rdquo;. Click that and select the option \u0026ldquo;Import\u0026rdquo;. In the form that appears, use 1860 in the ID. In the form that appears, you\u0026rsquo;ll be prompted to choose the Prometheus instance. Go ahead and choose the one we set up. Troubleshooting # You should be now in a shape to view your dashboard. If not, here are some places to look at:\nCheck if the data source is correctly set up and running. And you are specifying the correct port in Grafana UI. Make sure Prometheus config at /etc has the correct hostname/IPs. If you missed setting up static IP for your machine, it will be tricky. If you have a firewall enabled on any of your hosts, please allow port 9100 used by node-exporter from the host which has Prometheus on. Dashboard Overview # The dashboard we imported takes the data from the node exporter and visualizes it using graphs and colors.\nGrafana Dashboard Overview After having a glance.. do you remember the time when I asked you to pay attention to the job_name and targets?\nIn the dashboard UI, they are available as Job and Host. Look at the top of the screenshot, just below the navbar breadcrumb. You can use the Job dropdown to switch to the different hosts you have set the node exporter on.\nAnd that was pretty much it for this article.\nConclusion # Doing DIY stuff is fun and a great way to learn. I recommend having a Raspberry with any tech enthusiast. It provides an opportunity to self-host stuff and learn from the process of setting it up and maintaining it. By maintenance, I mean regular backup of generated data, etc.\nI\u0026rsquo;m hoping to do more with my raspi in the future. And Prometheus-Grafana duo has provided the base I was missing.\nI\u0026rsquo;ll share my journey with you as I go. So stay tuned.\n","date":"25 April 2024","externalUrl":null,"permalink":"/posts/2024/monitoring-my-homelab-with-prometheus-and-grafana/","section":"Posts","summary":"Introduction # I have enough machines in my house that I love to call the setup a home lab. Though I’m not saying my setup is as cool as the one you find on r/homelab. But I have my laptop which I have been using for 3 years now. I also bought a Raspberry Pi 4 back at the end of 2022. Recently I’ve built a PC for which I’m yet to upload video.\n","title":"Monitoring My HomeLab With Prometheus and Grafana","type":"posts"},{"content":"","date":"25 April 2024","externalUrl":null,"permalink":"/tags/prometheus/","section":"Tags","summary":"","title":"Prometheus","type":"tags"},{"content":" Privacy Policy # Effective Date: 18th March, 2024\nIntroduction # Welcome to Fullstack with Santosh! This Privacy Policy describes how Santosh Kumar (\u0026ldquo;we,\u0026rdquo; \u0026ldquo;us,\u0026rdquo; or \u0026ldquo;our\u0026rdquo;) collects, uses, and discloses your personal information when you visit our website at https://santoshk.dev (\u0026ldquo;Site\u0026rdquo;). We are committed to protecting your privacy and ensuring that you have a safe and enjoyable online experience.\nScope of this Policy # This Privacy Policy applies to all information collected through our Site. It does not apply to any information collected offline or through any third-party websites or applications, even if linked to our Site.\nInformation We Collect # We collect two main types of information about you:\nInformation You Provide Directly: Comments: When you leave a comment on a blog post or article, we may collect your name, email address, and the content of your comment. Contact Forms: If you contact us through a contact form, we may collect your name, email address, and the content of your message. Surveys and Polls: We may occasionally offer surveys or polls on our Site. Participation in these is entirely voluntary, and you can choose not to disclose any information. Information Collected Automatically: Log Data: When you visit our Site, our servers automatically collect information about your visit. This information may include your IP address, browser type, operating system, referring URL, pages visited, time spent on the Site, and other diagnostic data. Cookies: We may use cookies to store information about your preferences and visits to the Site. This could include things like your preferred language, browsing behavior, and past selections. You can control or disable cookies through your browser settings, but doing so may limit the functionality of the Site. Use of Information # We use the information we collect for the following purposes:\nTo operate and maintain the Site: We use the information we collect to ensure the smooth operation and technical functioning of the Site. This includes troubleshooting issues, analyzing usage data, and preventing fraud. To respond to your inquiries and requests: We use your information to respond to your comments, questions, and other requests submitted through the Site. To personalize your experience: We may use the information we collect to personalize your experience on the Site, such as remembering your preferences or recommending content that you might be interested in. To improve the Site: We analyze the information we collect to understand how users interact with the Site and to identify areas for improvement. To serve you advertising (if applicable): We may use the information we collect to serve you targeted advertising that is more relevant to your interests. However, we will always comply with applicable data privacy regulations regarding the use of personal information for advertising purposes. Sharing of Information # We will not share your personal information with any third party without your consent, except in the following limited circumstances:\nService Providers: We may share your information with third-party service providers who help us operate the Site, such as analytics providers, hosting providers, or comment system providers. These service providers are only authorized to use your information for the purpose of providing the services to us. Legal Requirements: We may be required to disclose your information if necessary to comply with a law, regulation, or court order. We may also disclose your information if we believe it is necessary to protect the rights and safety of ourselves or others. Your Rights # You have certain rights regarding your personal information, depending on the applicable data privacy regulation. These rights may include:\nThe right to access your personal information: You have the right to request a copy of the personal information we hold about you. The right to rectify inaccurate personal information: You have the right to request that we correct any inaccurate or incomplete personal information we hold about you. The right to request the erasure of your personal information: In certain circumstances, you have the right to request that we erase your personal information. The right to restrict the processing of your personal information: You have the right to request that we restrict the processing of your personal information in certain circumstances. The right to object to the processing of your personal information: You have the right to object to the processing of your personal information in certain circumstances, such as for direct marketing purposes. The right to data portability: In certain circumstances, you have the right to receive your personal information in a structured, commonly used, and machine-readable format and to transmit it to another controller. To exercise any of these rights, please contact me at sntshkmr60+santoshkprivacy@gmail.com\n","date":"14 March 2024","externalUrl":null,"permalink":"/privacy/","section":"Fullstack with Santosh","summary":"Privacy Policy # Effective Date: 18th March, 2024\nIntroduction # Welcome to Fullstack with Santosh! This Privacy Policy describes how Santosh Kumar (“we,” “us,” or “our”) collects, uses, and discloses your personal information when you visit our website at https://santoshk.dev (“Site”). We are committed to protecting your privacy and ensuring that you have a safe and enjoyable online experience.\n","title":"Privacy Policy","type":"page"},{"content":"","date":"15 September 2023","externalUrl":null,"permalink":"/tags/event-based/","section":"Tags","summary":"","title":"Event-Based","type":"tags"},{"content":"","date":"15 September 2023","externalUrl":null,"permalink":"/tags/mosquitto/","section":"Tags","summary":"","title":"Mosquitto","type":"tags"},{"content":"","date":"15 September 2023","externalUrl":null,"permalink":"/tags/mqtt/","section":"Tags","summary":"","title":"Mqtt","type":"tags"},{"content":" What is MQTT # MQTT (Message Queue Telemetry Transport) is a lightweight, publish-subscribe messaging protocol designed for low-bandwidth, high-latency, or unreliable networks. It is particularly well-suited for IoT (Internet of Things) devices and applications that require the transmission of small amounts of data over long periods of time.\nI always see MQTT as a sibling of AMQP. They are both messaging protocol. We\u0026rsquo;ll do a comparision later in the post.\nMQTT works by using a publish-subscribe model, where a client (e.g. an IoT device) publishes data to a broker (a server that receives and stores data) and another client subscribes to receive updates from the broker. The broker acts as a central hub, routing messages between clients and ensuring that only subscribed clients receive the data they are interested in.\nWhy MQTT # Low overhead. One of the key benefits of MQTT is its low overhead. It uses a compact binary message format and requires minimal network resources, making it ideal for use on low-bandwidth or constrained networks. It is also designed to be efficient in terms of battery usage, making it well-suited for use on battery-powered devices.\nScalable and Flexible. MQTT is also highly scalable and flexible, with support for multiple topics and the ability to handle millions of connections. This makes it well-suited for use in large-scale IoT deployments, where there may be a need to handle a large number of connected devices.\nSecure In addition, MQTT has built-in support for security and authentication, with the ability to use SSL/TLS to encrypt data in transit and authenticate clients. This is important for ensuring the privacy and security of IoT devices and data.\nMQTT vs AMQP # One of the main differences between MQTT and AMQP is the scope of their capabilities. MQTT is a lightweight protocol that is designed specifically for IoT applications and other scenarios where low-bandwidth, high-latency, or unreliable networks are a concern. It has a compact binary message format and requires minimal network resources, making it well-suited for use on constrained networks. In contrast, AMQP is a more feature-rich protocol that is designed for use in enterprise environments and has a more complex message format. It is generally better suited for use in high-bandwidth, low-latency networks.\nAnother difference between MQTT and AMQP is the way in which they handle messaging. MQTT uses a publish-subscribe model, where a client (e.g. an IoT device) publishes data to a broker (a server that receives and stores data) and another client subscribes to receive updates from the broker. In contrast, AMQP uses a more traditional message queuing model, where clients send messages to a queue and another client consumes those messages.\nIn terms of security, both MQTT and AMQP support the use of SSL/TLS to encrypt data in transit and authenticate clients. However, MQTT has more limited support for security features such as access control and authentication compared to AMQP.\nOverall, while both MQTT and AMQP are useful messaging protocols, they are designed for different use cases and have different capabilities. MQTT is a lightweight, efficient protocol that is well-suited for use in IoT applications and other scenarios where low-bandwidth, high-latency, or unreliable networks are a concern. AMQP is a more feature-rich protocol that is better suited for use in enterprise environments and high-bandwidth, low-latency networks.\nDigging Deeper into MQTT # To get a hang of the protocol, you need to establish these keywords in your mind.\nTopic: This is data structure where messages go through. One message can be part of many topics. Client: Client can be either produces, which post messages to topic. Or, subscriber, that listens on a topic for messages. Broker: Broker is the server part. All topics exist on broker. Clients can either post to it or subscribe to it. Install # We have leaned a lot now. Now we need to do some practicle to retain our knowledge for a long time. I am a Debian fan. I use Ubuntu for both the server and my local computer. I\u0026rsquo;ll show how can we install:\nNow you can install it on your raspberry pi. But here I\u0026rsquo;m going to install on one of my VPC.\nFirst, update the package manager index by running the following command: 1 sudo apt update Next, install mosquitto and its dependencies by running the following command: 1 sudo apt install mosquitto mosquitto-clients Once the installation is complete, you can start the mosquitto service by running the following command: 1 sudo systemctl start mosquitto To make sure that mosquitto starts automatically when the system boots up, you can enable the service by running the following command: 1 sudo systemctl enable mosquitto To check the status of the mosquitto service, you can run the following command: 1 sudo systemctl status mosquitto This was for the installation part, let\u0026rsquo;s see how can we use the accompanying CLI provided with the package:\nCLI sub/pub # If you want to test that mosquitto is working correctly, you can use the mosquitto_sub and mosquitto_pub utilities to publish and subscribe to messages. For example, to subscribe to the test topic, you can run the following command: 1 mosquitto_sub -h localhost -t test To publish a message to the test topic, you can run the following command:\n1 mosquitto_pub -h localhost -t test -m \u0026#34;Hello, World!\u0026#34; This should cause the message Hello, World! to be displayed in the terminal window where you are running the mosquitto_sub command.\nPost-install config # As you know, I have install mosquitto broker on a remote public machine. There is an additional step if you too have this kind of setup. What I have found out is, even if you client is reachable to the broker, you need to enable broker to allow client from different host.\nI am using below configuration. Please note that this is not the most secure setup out these. This is just for testing. Please let me know if want me to cover that in an another post.\nFor now, please stick this to your /etc/mosquitto/mosquitto.conf:\n1 2 3 listener 1883 allow_anonymous true protocol mqt Now you can connect your local client to the remote broker.\nBut that is not enough. Working from CLI is just for demonstration. In real life you\u0026rsquo;d be using different libraries in different language to interact with brokers.\nIn coming sections we\u0026rsquo;ll see demos in some of the most popular language of time.\nPython sub/pub # To be able to publish or subscribe to a MQTT topic, you first have some library on your local computer. paho is one of the most popular such library available in the wild.\nLet us first install paho:\n1 pip install paho-mqtt We can now write our clients.\nsubscriber.py:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 import sys import paho.mqtt.client as paho def on_message(client, userdata, msg): print(msg.topic + \u0026#34;: \u0026#34; + msg.payload.decode()) client = paho.Client() client.on_message = on_message if client.connect(\u0026#34;localhost\u0026#34;, 1883, 60) != 0: print(\u0026#34;Couldn\u0026#39;t connect to MQTT broker!\u0026#34;) sys.exit(-1) client.subscribe(\u0026#34;test/status\u0026#34;) try: print(\u0026#34;Press CTRL+C to exit\u0026#34;) client.loop_forever() except: print(\u0026#34;Exiting...\u0026#34;) client.disconnect() You should replace localhost with some public IP or hostname.\nAs you can see, there\u0026rsquo;s a on_message function which is called everytime a new message is seen on the topic. We are connecting to the broker in next few lines. And later on, we subscribe to the topc test/status. loop_forever tells the client to do not exit the script.\nThe code is pretty self explanatory.\nNow let us see the publisher.\npublisher.py:\n1 2 3 4 5 6 7 8 9 10 11 12 import sys import paho.mqtt.client as paho client = paho.Client() if client.connect(\u0026#34;localhost\u0026#34;, 1883, 60) != 0: print(\u0026#34;Couldn\u0026#39;t connect to MQTT broker!\u0026#34;) sys.exit(-1) client.publish(\u0026#34;test/status\u0026#34;, \u0026#34;Hello World from paho-mqtt!\u0026#34;) client.disconnect() This publisher code is also pretty self explanatory. We are publishing Hello World from paho-mqtt! to test/status topic.\nYou can run the subscriber in a different terminal and then publisher in the other. You can see that you see a new message.\nIn a real life message, you\u0026rsquo;d process those message and check if server needs to perform any action based on the content of the message.\nGolang sub/pub # The library which is used in Go world is eclipse/paho.mqtt.golang. You can get it via below command: 1 go get github.com/eclipse/paho.mqtt.golang Next, create a new Go file and import the MQTT library: 1 import \u0026#34;github.com/eclipse/paho.mqtt.golang\u0026#34; To create a new MQTT client, you can use the following code: 1 client := mqtt.NewClient(mqtt.NewClientOptions().AddBroker(\u0026#34;tcp://localhost:1883\u0026#34;)) I have added localhost above, but use the address of your broker.\nTo subscribe to a topic, you can use the following code: 1 2 3 4 5 6 token := client.Subscribe(\u0026#34;mytopic\u0026#34;, 0, func(client mqtt.Client, msg mqtt.Message) { // This function is called whenever a message is received fmt.Printf(\u0026#34;Received message on topic %s: %s\\n\u0026#34;, msg.Topic(), string(msg.Payload())) }) token.Wait() This code subscribes to the \u0026ldquo;mytopic\u0026rdquo; topic and sets up a callback function that is called whenever a message is received.\nTo publish a message to a topic, you can use the following code: 1 client.Publish(\u0026#34;mytopic\u0026#34;, 0, false, \u0026#34;Hello, World!\u0026#34;) This code publishes the message \u0026ldquo;Hello, World!\u0026rdquo; to the \u0026ldquo;mytopic\u0026rdquo; topic.\nThat\u0026rsquo;s it! You should now have a basic MQTT publish/subscribe client that can send and receive messages. You can use this as a starting point for more complex MQTT applications.\n","date":"15 September 2023","externalUrl":null,"permalink":"/posts/2023/mqtt-basics-and-how-to-get-started-on-ubuntu/","section":"Posts","summary":"What is MQTT # MQTT (Message Queue Telemetry Transport) is a lightweight, publish-subscribe messaging protocol designed for low-bandwidth, high-latency, or unreliable networks. It is particularly well-suited for IoT (Internet of Things) devices and applications that require the transmission of small amounts of data over long periods of time.\nI always see MQTT as a sibling of AMQP. They are both messaging protocol. We’ll do a comparision later in the post.\n","title":"MQTT basics and how to get started on Ubuntu","type":"posts"},{"content":"","date":"15 September 2023","externalUrl":null,"permalink":"/tags/paho/","section":"Tags","summary":"","title":"Paho","type":"tags"},{"content":"","date":"15 September 2023","externalUrl":null,"permalink":"/tags/telemetry/","section":"Tags","summary":"","title":"Telemetry","type":"tags"},{"content":"","date":"15 August 2023","externalUrl":null,"permalink":"/tags/firewall/","section":"Tags","summary":"","title":"Firewall","type":"tags"},{"content":" Introduction # In a previous post I have talked about introduction and basic usage with iptables. I did some more research on the subject and I found out more information about it which I would like to share with you.\nBest Practices for Configuring iptables Rules # When configuring iptables rules, it\u0026rsquo;s important to follow best practices to ensure that your firewall is effective and secure. Here are some best practices for configuring iptables rules:\nDefine a Default Policy - Always define a default policy for each chain, specifying whether to accept or drop packets that don\u0026rsquo;t match any of the rules.\nUse Specific Rules - Use specific rules to allow or deny traffic, rather than using broad rules that may inadvertently allow or deny traffic that you didn\u0026rsquo;t intend to.\nKeep Rules Simple - Keep your iptables rules simple and easy to understand. Complex rules can be difficult to troubleshoot and may not be effective in preventing attacks.\nTest Your Rules - Always test your iptables rules thoroughly before implementing them in a production environment. This can help you identify any issues or conflicts before they cause problems.\nAdvanced iptables Commands # There are many advanced iptables commands that you can use to create complex configurations. Here are some examples:\nLimiting Connections - You can use the \u0026ldquo;-m connlimit\u0026rdquo; module to limit the number of connections that can be made to a specific port.\nSource NAT - You can use the \u0026ldquo;-j SNAT\u0026rdquo; option to change the source IP address of outgoing traffic.\nDestination NAT - You can use the \u0026ldquo;-j DNAT\u0026rdquo; option to change the destination IP address of incoming traffic.\nUse Cases for iptables # Iptables can be used for a variety of purposes, such as load balancing, port forwarding, and traffic shaping. Here are some examples:\nLoad Balancing - You can use iptables to distribute incoming traffic among multiple servers to improve performance and reliability.\nPort Forwarding - You can use iptables to forward incoming traffic from one port to another, allowing you to run multiple services on a single server.\nTraffic Shaping - You can use iptables to limit the amount of bandwidth that certain types of traffic can use, allowing you to prioritize critical traffic over non-critical traffic.\nCommon iptables Pitfalls and Troubleshooting # When configuring iptables, it\u0026rsquo;s important to be aware of common issues that can cause problems. Here are some common iptables pitfalls and troubleshooting tips:\nSyntax Errors - Syntax errors can cause iptables rules to fail. Always double-check your syntax to ensure that your rules are correct.\nChain Order - The order of your chains can affect the way that your iptables rules are applied. Make sure that your rules are in the correct order to ensure that they are applied correctly.\nDebugging Output - Use the \u0026ldquo;-v\u0026rdquo; option to enable verbose output, which can help you identify issues and conflicts in your iptables rules.\nResources for Further Learning and Support # If you want to learn more about iptables, there are many resources available to help you. Here are some useful resources:\nIptables Documentation - The official documentation for iptables is a great resource for learning more about iptables.\nOnline Communities - There are many online communities where you can ask questions and get help with iptables, such as the LinuxQuestions.org forums.\nTutorials - There are many tutorials available online that can help you learn how to use iptables more effectively.\nConclusion # In conclusion, iptables is an essential tool for managing the security of your Linux system. By following best practices, using advanced configurations, and troubleshooting common issues, you can use iptables to\n","date":"15 August 2023","externalUrl":null,"permalink":"/posts/2023/iptables-advance-usage-and-common-pitfalls/","section":"Posts","summary":"Introduction # In a previous post I have talked about introduction and basic usage with iptables. I did some more research on the subject and I found out more information about it which I would like to share with you.\nBest Practices for Configuring iptables Rules # When configuring iptables rules, it’s important to follow best practices to ensure that your firewall is effective and secure. Here are some best practices for configuring iptables rules:\n","title":"iptables - Advance Usage and Common Pitfalls","type":"posts"},{"content":"","date":"15 August 2023","externalUrl":null,"permalink":"/tags/network-security/","section":"Tags","summary":"","title":"Network-Security","type":"tags"},{"content":"","date":"7 August 2023","externalUrl":null,"permalink":"/tags/docker/","section":"Tags","summary":"","title":"Docker","type":"tags"},{"content":"","date":"7 August 2023","externalUrl":null,"permalink":"/series/docker-go-sdk/","section":"Series","summary":"","title":"Docker Go SDK","type":"series"},{"content":"","date":"7 August 2023","externalUrl":null,"permalink":"/tags/go/","section":"Tags","summary":"","title":"Go","type":"tags"},{"content":"","date":"7 August 2023","externalUrl":null,"permalink":"/series/","section":"Series","summary":"","title":"Series","type":"series"},{"content":"In this blog post we continue our last post for most advanced use cases. We talk about managing docker containers in which we talk about spinning a container, listing, starting, stopping, and removing container. We talk about using docker network. We talk about streaming and retrieving logs from docker container. We also cover docker working with docker volumes using Go. At last, we talk about error handling and best practices.\nThis post is continuation of that post. You can follow the series here: Go Docker SDK. Now let\u0026rsquo;s continue on that knowledge.\nManaging Docker Containers # Spinning up containers using Go SDK # One of the key features of Docker is the ability to run containers. With the Docker Go SDK, you can programmatically spin up containers from images. Here\u0026rsquo;s an example of how you can achieve this:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 func createContainer(dockerClient *client.Client, imageName string, containerName string) (string, error) { ctx := context.Background() config := \u0026amp;container.Config{ Image: imageName, } hostConfig := \u0026amp;container.HostConfig{} resp, err := dockerClient.ContainerCreate(ctx, config, hostConfig, nil, containerName) if err != nil { return \u0026#34;\u0026#34;, err } containerID := resp.ID return containerID, nil } In the code snippet above, dockerClient.ContainerCreate() is used to create a container based on the specified image. You provide the image name, container configuration, and host configuration as parameters. The function returns a response object that contains the newly created container\u0026rsquo;s ID.\nYou can then use the container ID to perform further operations on the container, such as starting, stopping, or removing it.\nListing running containers # To obtain information about the running containers on your Docker host programmatically, you can use the Docker Go SDK. Here\u0026rsquo;s an example:\n1 2 3 4 5 6 7 8 9 10 func listContainers(dockerClient *client.Client) ([]types.Container, error) { ctx := context.Background() containers, err := dockerClient.ContainerList(ctx, types.ContainerListOptions{}) if err != nil { return nil, err } return containers, nil } In the code snippet above, dockerClient.ContainerList() is used to retrieve a list of running containers. The types.ContainerListOptions{} parameter allows you to specify filters or additional options for the container listing if needed. The function returns a slice of types.Container containing information about each running container.\nYou can then process the retrieved container list, extract relevant details such as the container ID, name, or status, and perform further operations based on your requirements.\nStarting, stopping, and removing containers programmatically # The Docker Go SDK allows you to start, stop, and remove containers programmatically. Here are some examples:\nStarting a container:\n1 2 3 4 5 6 7 8 9 10 func startContainer(dockerClient *client.Client, containerID string) error { ctx := context.Background() err := dockerClient.ContainerStart(ctx, containerID, types.ContainerStartOptions{}) if err != nil { return err } return nil } Stopping a container:\n1 2 3 4 5 6 7 8 9 10 11 func stopContainer(dockerClient *client.Client, containerID string) error { ctx := context.Background() timeout := time.Second * 10 err := dockerClient.ContainerStop(ctx, containerID, \u0026amp;timeout) if err != nil { return err } return nil } Removing a container:\n1 2 3 4 5 6 7 8 9 10 11 12 13 func removeContainer(dockerClient *client.Client, containerID string) error { ctx := context.Background() options := types.ContainerRemoveOptions{ Force: true, } err := dockerClient.ContainerRemove(ctx, containerID, options) if err != nil { return err } return nil } In the above code snippets, dockerClient.ContainerStart(), dockerClient.ContainerStop(), and dockerClient.ContainerRemove() are used to perform the respective operations on a specific container. You provide the container ID and any additional options as parameters.\nInteracting with Docker Networks # Creating custom Docker networks in Go # When working with Docker, networks play a crucial role in connecting containers and enabling communication between them. With the Docker Go SDK, you can programmatically create custom Docker networks. Here\u0026rsquo;s an example:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 func createNetwork(dockerClient *client.Client, networkName string) error { ctx := context.Background() options := types.NetworkCreate{ CheckDuplicate: true, Driver: \u0026#34;bridge\u0026#34;, Attachable: true, } _, err := dockerClient.NetworkCreate(ctx, networkName, options) if err != nil { return err } return nil } In the code snippet above, dockerClient.NetworkCreate() is used to create a custom Docker network. You provide the network name and network configuration options such as the driver and attachable properties. The function returns an error if the network creation fails.\nBy programmatically creating custom networks, you have fine-grained control over the network configuration, enabling you to tailor the network settings to your specific application requirements.\nConnecting containers to networks programmatically # Connecting containers to Docker networks is an essential aspect of containerization. The Docker Go SDK allows you to connect containers to networks programmatically. Here\u0026rsquo;s an example:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 func connectContainerToNetwork(dockerClient *client.Client, containerID string, networkID string) error { ctx := context.Background() options := types.NetworkConnectOptions{ Container: containerID, NetworkID: networkID, } err := dockerClient.NetworkConnect(ctx, networkID, containerID, options) if err != nil { return err } return nil } In the code snippet above, dockerClient.NetworkConnect() is used to connect a container to a Docker network. You provide the container ID, network ID, and any additional options required for the connection. The function returns an error if the connection process encounters any issues.\nBy programmatically connecting containers to networks, you can establish network communication between containers, enabling them to interact and share data. This gives you the flexibility to dynamically configure and manage network connections based on your application\u0026rsquo;s needs.\nRetrieving and Streaming Docker Logs # Fetching container logs using Docker Go SDK # Fetching container logs is a common operation when working with Docker, as logs provide valuable information for debugging, monitoring, and troubleshooting containerized applications. With the Docker Go SDK, you can easily retrieve container logs programmatically. Here\u0026rsquo;s an example:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 func fetchContainerLogs(dockerClient *client.Client, containerID string) (io.ReadCloser, error) { ctx := context.Background() options := types.ContainerLogsOptions{ ShowStdout: true, ShowStderr: true, } logStream, err := dockerClient.ContainerLogs(ctx, containerID, options) if err != nil { return nil, err } return logStream, nil } In the code snippet above, we use the dockerClient.ContainerLogs() method provided by the Docker Go SDK to fetch the logs of a specific container. We pass the container ID and set the ShowStdout and ShowStderr options to true to include the standard output and standard error streams in the log output. The function returns an io.ReadCloser interface, which allows you to read the log data from the container.\nBy leveraging the Docker Go SDK\u0026rsquo;s ContainerLogs() method, you can programmatically retrieve container logs, enabling you to analyze and process log information as needed.\nStreaming real-time logs from running containers # In addition to fetching container logs, the Docker Go SDK also provides functionality to stream real-time logs from running containers. Streaming logs in real-time is particularly useful for monitoring applications and diagnosing issues as they occur. Here\u0026rsquo;s an example:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 func streamContainerLogs(dockerClient *client.Client, containerID string) error { ctx := context.Background() options := types.ContainerLogsOptions{ ShowStdout: true, ShowStderr: true, Follow: true, } logStream, err := dockerClient.ContainerLogs(ctx, containerID, options) if err != nil { return err } defer logStream.Close() // Process log data from the logStream in real-time return nil } In the code snippet above, we use the ContainerLogs() method with the Follow option set to true. This instructs the Docker Go SDK to stream the container\u0026rsquo;s logs in real-time. The log data can be processed and analyzed from the returned io.ReadCloser interface.\nBy leveraging the real-time log streaming capability of the Docker Go SDK, you can monitor the logs of running containers and react promptly to events, such as errors or important log messages. This enables you to gain real-time insights into your containerized applications and facilitates effective debugging and troubleshooting.\nManaging Docker Volumes # Creating and managing Docker volumes programmatically # Docker volumes are a powerful feature that allows data to persist beyond the lifecycle of a container. With the Docker Go SDK, you can easily create and manage volumes programmatically. Here\u0026rsquo;s an example:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 func createDockerVolume(dockerClient *client.Client, volumeName string) error { ctx := context.Background() volumeOptions := types.VolumeCreateOptions{ Name: volumeName, } _, err := dockerClient.VolumeCreate(ctx, volumeOptions) if err != nil { return err } return nil } In the code snippet above, we use the VolumeCreate() method provided by the Docker Go SDK to create a Docker volume. We specify the name of the volume through the Name field in the VolumeCreateOptions struct. The function returns an error if the volume creation process encounters any issues.\nBy leveraging the Docker Go SDK\u0026rsquo;s volume management capabilities, you can programmatically create, inspect, and remove Docker volumes, enabling you to manage your data persistence requirements efficiently.\nMounting volumes to containers using Go SDK # Mounting volumes to containers allows you to share data between the host machine and the container or between multiple containers. With the Docker Go SDK, you can easily mount volumes programmatically. Here\u0026rsquo;s an example:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 func mountVolumeToContainer(dockerClient *client.Client, containerID string, volumeName string, mountPath string) error { ctx := context.Background() containerMounts := []mount.Mount{ { Type: mount.TypeVolume, Source: volumeName, Target: mountPath, }, } mountOptions := types.ContainerMountOptions{ Mounts: containerMounts, } err := dockerClient.ContainerMount(ctx, containerID, mountOptions) if err != nil { return err } return nil } In the code snippet above, we use the ContainerMount() method provided by the Docker Go SDK to mount a volume to a container. We specify the volume name, target mount path, and the type of mount (in this case, a volume mount) through the ContainerMountOptions struct. The function returns an error if the volume mounting process encounters any issues.\nBy leveraging the Docker Go SDK\u0026rsquo;s volume mounting capabilities, you can programmatically control the sharing and persistence of data between containers and the host system, allowing for seamless integration and management of data in your containerized environment.\nError Handling and Best Practices # Handling errors and exceptions in Docker Go SDK # When working with the Docker Go SDK, it is essential to handle errors and exceptions effectively to ensure the robustness and reliability of your application. Here are some best practices for handling errors:\nUse proper error handling techniques: Implement appropriate error handling techniques, such as using if statements or error return values, to handle errors returned by Docker Go SDK functions.\nCheck for specific errors: The Docker Go SDK provides specific error types that you can check for, such as dockerapi.ErrNotFound for resource not found errors or dockerapi.ErrConflict for conflicts. By checking for specific error types, you can handle them accordingly in your code.\nLog and report errors: Logging and reporting errors is crucial for troubleshooting and debugging purposes. Use a logging library or framework to log errors, including relevant information such as the operation being performed, the affected resource, and any relevant context.\nGraceful error handling: Gracefully handle errors by considering possible recovery actions, such as retrying the operation, providing fallback behavior, or gracefully terminating the application if necessary.\nBy following these error handling practices, you can ensure that your Docker Go SDK-based application handles errors effectively and provides a better user experience.\nBest practices and tips for using Docker Go SDK effectively # To make the most out of the Docker Go SDK and ensure efficient and reliable operations, consider the following best practices and tips:\nUnderstand Docker concepts: Familiarize yourself with Docker concepts, such as images, containers, networks, and volumes, as the Docker Go SDK closely aligns with these concepts. Understanding these concepts will help you use the SDK effectively and make informed decisions.\nLeverage the official Docker documentation: The Docker Go SDK is continuously evolving, and the official Docker documentation provides detailed information about the available methods, options, and usage patterns. Refer to the documentation to understand the functionalities and stay updated on any changes or new features.\nFollow best practices for security: Apply security best practices, such as using secure authentication methods when interacting with Docker, securely storing sensitive information, and ensuring appropriate access controls for Docker resources.\nConsider performance implications: Be mindful of the performance implications of using the Docker Go SDK. For example, avoid unnecessary or redundant API calls, batch operations when possible, and optimize resource allocation and usage.\nTest and validate your code: Thoroughly test your code that utilizes the Docker Go SDK. Write unit tests, integration tests, and perform comprehensive validation to ensure the correctness and reliability of your code.\nLeverage community resources: The Docker community is active and vibrant, with various forums, discussion boards, and GitHub repositories where you can find examples, share knowledge, and seek assistance if needed. Leverage these community resources to expand your understanding and get insights from experienced users.\nBy following these best practices and tips, you can use the Docker Go SDK effectively, develop robust applications, and ensure smooth interactions with Docker resources.\nConclusion # To further expand your knowledge and skills in using the Docker Go SDK, consider the following next steps:\nExplore the Docker Go SDK documentation: Dive deeper into the capabilities and usage of the Docker Go SDK by referring to the official Docker documentation. It provides detailed information about the available methods, options, and usage patterns.\nExperiment with additional Docker operations: The Docker Go SDK offers a wide range of functionalities beyond what we covered in this blog post. Take the opportunity to explore other operations, such as container inspection, image manipulation, network management, and more.\nJoin Docker community forums and discussions: Engage with the Docker community through forums, discussion boards, and social media platforms. Connect with fellow developers, ask questions, share your experiences, and learn from others\u0026rsquo; insights and use cases.\nBuild sample applications: Practice your skills by building small applications or scripts that leverage the Docker Go SDK. Experiment with different scenarios, such as deploying multi-container applications, automating Docker tasks, or integrating Docker with other technologies.\nExplore related tools and libraries: The Docker ecosystem offers various tools and libraries that can complement your Docker Go SDK knowledge. Look into tools like Docker Compose for defining and managing multi-container applications or explore other Docker SDKs available in different programming languages.\nBy delving deeper into the Docker Go SDK and continuing your learning journey, you can unlock even more possibilities for building robust and efficient applications that interact with Docker programmatically.\nIn conclusion, the Docker Go SDK empowers developers to interact with Docker seamlessly. By recapping the covered topics and exploring further resources, you are now equipped to continue exploring and leveraging the Docker Go SDK in your own projects. Happy coding!\n","date":"7 August 2023","externalUrl":null,"permalink":"/posts/2023/unlocking-dockers-power-with-go-a-developers-perspective-part-2/","section":"Posts","summary":"In this blog post we continue our last post for most advanced use cases. We talk about managing docker containers in which we talk about spinning a container, listing, starting, stopping, and removing container. We talk about using docker network. We talk about streaming and retrieving logs from docker container. We also cover docker working with docker volumes using Go. At last, we talk about error handling and best practices.\n","title":"Unlocking Docker's Power With Go: A Developer's Perspective - Part 2","type":"posts"},{"content":" Introduction # Not until recently, I was compelled to learn iptables when I was working with Firecracker VM. iptables is a tool to manipulate network packets.\nIptables is a powerful and versatile firewall tool that is used to protect and secure networks. It is an open-source program that is installed on Linux-based operating systems. Iptables works by inspecting and filtering network traffic based on a set of rules. These rules define what traffic is allowed and what is blocked, based on criteria such as the source and destination IP address, port number, and protocol.\nToday I\u0026rsquo;ll show you how and what I have learned iptables to manipulate those packets.\nUnderstanding the Jargon # There are 3 constructs which we should keep in mind while working with iptables. This will help later on when we are actually writing commands on the terminal. These constructs are:\nTables Chains Rules Tables # Tables themselves are of 5 types. 3 of which are mostly used.\nFilter Table This is the default tables. Meaning that if on command line if you don\u0026rsquo;t specify any table, program will assume filter table.\nRole of this table is to filter packets based on criteria such as the source and destination IP addresses, port numbers, and protocols.\nThis table contains three built-in chains - Input, Forward, and Output - that are used to process incoming, forwarded, and outgoing packets, respectively.\nNAT Table Role of this table is to modify destination or source headers in order to route packet in NAT setup where direct access is not possible. NAT Table is used to translate IP addresses and/or port numbers in packets as they pass through the firewall.\nThis table contains three built-in chains - PREROUTING, POSTROUTING, and OUTPUT - that are used to modify packets before and after routing.\nMangle Table The Mangle Table is used to modify packets in ways that are more complex than those allowed by the Filter and NAT Tables. This table can be used to alter packet header information, mark packets for special handling, and perform other advanced packet processing tasks.\nThe Mangle Table contains five built-in chains - PREROUTING, OUTPUT, INPUT, FORWARD, and POSTROUTING - that allow packets to be processed at various stages of their journey through the firewall.\nRaw Table Used for connection tracking.\nSecurity Table SELinux policy is applied on the packets.\nThe last two tables I have not studied deeply so I haven\u0026rsquo;t much to talk about it. You may explore more about it on your own.\nChains # Chains are like points in route of a packet where you can apply rules. There are 5 chains in iptables.\nList of chains are as follows:\nPre-routing Chain Pre-routing chain is applied to any incoming packet very soon after entring the network. This chain is processed before any routing descision have been made regarding where to send the packet. It is typically used to modify the destination IP address of incoming packets, such as when implementing port forwarding.\nInput Chain This chain comes after the pre-routing chain. The Input Chain is responsible for processing incoming packets that are destined for the local machine. This chain is typically used to enforce firewall rules that dictate which incoming packets should be accepted or rejected.\nForward Chain The Forward Chain is responsible for processing packets that are being forwarded from one network interface to another. This chain is typically used to enforce firewall rules that dictate which packets should be allowed to pass through the machine.\nOutput Chain Output chain is applied to packet originating or going out from the system. This chain is typically used to enforce firewall rules that dictate which packets should be allowed to leave the machine.\nPost-routing Chain Opposite of pre-routing. The Post-routing Chain is responsible for performing any necessary modifications to outgoing packets after they have been routed. This chain is typically used to modify the source IP address of outgoing packets, such as when implementing network address translation (NAT).\nUnderstanding the purpose of each chain is important for effectively configuring iptables to enforce the desired network security policies. By creating and configuring rules within each chain, network administrators can control how traffic flows through their networks and help to prevent security threats.\nAll chains are not available for all tables.\nRules # Rules is basically an entry of condition, when matches, a certain action is done. Suppose you can to DROP all the packets originating from say 101.101.101.101. You can have a rule in the filter table. Iptables rules define what traffic is allowed or blocked based on the source and destination IP address, port number, and protocol.\nEach packet is checked against each rule.\nRules are basically row inside the tables we described above.\nBasic iptables Command # Command Syntax # The basic syntax for iptables commands is as follows:\niptables [options] \u0026lt;chain\u0026gt; \u0026lt;rule\u0026gt;\nThe \u0026ldquo;chain\u0026rdquo; argument specifies which chain the rule should be added to, and the \u0026ldquo;rule\u0026rdquo; argument specifies the rule to be added. Options can be used to modify the behavior of the command.\nExamples of Basic iptables Commands # Here are some examples of basic iptables commands that you can use to manage your firewall rules:\nAdd a Rule To add a rule to a chain, use the \u0026ldquo;-A\u0026rdquo; option followed by the chain name and the rule. For example, to allow incoming traffic on port 80 (HTTP), you can use the following command:\niptables -A INPUT -p tcp --dport 80 -j ACCEPT\nThis adds a rule to the \u0026ldquo;INPUT\u0026rdquo; chain that allows incoming traffic on port 80.\nDelete a Rule To delete a rule from a chain, use the \u0026ldquo;-D\u0026rdquo; option followed by the chain name and the rule. For example, to delete the rule that we added in the previous example, you can use the following command:\niptables -D INPUT -p tcp --dport 80 -j ACCEPT\nThis removes the rule from the \u0026ldquo;INPUT\u0026rdquo; chain that allowed incoming traffic on port 80.\nList Rules To list the rules in a chain, use the \u0026ldquo;-L\u0026rdquo; option followed by the chain name. For example, to list the rules in the \u0026ldquo;INPUT\u0026rdquo; chain, you can use the following command:\niptables -L INPUT\nThis lists the rules in the \u0026ldquo;INPUT\u0026rdquo; chain, including the rule we added in the previous example (if it is still present).\nClear Rules To clear all the rules from a chain, use the \u0026ldquo;-F\u0026rdquo; option followed by the chain name. For example, to clear all the rules from the \u0026ldquo;INPUT\u0026rdquo; chain, you can use the following command:\niptables -F INPUT\nThis removes all the rules from the \u0026ldquo;INPUT\u0026rdquo; chain, effectively disabling the firewall for incoming traffic.\nSave Rules To save your iptables rules so that they persist across reboots, use the \u0026ldquo;iptables-save\u0026rdquo; command. For example, to save the current iptables rules to a file called \u0026ldquo;iptables-rules\u0026rdquo;, you can use the following command:\niptables-save \u0026gt; iptables-rules\nThis saves the current iptables rules to a file called \u0026ldquo;iptables-rules\u0026rdquo; in the current directory.\nRestore Rules To restore your saved iptables rules, use the \u0026ldquo;iptables-restore\u0026rdquo; command. For example, to restore the iptables rules from the \u0026ldquo;iptables-rules\u0026rdquo; file that we created in the previous example, you can use the following command:\niptables-restore \u0026lt; iptables-rules\nThis restores the iptables rules from the \u0026ldquo;iptables-rules\u0026rdquo; file, effectively restoring your firewall configuration.\nConclusion # In today\u0026rsquo;s post we learned about what iptables is and why is it important.\nWe learned about basic jargons which will get us started with more advanced workflow with the tool.\nAt last, we did some hands-on with the iptables command.\nIn next post, we\u0026rsquo;ll talk about more advanced usecase of iptables along with some common pitfalls.\n","date":"10 July 2023","externalUrl":null,"permalink":"/posts/2023/iptables-explained-an-introduction-to-linux-firewalling/","section":"Posts","summary":"Introduction # Not until recently, I was compelled to learn iptables when I was working with Firecracker VM. iptables is a tool to manipulate network packets.\nIptables is a powerful and versatile firewall tool that is used to protect and secure networks. It is an open-source program that is installed on Linux-based operating systems. Iptables works by inspecting and filtering network traffic based on a set of rules. These rules define what traffic is allowed and what is blocked, based on criteria such as the source and destination IP address, port number, and protocol.\n","title":"iptables Explained: An Introduction to Linux Firewalling","type":"posts"},{"content":" Introduction to Docker Go SDK # In the introductory section of the blog post, we will provide a comprehensive introduction to the Docker Go SDK, highlighting its purpose, benefits, and relevance in the context of building Docker applications using the Go programming language.\nBrief explanation of Docker Go SDK and its benefits # The Docker Go SDK, also known as the Docker Engine API for Go, is a powerful toolkit that allows developers to interact with Docker and perform Docker-related operations using the Go programming language. It provides a set of APIs and functions that enable seamless integration with Docker, empowering developers to manage Docker resources, automate tasks, and build Docker-centric applications.\nThe Docker Go SDK offers several benefits for developers working with Docker:\nProgrammatic Control: With the Docker Go SDK, developers have fine-grained control over Docker operations. They can create, manage, and manipulate Docker containers, images, networks, and volumes programmatically, using the power and flexibility of the Go language.\nNative Integration: The Docker Go SDK is designed to work seamlessly with Go applications, leveraging the language\u0026rsquo;s strengths in terms of simplicity, performance, and concurrency. It provides idiomatic Go interfaces and constructs, making it natural to use for Go developers.\nAutomation and Customization: By leveraging the Docker Go SDK, developers can automate Docker-related tasks and workflows. They can build custom tooling, pipelines, and automation scripts that interact with Docker programmatically, tailored to their specific requirements and use cases.\nExtensibility: The Docker Go SDK allows developers to extend Docker\u0026rsquo;s capabilities and build additional layers of abstraction on top of Docker. It provides a foundation for creating higher-level frameworks, libraries, and tools that simplify Docker usage and enable advanced Docker-based applications.\nWhy Go and Docker Go SDK are a Powerful Combination # In this blog post, we will delve into the practical usage of the Docker Go SDK, exploring its various features and demonstrating how to leverage its power to build Docker applications. Here is an overview of the topics covered:\nIntroduction to Docker Go SDK: We\u0026rsquo;ll provide an introduction to the Docker Go SDK, explaining its purpose, benefits, and its compatibility with the Go ecosystem.\nInstalling Docker Go SDK: Step-by-step instructions for installing the Docker Go SDK on your development machine, ensuring you have all the necessary prerequisites.\nEstablishing Connection with Docker: We\u0026rsquo;ll cover how to establish a connection with Docker using the Docker Go SDK, including importing the required packages, creating a Docker client object, and handling authentication and connection options.\nWorking with Docker Images: This section will focus on image-related operations using the Docker Go SDK, including pulling images, listing available images, and programmatically building and tagging custom Docker images.\nManaging Docker Containers: We\u0026rsquo;ll explore how to work with Docker containers using the Docker Go SDK, covering tasks such as spinning up containers, listing running containers, and programmatically starting, stopping, and removing containers.\nInteracting with Docker Networks: This topic will cover creating custom Docker networks in Go and connecting containers to networks programmatically using the Docker Go SDK.\nRetrieving and Streaming Docker Logs: We\u0026rsquo;ll demonstrate how to fetch container logs and stream real-time logs from running containers using the Docker Go SDK.\nManaging Docker Volumes: This section will explain how to create and manage Docker volumes programmatically and how to mount volumes to containers using the Docker Go SDK.\nError Handling and Best Practices: We\u0026rsquo;ll discuss error handling strategies and best practices for using the Docker Go SDK effectively, ensuring robust and reliable Docker operations in your applications.\nConclusion and Next Steps: We\u0026rsquo;ll wrap up the blog post with a recap of the covered topics, highlighting key takeaways. Additionally, we\u0026rsquo;ll provide suggestions for further exploration, including relevant resources and references to continue learning about Docker Go SDK.\nInstalling Docker Go SDK # Instructions for installing Docker Go SDK # Installing the Docker Go SDK is a straightforward process. Here are the step-by-step instructions to install the Docker Go SDK on your development machine:\nEnsure Go is Installed: Before installing the Docker Go SDK, make sure you have Go installed on your system. Visit the official Go website to download and install the Go binary suitable for your operating system.\nSet Up Go Environment Variables: Configure the necessary environment variables for Go, such as GOPATH and PATH, to ensure the Go binaries are accessible from the command line.\nInstall Docker Go SDK Package: Open your terminal or command prompt and run the following command to install the Docker Go SDK package using the go get command:\n1 go get -u github.com/docker/docker This command fetches the Docker Go SDK package from the official GitHub repository and installs it in your Go workspace.\nVerifying the installation and prerequisites # Once the Docker Go SDK is installed, it\u0026rsquo;s essential to verify the installation and ensure that you have all the prerequisites in place. Here\u0026rsquo;s how you can do it:\nCompilation Check: Create a new Go file (e.g., main.go) and import the Docker Go SDK package at the top of the file: 1 2 3 import ( \u0026#34;github.com/docker/docker/client\u0026#34; ) Save the file and try to compile it using the following command:\n1 go build main.go If the compilation succeeds without any errors, it indicates that the Docker Go SDK package is installed correctly.\nPrerequisite Check: The Docker Go SDK has a few prerequisites that need to be met for proper functioning. Ensure the following prerequisites are fulfilled: Docker Engine: Make sure you have Docker Engine installed and running on your machine. The Docker Go SDK interacts with the Docker Engine to perform Docker operations. if your system does not have Docker Engine installed, I\u0026rsquo;d direct you towards Docker\u0026rsquo;s official installation documentation.\nDocker API Version Compatibility: Verify that the version of the installed Docker Go SDK package is compatible with the Docker API version of your Docker Engine. The Docker Go SDK package may require a specific Docker API version to function correctly.\nDocker Engine Access: Ensure that the user account running your Go application has sufficient permissions to access and interact with the Docker Engine. Permissions may vary based on the operating system and the setup of your Docker environment. If you are using Linux, there are some post installation steps you need to follow for this.\nBy verifying the installation and prerequisites, you can confirm that the Docker Go SDK is installed correctly and ready to be used for Docker-related operations in your Go applications.\nNote: It\u0026rsquo;s important to regularly update the Docker Go SDK package to benefit from bug fixes, performance improvements, and new features. You can use the go get command with the -u flag to update the Docker Go SDK package to the latest version:\n1 go get -u github.com/docker/docker By following these installation instructions and verifying the prerequisites, you\u0026rsquo;ll have the Docker Go SDK up and running, ready to be utilized in your Go applications to interact with Docker and perform Docker-related tasks programmatically.\nEstablishing Connection with Docker # Importing necessary packages and dependencies # To establish a connection with Docker using the Docker Go SDK, you need to import the necessary packages and dependencies into your Go application. These packages provide the essential functionality required to interact with Docker. Here are the packages you need to import:\n1 2 3 4 5 import ( \u0026#34;context\u0026#34; \u0026#34;github.com/docker/docker/api/types\u0026#34; \u0026#34;github.com/docker/docker/client\u0026#34; ) context: This package provides the context for managing the lifecycle of Docker operations. It allows you to control timeouts, cancellation, and deadlines for Docker requests.\ngithub.com/docker/docker/api/types: This package contains the types and structs used in the Docker Go SDK. It provides the necessary data structures to interact with Docker resources such as containers, images, networks, and volumes.\ngithub.com/docker/docker/client: This package is the core component of the Docker Go SDK. It provides the client implementation for connecting to Docker and executing Docker operations.\nCreating a Docker client object # Once the required packages are imported, you can create a Docker client object to establish a connection with Docker. The Docker client acts as the interface between your Go application and the Docker Engine. Here\u0026rsquo;s how you can create a Docker client object:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 func createDockerClient() (*client.Client, error) { // Create a new Docker client with default configuration dockerClient, err := client.NewClientWithOpts(client.FromEnv) if err != nil { return nil, err } // Ping Docker to verify the connection _, err = dockerClient.Ping(context.Background()) if err != nil { return nil, err } return dockerClient, nil } In the code snippet above, client.NewClientWithOpts(client.FromEnv) creates a new Docker client object using the default configuration, which includes reading the Docker-related environment variables (e.g., DOCKER_HOST, DOCKER_CERT_PATH, DOCKER_TLS_VERIFY) to establish the connection.\nThe dockerClient.Ping() function is used to send a simple request to Docker to verify the connection. If the connection is successful, the function will return without any errors.\nHandling authentication and connection options: # The Docker Go SDK provides various options for handling authentication and connection configurations. Some common scenarios include using TLS certificates for secure communication, specifying a custom Docker API version, or setting connection timeouts. Here are a few examples:\nCustom Docker API Version: 1 dockerClient, err := client.NewClientWithOpts(client.WithVersion(\u0026#34;1.41\u0026#34;)) Connection Timeout:\n1 2 3 timeout := time.Second * 30 dockerClient, err := client.NewClientWithOpts(client.WithTimeout(timeout)) By utilizing these options, you can customize the authentication and connection configurations based on your Docker environment\u0026rsquo;s requirements.\nEstablishing a connection with Docker is a crucial step in leveraging the Docker Go SDK. By importing the necessary packages, creating a Docker client object, and handling authentication and connection options, you can ensure a successful connection between your Go application and the Docker Engine, enabling you to perform Docker operations programmatically.\nWorking with Docker Images # Pulling Docker images using Go SDK # One of the fundamental tasks when working with Docker is pulling images from Docker registries. The Docker Go SDK provides a convenient way to accomplish this programmatically. Here\u0026rsquo;s how you can pull Docker images using the Go SDK:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 func pullImage(dockerClient *client.Client, imageName string) error { ctx := context.Background() options := types.ImagePullOptions{} reader, err := dockerClient.ImagePull(ctx, imageName, options) if err != nil { return err } defer reader.Close() // Read the pull progress and handle it as needed // ... return nil } In the code snippet above, dockerClient.ImagePull() is used to initiate the image pull operation. You provide the image name or tag to pull and any additional pull options if required. The function returns a reader that allows you to monitor the progress of the pull operation.\nYou can then process the reader to retrieve the pull progress, handle errors, or display the progress to the user as desired. Remember to close the reader once you have finished processing it.\nListing available images # Listing the available images on your Docker host is another common task. The Docker Go SDK provides a simple way to retrieve this information programmatically. Here\u0026rsquo;s an example:\n1 2 3 4 5 6 7 8 9 10 func listImages(dockerClient *client.Client) ([]types.ImageSummary, error) { ctx := context.Background() images, err := dockerClient.ImageList(ctx, types.ImageListOptions{}) if err != nil { return nil, err } return images, nil } In the above code snippet, dockerClient.ImageList() is used to retrieve a list of image summaries. The types.ImageListOptions{} parameter allows you to specify filters or additional options for the image listing if needed. The function returns a slice of types.ImageSummary containing information about each available image.\nYou can then process the retrieved image list, extract relevant information such as the image ID or tags, and perform further operations based on your requirements.\nBuilding and tagging custom Docker images programmatically # Building custom Docker images programmatically using the Docker Go SDK provides flexibility and automation capabilities. Here\u0026rsquo;s an example of how you can achieve this:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 func buildImage(dockerClient *client.Client, buildContext string, imageName string) error { ctx := context.Background() options := types.ImageBuildOptions{ Context: buildContext, Dockerfile: \u0026#34;Dockerfile\u0026#34;, Tags: []string{imageName}, } response, err := dockerClient.ImageBuild(ctx, nil, options) if err != nil { return err } defer response.Body.Close() // Read and process the build output // ... return nil } In the above code snippet, dockerClient.ImageBuild() is used to initiate the image build process. You provide the build context path (directory containing the Dockerfile), the Dockerfile name, and the desired tags for the image.\nThe function returns a response object that allows you to access the build output. You can read and process the output to display build progress, handle errors, or extract relevant information.\nBy leveraging the Docker Go SDK, you can pull Docker images, list available images, and build custom images programmatically. These capabilities enable you to automate image management tasks within your Go applications and integrate them seamlessly with Docker workflows.\nI will have to end this blog post here only. I will continue on this article in next post. You can follow update on this series here: Go Docker SDK.\n","date":"22 June 2023","externalUrl":null,"permalink":"/posts/2023/unlocking-dockers-power-with-go-a-developers-perspective-part-1/","section":"Posts","summary":"Introduction to Docker Go SDK # In the introductory section of the blog post, we will provide a comprehensive introduction to the Docker Go SDK, highlighting its purpose, benefits, and relevance in the context of building Docker applications using the Go programming language.\nBrief explanation of Docker Go SDK and its benefits # The Docker Go SDK, also known as the Docker Engine API for Go, is a powerful toolkit that allows developers to interact with Docker and perform Docker-related operations using the Go programming language. It provides a set of APIs and functions that enable seamless integration with Docker, empowering developers to manage Docker resources, automate tasks, and build Docker-centric applications.\n","title":"Unlocking Docker's Power With Go: A Developer's Perspective - Part 1","type":"posts"},{"content":"","date":"30 May 2023","externalUrl":null,"permalink":"/tags/golang/","section":"Tags","summary":"","title":"Golang","type":"tags"},{"content":" Introduction # Testify is a popular testing toolkit for the Go programming language. It provides a wide range of assertion functions, test suite support, and mocking capabilities, making it a powerful tool for testing Go applications. Testify aims to simplify the process of writing tests and improve the quality of test coverage in Go applications.\nTesting is a critical aspect of software development that helps ensure that the software behaves as intended and catches bugs early in the development process. Writing tests can be time-consuming, but it is essential for delivering a reliable and maintainable software system. Testify can help make testing in Go more efficient and effective.\nTestify offers a variety of benefits for Go developers:\nEasy to use: Testify provides an intuitive and easy-to-use API for writing tests and assertions. Wide range of assertions: Testify provides a large number of built-in assertions, which helps developers write thorough and comprehensive tests. Test suite support: Testify supports test suites, which allows developers to group tests together for easier management. Mocking capabilities: Testify provides a powerful mocking framework, making it easy to create and use mock objects in tests. Integration with popular testing tools: Testify integrates with popular testing tools like GoConvey and Ginkgo, making it easy to incorporate into existing testing workflows. Getting Started with Testify # Before using Testify, you need to install it. You can use the following command to install Testify using the Go module system:\n1 go get github.com/stretchr/testify To create a test file with Testify, you can create a new file with the \u0026ldquo;_test.go\u0026rdquo; suffix as we usually do in Go development, and import the \u0026ldquo;testing\u0026rdquo; and \u0026ldquo;github.com/stretchr/testify/assert\u0026rdquo; packages. Here\u0026rsquo;s an example:\n1 2 3 4 5 6 7 8 9 10 11 12 13 package main_test import ( \u0026#34;testing\u0026#34; \u0026#34;github.com/stretchr/testify/assert\u0026#34; ) func TestAddition(t *testing.T) { result := 2 + 2 assert.Equal(t, 4, result, \u0026#34;The result should be 4\u0026#34;) } In this example, we have created a test function named \u0026ldquo;TestAddition\u0026rdquo;. We are using the \u0026ldquo;assert\u0026rdquo; package from Testify to check if the result of the addition operation is equal to 4. But you will obviously be testing your business logic.\nTestify provides a wide range of assertion functions that you can use in your test functions. Here are some examples:\n1 2 3 4 5 6 7 8 9 10 func TestAssertions(t *testing.T) { assert.Equal(t, 1, 1, \u0026#34;1 should be equal to 1\u0026#34;) assert.NotEqual(t, 1, 2, \u0026#34;1 should not be equal to 2\u0026#34;) assert.Nil(t, nil, \u0026#34;Nil should be equal to nil\u0026#34;) assert.NotNil(t, \u0026#34;test\u0026#34;, \u0026#34;The string should not be nil\u0026#34;) assert.True(t, true, \u0026#34;True should be true\u0026#34;) assert.False(t, false, \u0026#34;False should be false\u0026#34;) assert.Empty(t, \u0026#34;\u0026#34;, \u0026#34;The string should be empty\u0026#34;) assert.NotEmpty(t, \u0026#34;test\u0026#34;, \u0026#34;The string should not be empty\u0026#34;) } Testify Assertions # Testify provides a large number of assertion functions that you can use to check the expected behavior of your code in tests. These functions are simple to use and provide detailed error messages when assertions fail.\nAs we mentioned in previous section, here are some commonly used assertions provided by Testify:\nassert.Equal: checks if two values are equal. assert.NotEqual: checks if two values are not equal. assert.Nil: checks if a value is nil. assert.NotNil: checks if a value is not nil. assert.True: checks if a value is true. assert.False: checks if a value is false. assert.Empty: checks if a value is empty. assert.NotEmpty: checks if a value is not empty. Refer to previous section on how to use them.\nTestify also provides a mechanism for creating custom assertions if the built-in assertions are not enough. You can create a custom assertion by defining a function that takes a testing.TB interface, an expected value, and an actual value, and returns a boolean indicating whether the assertion passed or failed. Here is an example:\n1 2 3 4 5 6 7 8 9 10 func assertGreaterThan(t testing.TB, expected, actual int) bool { if actual \u0026gt; expected { return true } return false } func TestCustomAssertions(t *testing.T) { assert.Condition(t, func() bool { return assertGreaterThan(t, 5, 10) }, \u0026#34;The value should be greater than 5\u0026#34;) } In this example, we define a custom assertion function named \u0026ldquo;assertGreaterThan\u0026rdquo; that checks if the actual value is greater than the expected value. We then use this custom assertion function in a test function using Testify\u0026rsquo;s \u0026ldquo;assert.Condition\u0026rdquo; function.\nTestify Mocking # Testify provides a mocking framework that allows you to create mock objects for testing. Mock objects are objects that simulate the behavior of real objects in a controlled way, making it easier to test your code in isolation. Testify\u0026rsquo;s mocking framework is based on Go interfaces and provides an easy-to-use API for creating and using mock objects.\nHere is an example of creating and using a mock object with Testify:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 type DB interface { Get(id string) (string, error) Put(id string, value string) error } type MockDB struct { mock.Mock } func (m *MockDB) Get(id string) (string, error) { args := m.Called(id) return args.String(0), args.Error(1) } func (m *MockDB) Put(id string, value string) error { args := m.Called(id, value) return args.Error(0) } func TestMocking(t *testing.T) { // Create a new instance of the mock DB mockDB := new(MockDB) // Define the expected behavior of the Get method mockDB.On(\u0026#34;Get\u0026#34;, \u0026#34;test-id\u0026#34;).Return(\u0026#34;test-value\u0026#34;, nil) // Define the expected behavior of the Put method mockDB.On(\u0026#34;Put\u0026#34;, \u0026#34;test-id\u0026#34;, \u0026#34;test-value\u0026#34;).Return(nil) // Call the Get method value, err := mockDB.Get(\u0026#34;test-id\u0026#34;) assert.NoError(t, err) assert.Equal(t, \u0026#34;test-value\u0026#34;, value) // Call the Put method err = mockDB.Put(\u0026#34;test-id\u0026#34;, \u0026#34;test-value\u0026#34;) assert.NoError(t, err) // Check that the expected methods were called mockDB.AssertExpectations(t) } In this example, we define an interface for a database with two methods, Get and Put. We then define a mock implementation of this interface using Testify\u0026rsquo;s Mock type. The mock implementation overrides the Get and Put methods and uses Testify\u0026rsquo;s mock.Mock type to define the expected behavior of these methods.\nIn the TestMocking function, we create a new instance of the mock DB and define the expected behavior of the Get and Put methods using the On method. We then call these methods and check that they return the expected values. Finally, we use Testify\u0026rsquo;s AssertExpectations method to check that the expected methods were called.\nBest practices for using Testify mocking # Use mocking sparingly\nMocking is a powerful technique for testing code in isolation, but it can also be overused. Mock objects should only be used when necessary, such as when testing code that has external dependencies or when testing code that is difficult to set up in a test environment.\nMock only what is necessary\nWhen using mocking, it\u0026rsquo;s important to only mock what is necessary for the test. Over-mocking can lead to brittle tests that break easily when the implementation changes. Only mock the behavior that is required for the test to pass.\nKeep mocks simple\nMocks should be simple and easy to understand. Avoid creating overly complex mock objects that are difficult to maintain or that obscure the intent of the test.\nAvoid using global mocks\nGlobal mock objects can make it difficult to understand the behavior of the test and can lead to unexpected behavior when running tests in parallel. Avoid using global mock objects and instead create new mock objects for each test.\nUse the AssertExpectations method\nTestify\u0026rsquo;s AssertExpectations method can be used to check that the expected methods were called on the mock object. Always use this method to ensure that the expected behavior was observed during the test.\nUse callback functions for complex mock behavior\nWhen defining complex mock behavior, use callback functions to define the behavior of the mock object. This can make it easier to understand the behavior of the mock and can simplify the code.\nUse AfterTest to clean up mocks\nTestify\u0026rsquo;s AfterTest method can be used to clean up any resources created during the test, such as mock objects. Always use this method to ensure that your tests are clean and don\u0026rsquo;t leak resources. For example:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 func TestMyTest(t *testing.T) { // Create a new instance of the mock object mockObj := new(MockObject) // Define the expected behavior of the mock object mockObj.On(\u0026#34;Method\u0026#34;, \u0026#34;arg\u0026#34;).Return(\u0026#34;result\u0026#34;) // Call the method being tested result := myFunc(mockObj) // Verify the expected behavior of the mock object mockObj.AssertExpectations(t) // Clean up the mock object after the test t.Cleanup(func() { mockObj.AssertExpectations(t) }) } In this example, we define a Cleanup function using Testify\u0026rsquo;s t.Cleanup method. This function ensures that the expected behavior of the mock object is verified again after the test is complete, using the AssertExpectations method. This ensures that the mock object is cleaned up properly after the test.\nTestify Suites and Setup/Teardown # Testify provides a mechanism for grouping related tests into suites. Testify suites allow you to define setup and teardown functions that are called before and after the tests in the suite are run. This can be useful for setting up common test data or resources that are required for multiple tests in the suite.\nUsing setup and teardown functions # Testify allows you to define setup and teardown functions for each test, as well as for the suite as a whole. These functions can be used to set up test data, initialize resources, or perform any other necessary setup or teardown operations.\nHere\u0026rsquo;s an example of defining setup and teardown functions for a single test:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 func TestMyTest(t *testing.T) { // Define the setup function setup := func() { // Set up test data or resources } // Define the teardown function teardown := func() { // Clean up test data or resources } // Call the test with setup and teardown functions t.Run(\u0026#34;MyTest\u0026#34;, func(t *testing.T) { setup() defer teardown() // Test code goes here }) } In this example, we define a setup function that sets up test data or resources, and a teardown function that cleans up test data or resources. We then call the test using the t.Run method, which allows us to pass in the setup and teardown functions. The defer keyword is used to ensure that the teardown function is always called, even if the test fails.\nHere\u0026rsquo;s an example of defining setup and teardown functions for a Testify suite:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 func TestSuite(t *testing.T) { // Define the suite suite := testify.NewSuite() // Define the setup function for the suite suite.SetupSuite(func() { // Set up resources required by all tests in the suite }) // Define the teardown function for the suite suite.TearDownSuite(func() { // Clean up resources used by all tests in the suite }) // Add tests to the suite suite.Run(t, \u0026#34;MyTestSuite\u0026#34;, testify.New(testMyFunc), testify.New(testAnotherFunc), // ... ) } In this example, we define a Testify suite using the testify.NewSuite method. We then define the setup and teardown functions for the suite using the SetupSuite and TearDownSuite methods. Finally, we add tests to the suite using the Run method.\nExamples of using suites and setup/teardown # Example 1: Testing an HTTP server\nSuppose you have an HTTP server that you want to test. You can use a Testify suite to group related tests, and setup and teardown functions to start and stop the server before and after the tests.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 func TestHTTPServer(t *testing.T) { // Define the suite suite := testify.NewSuite() // Define the setup function for the suite suite.SetupSuite(func() { // Start the server server.Start() }) // Define the teardown function for the suite suite.TearDownSuite(func() { // Stop the server server.Stop() }) // Add tests to the suite suite.Run(t, \u0026#34;HTTPServerTests\u0026#34;, testify.New(testHTTPGet), testify.New(testHTTPPost), // ... ) } func testHTTPGet(t *testing.T) { // Test the HTTP GET method // ... } func testHTTPPost(t *testing.T) { // Test the HTTP POST method // ... } In this example, we define a Testify suite to group tests related to an HTTP server. We use the SetupSuite function to start the server before the tests and the TearDownSuite function to stop the server after the tests. We then add tests to the suite using the Run function.\nExample 2: Testing a database application\nSuppose you have a database application that you want to test. You can use a Testify suite to group related tests, and setup and teardown functions to set up and tear down the database before and after the tests.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 func TestDatabase(t *testing.T) { // Define the suite suite := testify.NewSuite() // Define the setup function for the suite suite.SetupSuite(func() { // Set up the database db.Setup() }) // Define the teardown function for the suite suite.TearDownSuite(func() { // Tear down the database db.TearDown() }) // Add tests to the suite suite.Run(t, \u0026#34;DatabaseTests\u0026#34;, testify.New(testInsert), testify.New(testUpdate), // ... ) } func testInsert(t *testing.T) { // Test inserting data into the database // ... } func testUpdate(t *testing.T) { // Test updating data in the database // ... } In this example, we define a Testify suite to group tests related to a database application. We use the SetupSuite function to set up the database before the tests and the TearDownSuite function to tear down the database after the tests. We then add tests to the suite using the Run function.\nThese are just a few examples of how Testify suites and setup/teardown functions can be used to organize and simplify your tests. By using these features, you can write more maintainable and robust tests for your Go applications.\nConclusion # In this blog post, we have covered the key features and benefits of Testify, a popular testing toolkit for Go. Testify provides a wide range of assertion functions and mocking capabilities that make it easier to write robust and maintainable tests for your Go applications. Some of the key benefits of using Testify include:\nA rich set of assertion functions: Testify provides a large number of assertion functions that cover a wide range of use cases. These functions are easy to use and can help you write more comprehensive tests with less code.\nSupport for mocking: Testify provides a powerful mocking framework that makes it easier to test complex code that relies on external dependencies. By using Testify mocking, you can isolate your code from its dependencies and write more focused tests.\nEasy to learn and use: Testify has a simple and intuitive API that makes it easy to get started with testing in Go. The toolkit is well-documented and there are many examples available online to help you learn how to use it effectively.\nIntegration with popular testing frameworks: Testify can be easily integrated with popular testing frameworks like Go\u0026rsquo;s built-in testing package, making it a versatile choice for testing Go applications.\nIf you\u0026rsquo;re interested in using Testify to test your Go applications, here are some next steps you can take:\nRead the official Testify documentation: The official Testify documentation provides a comprehensive guide to using the toolkit, including detailed explanations of its features and API.\nExplore the Testify examples: There are many examples of Testify in action available online, including on the official Testify GitHub repository. Reviewing these examples can help you understand how Testify can be used to test real-world Go applications.\nPractice writing tests with Testify: The best way to learn how to use Testify is to practice writing tests with it. Start by writing some simple tests and gradually increase their complexity as you become more comfortable with the toolkit.\nContribute to the Testify project: If you find Testify useful and want to help improve it, consider contributing to the project on GitHub. You can contribute code, report bugs, or help improve the documentation to make Testify even more useful for the Go community.\nBy taking these steps, you can learn how to use Testify to write more effective tests for your Go applications and become a more skilled and effective Go developer.\n","date":"30 May 2023","externalUrl":null,"permalink":"/posts/2023/improving-code-quality-with-testify-in-go-a-deep-dive-into-testing/","section":"Posts","summary":"Introduction # Testify is a popular testing toolkit for the Go programming language. It provides a wide range of assertion functions, test suite support, and mocking capabilities, making it a powerful tool for testing Go applications. Testify aims to simplify the process of writing tests and improve the quality of test coverage in Go applications.\nTesting is a critical aspect of software development that helps ensure that the software behaves as intended and catches bugs early in the development process. Writing tests can be time-consuming, but it is essential for delivering a reliable and maintainable software system. Testify can help make testing in Go more efficient and effective.\n","title":"Improving Code Quality with Testify in Go: A Deep Dive into Testing","type":"posts"},{"content":"","date":"30 May 2023","externalUrl":null,"permalink":"/tags/tdd/","section":"Tags","summary":"","title":"Tdd","type":"tags"},{"content":"","date":"30 May 2023","externalUrl":null,"permalink":"/tags/testing/","section":"Tags","summary":"","title":"Testing","type":"tags"},{"content":"","date":"10 May 2023","externalUrl":null,"permalink":"/tags/http/","section":"Tags","summary":"","title":"Http","type":"tags"},{"content":" Introduction # HTTP response headers provide additional information about the server\u0026rsquo;s response to an HTTP request. These headers are essential for web developers and server administrators to ensure efficient and secure communication between clients and servers. In this blog post, we will discuss some of the most common HTTP response headers and provide example values for each header.\nAn response header example in Chrome Response Headers # Content-Type # This header specifies the MIME type of the content being sent in the response. The MIME type tells the browser or other client how to interpret the content. For example, if the Content-Type is set to \u0026ldquo;text/html\u0026rdquo;, the browser knows to display the response as an HTML document. Other common MIME types include \u0026ldquo;application/json\u0026rdquo; for JSON data and \u0026ldquo;image/png\u0026rdquo; for PNG images.\nExample 1: Content-Type: text/html\nExample 2: Content-Type: application/json\nExample 3: Content-Type: image/png\nCache-Control # This header specifies how the response should be cached. The value can include directives such as \u0026ldquo;public\u0026rdquo; to allow caching by any client, \u0026ldquo;private\u0026rdquo; to limit caching to the browser or other client, \u0026ldquo;no-cache\u0026rdquo; to indicate that the response should not be cached, and \u0026ldquo;max-age\u0026rdquo; to specify how long the response can be cached.\nExample 1: Cache-Control: max-age=3600, public\nExample 2: Cache-Control: no-cache\nExample 3: Cache-Control: must-revalidate\nLocation # This header is used to redirect the client to a different URL. The client should automatically follow the URL in the Location header. This is commonly used for HTTP redirects, such as when a user submits a form and is redirected to a confirmation page.\nExample 1: Location: https://www.example.com/newpage\nExample 2: Location: /error.html\nExample 3: Location: https://www.example.com/redirected\nSet-Cookie # This header is used to set a cookie in the client\u0026rsquo;s browser. Cookies are small pieces of data that can be used to store information about the user or their session. The Set-Cookie header can include a name, value, expiration date, and other parameters.\nExample 1: Set-Cookie: sessionId=12345; Expires=Thu, 01 Jan 2024 00:00:00 GMT\nExample 2: Set-Cookie: userId=67890; Secure; HttpOnly\nExample 3: Set-Cookie: preference=darkmode; Domain=example.com\nServer # This header identifies the web server software being used to serve the response. The value of the header typically includes the name and version of the server software.\nExample 1: Server: Apache/2.4.41 (Unix)\nExample 2: Server: Microsoft-IIS/10.0\nExample 3: Server: nginx/1.21.0\nExpires # This header specifies the date and time after which the response should be considered stale. If a client requests the same resource before the expiration date has passed, the server can return a cached copy of the resource.\nExample 1: Expires: Sat, 08 May 2023 12:00:00 GMT\nExample 2: Expires: 0\nExample 3: Expires: Thu, 01 Jan 1970 00:00:00 GMT\nContent-Length # This header specifies the length of the content being sent in the response. This can be useful for the client to know how much data to expect and for the server to know how much data to send. When you are downloading a file from internet, the broweser looks at this header to determine how big the file is.\nExample 1: Content-Length: 1024\nExample 2: Content-Length: 2048\nExample 3: Content-Length: 4096\nLast-Modified # This header specifies the date and time when the content being sent in the response was last modified. This can be used by clients to determine if the resource has changed since it was last requested.\nExample 1: Last-Modified: Tue, 04 May 2023 16:00:00 GMT\nExample 2: Last-Modified: Wed, 03 May 2023 10:30:00 GMT\nExample 3: Last-Modified: Mon, 01 May 2023 00:00:00 GMT\nETag # This header provides a unique identifier for the content being sent in the response. This can be used by clients to determine if the resource has changed since it was last requested, without having to download the entire resource again.\nExample 1: ETag: \u0026quot;abc123\u0026quot;\nExample 2: ETag: \u0026quot;xyz789\u0026quot;\nExample 3: ETag: \u0026quot;efg456\u0026quot;\nAccess-Control-Allow-Origin # This header is used to specify which domains are allowed to access the resource being requested, in the case of cross-origin requests. This can help prevent malicious scripts from accessing resources that they should not have access to.\nExample 1: Access-Control-Allow-Origin: https://www.example.com\nExample 2: Access-Control-Allow-Origin: *\nExample 3: Access-Control-Allow-Origin: https://api.example.com\nX-Frame-Options # This header is used to prevent clickjacking attacks by restricting which sites are allowed to embed the page in an iframe. The value of the header can include \u0026ldquo;deny\u0026rdquo; to prevent embedding on any site, \u0026ldquo;sameorigin\u0026rdquo; to allow embedding only on the same domain, or a specific domain to allow embedding on that domain only.\nExample 1: X-Frame-Options: SAMEORIGIN\nExample 2: X-Frame-Options: DENY\nExample 3: X-Frame-Options: ALLOW-FROM https://www.example.com\nX-XSS-Protection # This header is used to enable or disable cross-site scripting (XSS) protection in the client\u0026rsquo;s browser. The value of the header can include \u0026ldquo;1\u0026rdquo; to enable protection or \u0026ldquo;0\u0026rdquo; to disable it.\nExample 1: X-XSS-Protection: 1; mode=block\nExample 2: X-XSS-Protection: 0\nExample 3: X-XSS-Protection: 1; report=https://www.example.com/report\nReferrer-Policy # This header is used to control how much information is sent in the referrer header when a user clicks on a link. The value of the header can include \u0026ldquo;no-referrer\u0026rdquo; to send no referrer information, \u0026ldquo;same-origin\u0026rdquo; to send referrer information only for same-origin requests, or \u0026ldquo;strict-origin\u0026rdquo; to send referrer information only for requests to the same domain.\nExample 1: Referrer-Policy: strict-origin-when-cross-origin\nExample 2: Referrer-Policy: no-referrer\nExample 3: Referrer-Policy: origin\nX-Forwarded-For # This header is used to identify the original IP address of a client that is connecting to a web server through a proxy or load balancer. When a request passes through a proxy or load balancer, the IP address of the client is usually replaced with the IP address of the proxy or load balancer. By setting this header, the proxy or load balancer can add the original IP address of the client to the request headers, allowing the server to identify the client.\nExample 1: X-Forwarded-For: 192.168.1.1\nExample 2: X-Forwarded-For: 192.168.1.1, 10.0.0.1\nExample 3: X-Forwarded-For: 192.168.1.1, 10.0.0.1, 172.16.0.1\nX-Powered-By # This header identifies the technology stack or software used to generate the response. While this header can provide useful information for developers and system administrators, it can also be a security risk if it reveals sensitive information about the server. For this reason, it is often recommended to remove this header or obfuscate its value.\nExample 1: X-Powered-By: PHP/7.4.16\nExample 2: X-Powered-By: ASP.NET\nExample 3: X-Powered-By: Express\nKeep-Alive # This header is used to enable persistent connections between the client and the server, allowing multiple requests and responses to be sent over a single TCP connection. By keeping the connection open, the overhead of establishing a new connection for each request can be avoided, improving performance. The header value can specify the maximum number of requests that can be sent over a single connection or the maximum amount of time that a connection can be idle before it is closed.\nExample 1: Keep-Alive: timeout=5, max=100\nExample 2: Keep-Alive: timeout=10, max=50\nExample 3: Keep-Alive: timeout=15, max=200\nConclusion # In conclusion, HTTP response headers provide valuable information about the data being sent in the response from the server to the client. By understanding these headers and\n","date":"10 May 2023","externalUrl":null,"permalink":"/posts/2023/most-common-http-response-headers-with-example/","section":"Posts","summary":"Introduction # HTTP response headers provide additional information about the server’s response to an HTTP request. These headers are essential for web developers and server administrators to ensure efficient and secure communication between clients and servers. In this blog post, we will discuss some of the most common HTTP response headers and provide example values for each header.\n","title":"Most Common HTTP Response Header with Examples","type":"posts"},{"content":"","date":"10 May 2023","externalUrl":null,"permalink":"/tags/rest/","section":"Tags","summary":"","title":"Rest","type":"tags"},{"content":"","date":"25 April 2023","externalUrl":null,"permalink":"/tags/ffmpeg/","section":"Tags","summary":"","title":"Ffmpeg","type":"tags"},{"content":" Introduction # Okay, so I have encountered ffmpeg in my early career already. And now, although I have got away from that industry, I still love to do some physics simulation for refreshment.\nIf you have worked with 3D software before, you might know that we render the output in image sequence or simply called frames. To do anything with my simulation, I need to convert that sequence of images into a video.\nWhat is ffmpeg? # ffmpeg is one of the industry standard transcoder. According to Wikipedia:\nFFmpeg is a free and open-source software project consisting of a suite of libraries and programs for handling video, audio, and other multimedia files and streams. At its core is the command-line ffmpeg tool itself, designed for processing of video and audio files. It is widely used for format transcoding, basic editing (trimming and concatenation), video scaling, video post-production effects and standards compliance (SMPTE, ITU).\nLet\u0026rsquo;s not get into jargon now. Just understand that we can use ffmpeg to convert from one codec to another. We can also use this package to convert from sequence to video.\nBasic commands to get comfortable # 1. See metadata about any video/audio/image: # $ ffmpeg -i story.mp4 It will throw an output something like this:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 ffmpeg version 4.4.1-3ubuntu5 Copyright (c) 2000-2021 the FFmpeg developers built with gcc 11 (Ubuntu 11.2.0-18ubuntu1) configuration: --prefix=/usr --extra-version=3ubuntu5 --toolchain=hardened --libdir=/usr/lib/x86_64-linux-gnu --incdir=/usr/include/x86_64-linux-gnu --arch=amd64 --enable-gpl --disable-stripping --enable-gnutls --enable-ladspa --enable-libaom --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --enable-libcdio --enable-libcodec2 --enable-libdav1d --enable-libflite --enable-libfontconfig --enable-libfreetype --enable-libfribidi --enable-libgme --enable-libgsm --enable-libjack --enable-libmp3lame --enable-libmysofa --enable-libopenjpeg --enable-libopenmpt --enable-libopus --enable-libpulse --enable-librabbitmq --enable-librubberband --enable-libshine --enable-libsnappy --enable-libsoxr --enable-libspeex --enable-libsrt --enable-libssh --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx265 --enable-libxml2 --enable-libxvid --enable-libzimg --enable-libzmq --enable-libzvbi --enable-lv2 --enable-omx --enable-openal --enable-opencl --enable-opengl --enable-sdl2 --enable-pocketsphinx --enable-librsvg --enable-libmfx --enable-libdc1394 --enable-libdrm --enable-libiec61883 --enable-chromaprint --enable-frei0r --enable-libx264 --enable-shared libavutil 56. 70.100 / 56. 70.100 libavcodec 58.134.100 / 58.134.100 libavformat 58. 76.100 / 58. 76.100 libavdevice 58. 13.100 / 58. 13.100 libavfilter 7.110.100 / 7.110.100 libswscale 5. 9.100 / 5. 9.100 libswresample 3. 9.100 / 3. 9.100 libpostproc 55. 9.100 / 55. 9.100 Input #0, mov,mp4,m4a,3gp,3g2,mj2, from \u0026#39;/home/santosh/Videos/249774011_924906985088194_8434993645291029887_n.mp4\u0026#39;: Metadata: major_brand : isom minor_version : 512 compatible_brands: isomiso2avc1mp41 encoder : Lavf58.20.100 Duration: 00:00:13.78, start: 0.000000, bitrate: 797 kb/s Stream #0:0(und): Video: h264 (Constrained Baseline) (avc1 / 0x31637661), yuv420p(tv, bt709), 480x854, 744 kb/s, 30 fps, 30 tbr, 15360 tbn, 60 tbc (default) Metadata: handler_name : VideoHandler vendor_id : [0][0][0][0] Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 48 kb/s (default) Metadata: handler_name : SoundHandler vendor_id : [0][0][0][0] At least one output file must be specified If you want to avoid the initial metadata about the ffmpeg itself, you can do so by appending -hide_banner to the command. I actually like to create an alias for it in my ~/.bashrc file.\n$ ffmpeg -i story.mp4 -hide_banner 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Input #0, mov,mp4,m4a,3gp,3g2,mj2, from \u0026#39;/home/santosh/Videos/249774011_924906985088194_8434993645291029887_n.mp4\u0026#39;: Metadata: major_brand : isom minor_version : 512 compatible_brands: isomiso2avc1mp41 encoder : Lavf58.20.100 Duration: 00:00:13.78, start: 0.000000, bitrate: 797 kb/s Stream #0:0(und): Video: h264 (Constrained Baseline) (avc1 / 0x31637661), yuv420p(tv, bt709), 480x854, 744 kb/s, 30 fps, 30 tbr, 15360 tbn, 60 tbc (default) Metadata: handler_name : VideoHandler vendor_id : [0][0][0][0] Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 48 kb/s (default) Metadata: handler_name : SoundHandler vendor_id : [0][0][0][0] At least one output file must be specified In the output you can see the video has one video stream, and one audio stream. The video is encoded using h264 and audio using aac.\nFor video, you can also see that its resolution is 480x854 and bps is 744. Also, it\u0026rsquo;s a 30fps video. The audio is 48000 Hz, and it\u0026rsquo;s a stereo audio.\n2. Convert video from one format to another # You can convert from one format to another by specifying. For more granularity, you can even specify the encoders to use.\nffmpeg -i story.mp4 -c:v copy -c:a libvorbis story.avi In above example, we are taking story.mp4, we have to convert it into story.avi. Let\u0026rsquo;s interpret other two parameters. -c:v copy -c:a libvorbis actually can be split into 2 parts.\n-c:v is used to specify codec for video. We specify copy which means it will copy same video as it is in source. And for audio, we are using libvorbis.\nYou can find the entire list on the ffmpeg codecs documentation.\n3. Separate audio/video from a video # There could be a moment where you would want to extract audio from a video. E.g. audio from a video song from YouTube. With ffmpeg you can achieve that with following command:\nffmpeg -i video.mp4 -vn audio.mp3 Now for -vn, you can take it memorize it as video no.\nSimilarly, you can do a following for an audio no. The resultant video will have no audio.\nffmpeg -i video.mp4 -an audio.mp4 4. Convert video into sequence of images # ffmpeg -i video.mp4 image%d.jpg This is the simplest kind of conversion. In the start of the command, as always, we take the video input. For output, we output in format of image.jpg. But we should also note that the resultant will be a sequence of images, not a single image. We put a %d in middle of sequence name and the file format. In this case, the image names will go from image1.jpg to imageN.jpg.\nYou can modify the image name to be following:\nffmpeg -i video.mp4 image%4d.jpg The resultant image will now start with image0001.jpg. Not that it is padded with leading 0.\nIf you want to take this further, you can also do this:\nffmpeg -i video.mp4 -r 24 -f image2 image-%4d.png The -r 24 here is the frame rate. Without this, ffmpeg is going to assume frame rate to be 30 fps. The image format is specified with -f image2.\nI frankly don\u0026rsquo;t know what image2 is at the moment. But I anticipate it is very common with jpg/png input and output.\n5. Cut media files # If you want to 1 minute chunk from middle of a 6 minutes video, you should consider something like this:\nffmpeg -i video.mp4 -ss 00:03:00 -to 00:04:00 video_out.mp4 The resultant video will start from 3rd minute and will go till 4th minute of the source video.\n6. Record screen/webcam/audio # You can capture all of those with ffmpeg, yes.\n6.1 Let\u0026rsquo;s start with the screen recording first # Simplest way to capture your screen is to:\nffmpeg -f x11grab -s 1600x900 -i :0.0 output.mp4 Here, the encoder is x11grab. -s is the screen size, in my case it is 1600x900. The the input is quite cryptic here. :0.0 actually means my first monitor. So for those who are only having 1 monitor setup, this command would work. For those with\nAdvance version of this would be to capture only a part of the screen. For that, obviously you\u0026rsquo;d have to lower down the resolution. And secondly, you\u0026rsquo;d want to append the top left coordinate of the screen. So the input part would look like this -i :0.0+100,200. It will take top left point to be 100 in x, and 200 in y. And say if you have passed -s 500x500, it will add 500 in both of x and y. That will be bottom right corner of the record.\n6.2 Capture audio # Most basic command.\nffmpeg -f alsa -i default output.mp3 But what if you want to capture screen, as well as audio?\nffmpeg -f x11grab -s 1600x900 -i :0.0 -f alsa -ac 2 -i default output.mkv The -ac above is audio channel, which is set to be 2.\n6.3 Capture webcam # Capturing webcam is quite simpler.\nffmpeg -i /dev/video0 output.mkv This simply captures the webcam, which is /dev/video0 in Linux.\nBut what I noticed in my case it, the default setting is the it does not captures full resolution which my webcam supports. For that, I had to use the following command.\nffmpeg -f v4l2 -framerate 24 -video_size 1280x720 -i /dev/video0 webcam.mkv Here I\u0026rsquo;m explicitly setting the -video_size. -framerate or -r is also set. I think it would already would have been 24 by default. I have also specified the encoder, i.e. v4l2. By the way I haven\u0026rsquo;t seen any difference without it.\n7. Convert sequence of images into video/gif # cat my_photos/* | ffmpeg -framerate 24 -f image2pipe -i - -c:v copy video.mkv If I simply drop down the framerate enough, I will be making a slideshow.\nConclusion # FFmpeg is a free and open-source command-line tool used for processing and converting multimedia files. It can handle various types of audio, video, and image formats and can perform tasks such as converting one format to another, resizing, cropping, adding subtitles, and more. Essentially, it\u0026rsquo;s a powerful tool for working with media files.\nFFmpeg is backend to many application. If you are using any GUI such as for video editing, or screen recording. Then there is a good chance that you are ffmpeg in backend.\n","date":"25 April 2023","externalUrl":null,"permalink":"/posts/2023/introduction-to-ffmpeg-by-basic-common-examples/","section":"Posts","summary":"Introduction # Okay, so I have encountered ffmpeg in my early career already. And now, although I have got away from that industry, I still love to do some physics simulation for refreshment.\nIf you have worked with 3D software before, you might know that we render the output in image sequence or simply called frames. To do anything with my simulation, I need to convert that sequence of images into a video.\n","title":"Introduction to FFmpeg by Basic Common Examples","type":"posts"},{"content":"","date":"25 April 2023","externalUrl":null,"permalink":"/tags/transcoding/","section":"Tags","summary":"","title":"Transcoding","type":"tags"},{"content":"","date":"25 April 2023","externalUrl":null,"permalink":"/tags/video-editing/","section":"Tags","summary":"","title":"Video-Editing","type":"tags"},{"content":"","date":"7 April 2023","externalUrl":null,"permalink":"/tags/esp8266/","section":"Tags","summary":"","title":"Esp8266","type":"tags"},{"content":" Introduction # At the time I\u0026rsquo;m writing this post, the Internet of Things is not a new thing. If you own a vehicle, and it connects to the internet in some manner, your vehicle manufacturer is already gathering so much data with planted sensors in the vehicle.\nIt\u0026rsquo;s not only about your vehicle or your phone. IoT is used in tracking the orders you have booked at your favorite e-commerce site. You as an enthusiast can also automate certain aspects of your house.\nAutomating my home was my motivation to get started start digging around IoT. I own a raspberry pi already. But Raspberry is best suited for being the central server of your entire house. It\u0026rsquo;s more like a mini-computer.\nWhile you can learn to work with sensors and actuators on a Raspberry Pi, there will come a time when you\u0026rsquo;ll have to explore MCUs aka micro-controllers. ESP8266 is one such popular MCUs in the market. It is popular among IoT enthusiasts and is pretty cheap compared to raspi. These ESPs are a good compliment to raspis. It is developed by Espressif Systems.\nOne of the most popular selling points of ESP8266 is its connectivity. It comes with a WiFi chip onboard. So where does NodeMCU fits in? NodeMCU is a firmware for ESP8266. It is analogous to the operating system on a fully-fledged computer.\nFrom now onwards, I am assuming you have an ESP8266 and you have a computer running Ubuntu already.\nDownload Arduino IDE # Now the first step to download Arduino IDE is to head over to https://www.arduino.cc/en/software.\nAt the moment I\u0026rsquo;m writing this, I can see two options:\nLinux AppImage 64 bits (X86-64)\nLinux ZIP file 64 bits (X86-64)\nClick on the first option. You will be taken to the donate page where you can see a link to download the AppImage file.\nSpin up your terminal and navigate to the download directory.\nRename the file to arduino-ide.\nMake the file executable.\nMove it to ~/.local/bin.\nHere is me doing all the above steps in the terminal:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 santosh at kubuntu in ~ $ cd ~/Downloads/ santosh at kubuntu in ~/Downloads $ ls -l *appimage -rw-rw-r-- 1 santosh santosh 203484377 Mar 31 00:42 arduino-ide_2.0.4_Linux_64bit.appimage santosh at kubuntu in ~/Downloads $ mv arduino-ide_2.0.4_Linux_64bit.appimage arduino-ide renamed \u0026#39;arduino-ide_2.0.4_Linux_64bit.appimage\u0026#39; -\u0026gt; \u0026#39;arduino-ide\u0026#39; santosh at kubuntu in ~/Downloads $ chmod +x arduino-ide santosh at kubuntu in ~/Downloads $ mv arduino-ide ~/.local/bin/ renamed \u0026#39;arduino-ide\u0026#39; -\u0026gt; \u0026#39;/home/santosh/.local/bin/arduino-ide\u0026#39; santosh at kubuntu in ~/Downloads $ ls -l ~/.local/bin/arduino-ide -rwxrwxr-x 1 santosh santosh 203484377 Mar 31 00:43 /home/santosh/.local/bin/arduino-ide* You should be able to execute arduino-ide from the terminal. If at this point you are not able to call the executable, you might have to update your path. Stick the following line inside your ~/.bashrc file. Your shell rc file will be different if you are using another shell such as zsh or fish.\n1 PATH=$HOME/.local/bin:$PATH Configure Arduino IDE to work with ESP8266 # As the name \u0026lsquo;Arduino IDE\u0026rsquo; suggests, it was originally an IDE for Arduino boards. But the community has grown and it now supported more than Arduino boards. This section is about configuring Arduino IDE to make it aware of the ESP board we are going to use with it.\nWe do so by going to File \u0026gt; Preferences. And then add the following URL in the textarea labeled as Additional boards manager URLs. Hit OK when you are done.\n1 http://arduino.esp8266.com/stable/package_esp8266com_index.json The above URL is not the actual firmware, instead, it is a list of all firmware that is based on ESP8266.\nAdditonal Boards Manager The next step is to install the actual firmware.\nYou can do this by going to: Tools \u0026gt; Boards Manager \u0026gt; Search 8266 \u0026gt; Install\nConfigure your system to work with USB # Let\u0026rsquo;s take a step back to configure ubuntu properly. It was a problem for me, so there is a good chance that it will be a problem for you too.\nYour current user needs to be in the dialout group, otherwise uploading the code to NodeMCU will fail. On the next line is the command to add your logged-in user to the dialout group.\n1 sudo usermod -a -G dialout $USER When you are done running that command, make sure you log out and log in again.\nHello World # This blog post is incomplete without a hello world sketch. By the way, a sketch is a code or program which you upload to the MCU. Let\u0026rsquo;s do our legendary project, the hello world.\nAt this point, you should plug your MCU into the USB on your system.\nNext, you will be seeing a \u0026ldquo;Select Board\u0026rdquo; dropdown. Click on that and type NodeMCU. Click OK. Select the MCU to upload the code to.\nHere is a sketch that will print hello world at 5 seconds intervals.\n1 2 3 4 5 6 7 8 void setup() { Serial.begin(115200); // Set the baud rate to 115200, same as the software settings } void loop() { Serial.println(\u0026#34;Hello World!\u0026#34;); // Print character string \u0026#34;Hello World！\u0026#34; on the serial delay(5000); // Delay 5 second } Conventionally, there are two functions in each sketch. setup is run initially when the chip is powered on. The loop keeps running infinitely until and unless power is still on.\nClick the Upload button and while the code is sent to the MCU, open up the Serial Monitor from the Tools menu.\nEnjoy the hello world program.\nConclusion # This is the start of something big. There is a whole lot that can be done with ESP8266. In upcoming posts, I\u0026rsquo;ll be coming up with exciting projects which are fun to do on this mini-board.\n","date":"7 April 2023","externalUrl":null,"permalink":"/posts/2023/getting-started-with-nodemcu-development-on-ubuntu/","section":"Posts","summary":"Introduction # At the time I’m writing this post, the Internet of Things is not a new thing. If you own a vehicle, and it connects to the internet in some manner, your vehicle manufacturer is already gathering so much data with planted sensors in the vehicle.\nIt’s not only about your vehicle or your phone. IoT is used in tracking the orders you have booked at your favorite e-commerce site. You as an enthusiast can also automate certain aspects of your house.\n","title":"Getting Started With NodeMCU Development on Ubuntu","type":"posts"},{"content":"","date":"7 April 2023","externalUrl":null,"permalink":"/tags/home-automation/","section":"Tags","summary":"","title":"Home-Automation","type":"tags"},{"content":"","date":"7 April 2023","externalUrl":null,"permalink":"/tags/iot/","section":"Tags","summary":"","title":"Iot","type":"tags"},{"content":"","date":"7 April 2023","externalUrl":null,"permalink":"/tags/nodemcu/","section":"Tags","summary":"","title":"Nodemcu","type":"tags"},{"content":"","date":"19 March 2023","externalUrl":null,"permalink":"/tags/asgi/","section":"Tags","summary":"","title":"Asgi","type":"tags"},{"content":"","date":"19 March 2023","externalUrl":null,"permalink":"/tags/nginx/","section":"Tags","summary":"","title":"Nginx","type":"tags"},{"content":"","date":"19 March 2023","externalUrl":null,"permalink":"/tags/python/","section":"Tags","summary":"","title":"Python","type":"tags"},{"content":" Introduction # When it comes to deploying Python web applications, a common architecture involves using a web server like Nginx to handle incoming requests, while the Python application itself runs on a separate server. In this architecture, the web server serves as a reverse proxy, forwarding requests to the Python application server and then returning the response back to the client.\nThe communication between the web server and the application server is handled by a protocol that specifies how the two servers should interact with each other. In the case of Python web applications, this protocol is either WSGI (Web Server Gateway Interface) or ASGI (Asynchronous Server Gateway Interface), depending on whether the application uses synchronous or asynchronous programming.\nSo, what is the role of the WSGI/ASGI layer when we have Nginx? Let\u0026rsquo;s take a closer look.\nWSGI # WSGI is a specification that defines how a web server should communicate with a Python application server. It specifies a simple interface for handling HTTP requests and responses, allowing developers to write web applications in Python without having to worry about the details of the underlying server.\nWhen using Nginx with a WSGI application, Nginx acts as a reverse proxy, forwarding incoming requests to the WSGI server, which then processes the request and generates a response. The WSGI server sends the response back to Nginx, which in turn returns it to the client.\nThe WSGI layer provides a simple and standardized interface for Python web applications, making it easy to write and deploy web applications that can be run on a variety of web servers.\nASGI # ASGI is a newer specification that is designed to support asynchronous programming in Python web applications. Unlike WSGI, which is designed for synchronous programming, ASGI allows developers to write web applications that can handle multiple requests at once, without blocking the main thread.\nWhen using Nginx with an ASGI application, Nginx acts as a reverse proxy, forwarding incoming requests to the ASGI server, which then processes the request and generates a response. Because the ASGI server can handle multiple requests at once, it can process incoming requests asynchronously, allowing it to handle more traffic than a traditional synchronous web application.\nThe ASGI layer provides a powerful and flexible interface for Python web applications, making it possible to write high-performance, scalable web applications that can handle large amounts of traffic.\nWhy use WSGI/ASGI when we have nginx? Why can\u0026rsquo;t python application directly communicate to nginx? # Nginx is a popular web server that is often used to serve static content, reverse proxy requests, and load balance traffic. While Nginx can handle HTTP requests and responses efficiently, it is not designed to interact directly with Python applications. This is where the Web Server Gateway Interface (WSGI) and Asynchronous Server Gateway Interface (ASGI) come into play.\nWSGI and ASGI are Python specifications that define how web servers can communicate with web applications written in Python. They provide a standard interface between web servers and Python applications, allowing them to work together seamlessly.\nHere are a few reasons why using WSGI/ASGI with Nginx is preferred over direct communication between Nginx and Python applications:\nSeparation of Concerns: Nginx is optimized for serving static files and handling HTTP requests and responses, while Python is optimized for handling complex business logic. Separating the web server and application layers allows each to focus on what it does best.\nScalability: When a Python application is deployed behind Nginx, it can be scaled horizontally by adding more instances of the application. Nginx can then be configured to balance traffic between these instances, ensuring that requests are distributed evenly and efficiently.\nSecurity: Nginx provides a number of security features, such as SSL termination, rate limiting, and IP filtering. By deploying a Python application behind Nginx, these security features can be utilized to provide an additional layer of protection for the application.\nFlexibility: Using WSGI/ASGI allows Python applications to be run on any web server that supports the standard interface, not just Nginx. This gives developers the flexibility to choose the web server that best suits their needs, without having to worry about compatibility issues.\nIn summary, while Nginx is an excellent web server, it is not designed to interact directly with Python applications. Using WSGI/ASGI to interface between Nginx and Python applications provides a number of benefits, including separation of concerns, scalability, security, and flexibility.\nWhether you choose to use WSGI or ASGI depends on the requirements of your application. If you need to handle large amounts of traffic and want to take advantage of asynchronous programming, ASGI may be the best choice. If you\u0026rsquo;re writing a simpler application that doesn\u0026rsquo;t require asynchronous programming, WSGI may be the better option.\nConclusion # In summary, the role of the WSGI/ASGI layer when we have Nginx is to provide a standardized interface for communication between the web server and the Python application server. This allows developers to write web applications in Python without having to worry about the details of the underlying server, and makes it possible to deploy Python web applications on a variety of web servers.\n","date":"19 March 2023","externalUrl":null,"permalink":"/posts/2023/why-use-wsgi-asgi-when-we-have-nginx/","section":"Posts","summary":"Introduction # When it comes to deploying Python web applications, a common architecture involves using a web server like Nginx to handle incoming requests, while the Python application itself runs on a separate server. In this architecture, the web server serves as a reverse proxy, forwarding requests to the Python application server and then returning the response back to the client.\n","title":"Why Use WSGI/ASGI When We Have Nginx?","type":"posts"},{"content":"","date":"19 March 2023","externalUrl":null,"permalink":"/tags/wsgi/","section":"Tags","summary":"","title":"Wsgi","type":"tags"},{"content":"","date":"25 February 2023","externalUrl":null,"permalink":"/tags/observability/","section":"Tags","summary":"","title":"Observability","type":"tags"},{"content":"","date":"25 February 2023","externalUrl":null,"permalink":"/tags/opentelemetry/","section":"Tags","summary":"","title":"Opentelemetry","type":"tags"},{"content":"","date":"25 February 2023","externalUrl":null,"permalink":"/series/opentelemetry-for-first-timers/","section":"Series","summary":"","title":"OpenTelemetry for First Timers","type":"series"},{"content":" Introduction # Let us start by asking this question:\nWhat is observability and why do we need telemetry?\nAlthough observability and telemetry might sound like interchangeable words to use, observability is the whole thing that consists of instrumentation to viewing the metrics.\nHere telemetry is the instrumentation part. And in this post, we are majorly going to talk about this part only.\nIn my words, telemetry provides data about a running system which could be useful for taking more aware/educated decisions in the business. Telemetry enables software developers to proactively identify issues, understand the root cause of problems, and optimize performance in real time.\nLogs are the most basic kind of information we can gather from a running system if we have put proper logs statements in the codebase. The act of writing console.log or log.Println in the code to print out the state of the system can be called instrumentation too. In fact, logging is among the other 2 mechanisms using which we can instrument. We\u0026rsquo;ll use the word instrumentation a lot in this post. ;)\nLater in this post, we will learn about how logs are not the only type of information we can do analytics on.\nLet us first see what is OpenTelemetry.\nWhat is OpenTelemetry? # OpenTelemetry is the standard available for telemetry in software systems.\nAs per OpenTelemetry landing page\nOpenTelemetry is a collection of tools, APIs, and SDKs. Use it to instrument, generate, collect, and export telemetry data (metrics, logs, and traces) to help you analyze your software’s performance and behavior.\nA more formal introduction can be found at: https://opentelemetry.io/docs/concepts/what-is-opentelemetry/\nWhy OpenTelemetry?\nIt\u0026rsquo;s open source It\u0026rsquo;s a standard. And implementation is available in a wide range of programming languages. What to expect from this post\nI was not an OTel wizard when I started writing this post. OpenTelemetry has been a recent encounter for me at my workplace. I had heard about it previously but never had hands-on.\nThis article is nothing more than an introduction to OpenTelemetry from my perspective.\nBut what I could say is, if you have previously used AWS Cloudtrail, you might be able to relate.\nUnderstanding the Concepts # Telemetry refers to data collected from a running system. This data can come in these forms:\nLogging Metrics Tracing Logging: Logs are basically timestamped pieces of information emitted from a service. If you used logging for observation, there is a better alternative.\nMetrics: Metrics refer to the collection of data over a particular time. These data can include system error rate, CPU utilization, request rate, etc for a given service.\nTracing: Trace is the new Log. While logs can\u0026rsquo;t associate with any particular user request or transaction, traces can. Which is very useful for tracking code executions.\nThere are different concepts related to them so let\u0026rsquo;s see them.\nLogs # Data that is not part of Metrics or Traces are known as logs.\nIn the context of OTel, an Event is a type of log.\nMetrics # Metrics are important indicators of availability and performance. Collected data can be used to alert of an outage or trigger scheduling decisions to scale up a deployment automatically upon high demand.\nThis part is where you might be able to relate it to AWS Cloudtrail and ASG.\nFrankly speaking, I don\u0026rsquo;t have first-hand experience with Metrics on OTel. But when I ran down through the docs, I found some interesting information:\nOpenTelemetry defines three metric instruments today:\ncounter: a value that is summed over time – you can think of this like an odometer on a car; it only ever goes up. measure: a value that is aggregated over time. This is more akin to the trip odometer on a car, it represents a value over some defined range. observer: captures a current set of values at a particular point in time, like a fuel gauge in a vehicle. And then this:\nUnlike request tracing, which is intended to capture request lifecycles and provide context to the individual pieces of a request, metrics are intended to provide statistical information in the aggregate. Some examples of use cases for metrics include:\nReporting the total number of bytes read by a service, per protocol type. Reporting the total number of bytes read and the bytes per request. Reporting the duration of a system call. Reporting request sizes in order to determine a trend. Reporting CPU or memory usage of a process. Reporting average balance values from an account. Reporting current active requests being handled. You can read more about it here: https://opentelemetry.io/docs/concepts/signals/metrics/\nTraces # To understand what a trace is, imagine you have a system that has 3 microservices and when you make a request, the request goes through 2 of them depending on the flow of the app.\nThe idea of tracing is to be able to relate information about the execution of two different services/processes to a single entity. Because that is what those 3 services as a whole are; a single entity.\nTraces are where our main area of focus is today. Traces are also preferred over conventional logs as we have already read about them. Here are some questions to ask if you are new to the open telemetry world.\nTraces and Spans What is a Tracer? # Tracer is an object which stores information about a trace. Suppose you have a system that has 3 microservices and when you make a request, the request goes through 2 of them depending on the flow of the app. Although they are 2 different processes, Tracer will have info from both of them as part of a single trace.\nOne Tracer can have multiple Spans.\nTracers are created by TracerProviders.\nWhat is a TracerProvider? # TracerProvider is a factory for Tracer.\nTracer Provider initialization also includes Resource and Exporter initialization. It is typically the first step in tracing with OpenTelemetry.\nWhat is a Resource? # Resource describes the application being instrumented. It is used for identification which will later be used within the visualization phase.\nWhat is an Exporter? # Let\u0026rsquo;s talk about Exporter or Trace Exporters now.\nTo visualize and analyze your traces and metrics, you will need to export them to a backend.\nExporters are packages that allow telemetry data to be emitted somewhere - either to the console or to a remote system or collector for further analysis and/or enrichment. OpenTelemetry supports a variety of exporters through its ecosystem including popular open-source tools like Jaeger, Zipkin, Prometheus, and oltp exporter, which is understood by many community packages.\nWhat is a Span? # Taking from the example of 3 microservices, we made only one request to the entrypoint of the microservice, but different services got invoked. When a request reaches the first service, it realizes it has some dependency, and the trace id is passed to the second system as a Span.\nSpans are part of a Trace. One Trace can have multiple spans and one span can have other sub-spans. A Span can consist of execution time data, logs, and attributes. All of which can be configured with SDK.\nTraces and Spans If we are creating a span, we have to pass a context to the child span so that it can be visualized later in the observability pipeline.\nEach span can have multiple kinds of data associated with it. You can read more at Spans in OpenTelemetry.\nWhat is a Collector? # The main function of an OpenTelemetry Collector is to collect, process, and export telemetry data from applications and services. The collector serves as a central hub for collecting telemetry data from a variety of sources, including distributed tracing data, logs, and metrics.\nSome examples are Jeager, SigNoz, Prometheus, etc.\nInstrumenting OpenTelemetry # Instrumentation for most frameworks is available already. So you just need to modify your code at one place where the framework is loading instead of instrumenting every handler etc. This is known as automatic instrumentation.\nOn the contrary, it is not always possible to use automatic instrumentation for whatever reason. Maybe the framework is not supported yet, or you want to observe something which is out of the scope of current implementations. In that case, you would revert back to the good old days by writing instrumentation manually.\nIn the next post, we would see an example of instrumentation with the golang application. Used their SDK to export to the console. More info: https://opentelemetry.io/docs/instrumentation/go/getting-started/\n","date":"25 February 2023","externalUrl":null,"permalink":"/posts/2023/opentelemetry-for-the-first-timers/","section":"Posts","summary":"Introduction # Let us start by asking this question:\nWhat is observability and why do we need telemetry?\nAlthough observability and telemetry might sound like interchangeable words to use, observability is the whole thing that consists of instrumentation to viewing the metrics.\nHere telemetry is the instrumentation part. And in this post, we are majorly going to talk about this part only.\n","title":"OpenTelemetry for the First Timers","type":"posts"},{"content":" Introduction # In a previous post, we have done a comparison between __new__ and __init__. In this post, we are going to do something similar, but with a different pair.\nFor those who don\u0026rsquo;t know, __str__() and __repr__() are two dunder methods which you can override in a class to change the way class is represented.\nThese two little functions have been a source of confusion in my early career. Let us first go through them one by one and let us understand what they do first. Then we\u0026rsquo;ll do comparison among them with respect to their similarities and differences.\n__str__() and str() # If you have worked with other language before coming to Python, you might be familiar with str() or string() function, which would cast any other data type to String data type or class.\nPython behaves the same. Here is an example in Python REPL.\n1 2 3 4 5 6 \u0026gt;\u0026gt;\u0026gt; my_int = 2 \u0026gt;\u0026gt;\u0026gt; type(my_int) \u0026lt;class \u0026#39;int\u0026#39;\u0026gt; \u0026gt;\u0026gt;\u0026gt; my_str = str(my_int) \u0026gt;\u0026gt;\u0026gt; type(my_str) \u0026lt;class \u0026#39;str\u0026#39;\u0026gt; Simlar is the repr() function. repr also returns a string.\n1 2 3 4 5 6 \u0026gt;\u0026gt;\u0026gt; my_int = 2 \u0026gt;\u0026gt;\u0026gt; type(my_int) \u0026lt;class \u0026#39;int\u0026#39;\u0026gt; \u0026gt;\u0026gt;\u0026gt; my_repr = repr(my_int) \u0026gt;\u0026gt;\u0026gt; type(my_repr) \u0026lt;class \u0026#39;str\u0026#39;\u0026gt; __repr__() and repr() # By now you understand that both the dunder method has their corresponding function symbols in which you can pass another data type.\nWe also saw in the previous section that when you pass anything to repr(), it simply turns into a str data type.\n__str__() and __repr__() in a Class # If you still don\u0026rsquo;t know about dunder methods, this section might help you undestand it.\nThere is a massive list of Python dunder methods. Dunder methods allow the way your handwritten class to behave in a certain manner.\nThe difference between the two methods is that __str__ is intended to provide a human-readable representation of the object, while __repr__ should provide a representation that can be used to recreate the object.\nLet\u0026rsquo;s see the difference between the two with the help of a simple code example:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 class Point: def __init__(self, x, y): self.x = x self.y = y def __str__(self): return f\u0026#34;({self.x}, {self.y})\u0026#34; def __repr__(self): return f\u0026#34;Point({self.x}, {self.y})\u0026#34; p = Point(1, 2) print(p) # Output: (1, 2) print(repr(p)) # Output: Point(1, 2) In the example above, __str__ returns a string representation of the Point object that is intended to be human-readable. The __repr__ method returns a string representation that could be used to recreate the object.\nIn general, it is recommended to always define a __repr__ method for your objects, so that you have an unambiguous representation of the object that can be used for debugging or in a REPL environment. On the other hand, __str__ is typically only defined when a more human-readable representation is needed. If __str__ is not defined for an object, repr(obj) is used as a fallback.\nIt\u0026rsquo;s worth noting that the __str__ method should return a string, while the __repr__ method should return a string that, when passed to the eval() function, returns an object with the same value. The recommended format for __repr__ is a string that, when passed to eval(), creates an object with the same value as the original. In the example above, the Point object can be recreated by calling Point(1, 2).\nConclusion # In conclusion, __str__ and __repr__ are two important methods in Python that allow you to define string representations of objects. Understanding the difference between the two and when to use them is important in creating robust and well-designed code.\nKeep reading:\nhttps://www.geeksforgeeks.org/str-vs-repr-in-python/ ","date":"11 February 2023","externalUrl":null,"permalink":"/posts/2023/__str__-vs-__repr__-and-when-to-use-them/","section":"Posts","summary":"Introduction # In a previous post, we have done a comparison between __new__ and __init__. In this post, we are going to do something similar, but with a different pair.\nFor those who don’t know, __str__() and __repr__() are two dunder methods which you can override in a class to change the way class is represented.\nThese two little functions have been a source of confusion in my early career. Let us first go through them one by one and let us understand what they do first. Then we’ll do comparison among them with respect to their similarities and differences.\n","title":"__str__ vs __repr__ in Python and When to Use Them","type":"posts"},{"content":"","date":"11 February 2023","externalUrl":null,"permalink":"/tags/syntax/","section":"Tags","summary":"","title":"Syntax","type":"tags"},{"content":" Introduction # Why rootless docker?\n\u0026ldquo;Rootless\u0026rdquo; Docker refers to running Docker containers without granting the Docker daemon full root privileges on the host system. This can provide added security, as it reduces the potential attack surface and damage that could be caused by a malicious or compromised Docker daemon.\nIn normal Docker usage, the Docker daemon runs as the root user, which means it has complete access to the host system. While this is necessary for many Docker features and functionality, it can also represent a significant security risk, especially when running untrusted or unknown containers.\nBy using rootless Docker, you can reduce the level of privileged access that the Docker daemon has on your system, making it harder for attackers to exploit vulnerabilities in the Docker software or its containers.\nIt\u0026rsquo;s worth noting that some features and functionality may not be available in a rootless Docker setup, as they require full root access. However, for many users and use cases, rootless Docker provides a good balance between security and functionality.\nIn summary, the main difference between normal Docker and rootless Docker is the level of privileged access that the Docker daemon has on the host system. While normal Docker requires full root access, rootless Docker operates with reduced privileges for added security.\nHow to install # These instructions are for Linux. Similar instructions might exist for macOS. But certainly, Windows does not have a root concept, and I haven\u0026rsquo;t explored the Windows side of rootless docker. If you write a follow up post on Windows side, please left it in the comments section for other to view.\nI am following these instructions on Ubuntu. If you are using the same, you might not be aware that docker command is provided by 2 different packages. One is from docker\u0026rsquo;s official repo which is available as docker-ce. Another one is from Debian\u0026rsquo;s official repository which is available as docker.io.\nIf you want to proceed, please uninstall docker.io if you are using it:\nsudo apt remove docker docker-engine docker.io containerd runc Install docker-ce. If you are already using docker-ce, you\u0026rsquo;re good to go:\nsudo apt update sudo apt install docker-ce docker-ce-cli containerd.io I am assuming that your docker service is running, stop them using the following command:\nsudo systemctl disable --now docker.service docker.socket There is one dependency that is remaining, it\u0026rsquo;s uidmap. Do it like this:\nsudo apt install uidmap Now we are ready to do the final step:\ncurl -fsSL https://get.docker.com/rootless | sh You can see the installation progressing, here is my log:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 # Installing stable version 23.0.0 # Executing docker rootless install script, commit: 6647403 % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 63.7M 100 63.7M 0 0 143M 0 --:--:-- --:--:-- --:--:-- 143M % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 19.1M 100 19.1M 0 0 85.1M 0 --:--:-- --:--:-- --:--:-- 85.4M + PATH=/home/ubuntu/bin:/home/ubuntu/.local/bin:/home/ubuntu/.vscode-server/bin/e2816fe719a4026ffa1ee0189dc89bdfdbafb164/bin/remote-cli:/home/ubuntu/.local/bin:/home/ubuntu/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/go/bin:/home/ubuntu/go/bin:/var/lib/snapd/snap/bin:/usr/local/go/bin:/home/ubuntu/go/bin:/var/lib/snapd/snap/bin:/usr/local/go/bin:/home/ubuntu/go/bin:/var/lib/snapd/snap/bin /home/ubuntu/bin/dockerd-rootless-setuptool.sh install [INFO] Creating /home/ubuntu/.config/systemd/user/docker.service [INFO] starting systemd service docker.service + systemctl --user start docker.service + sleep 3 + systemctl --user --no-pager --full status docker.service ● docker.service - Docker Application Container Engine (Rootless) Loaded: loaded (/home/ubuntu/.config/systemd/user/docker.service; disabled; vendor preset: enabled) Active: active (running) since Sat 2023-02-11 10:27:47 UTC; 3s ago Docs: https://docs.docker.com/go/rootless/ Main PID: 1523 (rootlesskit) Tasks: 35 Memory: 31.3M CPU: 535ms CGroup: /user.slice/user-1000.slice/user@1000.service/app.slice/docker.service ├─1523 rootlesskit --net=slirp4netns --mtu=65520 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run --propagation=rslave /home/ubuntu/bin/dockerd-rootless.sh ├─1532 /proc/self/exe --net=slirp4netns --mtu=65520 --slirp4netns-sandbox=auto --slirp4netns-seccomp=auto --disable-host-loopback --port-driver=builtin --copy-up=/etc --copy-up=/run --propagation=rslave /home/ubuntu/bin/dockerd-rootless.sh ├─1546 slirp4netns --mtu 65520 -r 3 --disable-host-loopback --enable-sandbox --enable-seccomp 1532 tap0 ├─1554 dockerd └─1574 containerd --config /run/user/1000/docker/containerd/containerd.toml --log-level info Feb 11 10:27:47 ip-10-2-1-39 dockerd-rootless.sh[1554]: time=\u0026#34;2023-02-11T10:27:47.220314464Z\u0026#34; level=warning msg=\u0026#34;WARNING: No io.max (wbps) support\u0026#34; Feb 11 10:27:47 ip-10-2-1-39 dockerd-rootless.sh[1554]: time=\u0026#34;2023-02-11T10:27:47.220323424Z\u0026#34; level=warning msg=\u0026#34;WARNING: No io.max (riops) support\u0026#34; Feb 11 10:27:47 ip-10-2-1-39 dockerd-rootless.sh[1554]: time=\u0026#34;2023-02-11T10:27:47.220336374Z\u0026#34; level=warning msg=\u0026#34;WARNING: No io.max (wiops) support\u0026#34; Feb 11 10:27:47 ip-10-2-1-39 dockerd-rootless.sh[1554]: time=\u0026#34;2023-02-11T10:27:47.220346775Z\u0026#34; level=warning msg=\u0026#34;WARNING: bridge-nf-call-iptables is disabled\u0026#34; Feb 11 10:27:47 ip-10-2-1-39 dockerd-rootless.sh[1554]: time=\u0026#34;2023-02-11T10:27:47.220355915Z\u0026#34; level=warning msg=\u0026#34;WARNING: bridge-nf-call-ip6tables is disabled\u0026#34; Feb 11 10:27:47 ip-10-2-1-39 dockerd-rootless.sh[1554]: time=\u0026#34;2023-02-11T10:27:47.220391885Z\u0026#34; level=info msg=\u0026#34;Docker daemon\u0026#34; commit=d7573ab graphdriver=overlay2 version=23.0.0 Feb 11 10:27:47 ip-10-2-1-39 dockerd-rootless.sh[1554]: time=\u0026#34;2023-02-11T10:27:47.220586509Z\u0026#34; level=info msg=\u0026#34;Daemon has completed initialization\u0026#34; Feb 11 10:27:47 ip-10-2-1-39 dockerd-rootless.sh[1554]: time=\u0026#34;2023-02-11T10:27:47.261491765Z\u0026#34; level=info msg=\u0026#34;[core] [Server #10] Server created\u0026#34; module=grpc Feb 11 10:27:47 ip-10-2-1-39 systemd[919]: Started Docker Application Container Engine (Rootless). Feb 11 10:27:47 ip-10-2-1-39 dockerd-rootless.sh[1554]: time=\u0026#34;2023-02-11T10:27:47.269415389Z\u0026#34; level=info msg=\u0026#34;API listen on /run/user/1000/docker.sock\u0026#34; + DOCKER_HOST=unix:///run/user/1000/docker.sock /home/ubuntu/bin/docker version Client: Version: 23.0.0 API version: 1.42 Go version: go1.19.5 Git commit: e92dd87 Built: Wed Feb 1 17:43:29 2023 OS/Arch: linux/amd64 Context: default Server: Docker Engine - Community Engine: Version: 23.0.0 API version: 1.42 (minimum version 1.12) Go version: go1.19.5 Git commit: d7573ab Built: Wed Feb 1 17:46:24 2023 OS/Arch: linux/amd64 Experimental: false containerd: Version: v1.6.16 GitCommit: 31aa4358a36870b21a992d3ad2bef29e1d693bec runc: Version: 1.1.4 GitCommit: v1.1.4-0-g5fd4c4d docker-init: Version: 0.19.0 GitCommit: de40ad0 rootlesskit: Version: 1.1.0 ApiVersion: 1.1.1 NetworkDriver: slirp4netns PortDriver: builtin StateDir: /tmp/rootlesskit896353388 slirp4netns: Version: 1.0.1 GitCommit: 6a7b16babc95b6a3056b33fb45b74a6f62262dd4 + systemctl --user enable docker.service Created symlink /home/ubuntu/.config/systemd/user/default.target.wants/docker.service → /home/ubuntu/.config/systemd/user/docker.service. [INFO] Installed docker.service successfully. [INFO] To control docker.service, run: `systemctl --user (start|stop|restart) docker.service` [INFO] To run docker.service on system startup, run: `sudo loginctl enable-linger ubuntu` [INFO] Creating CLI context \u0026#34;rootless\u0026#34; Successfully created context \u0026#34;rootless\u0026#34; [INFO] Use CLI context \u0026#34;rootless\u0026#34; Current context is now \u0026#34;rootless\u0026#34; [INFO] Make sure the following environment variables are set (or add them to ~/.bashrc): export PATH=/home/ubuntu/bin:$PATH Some applications may require the following environment variable too: export DOCKER_HOST=unix:///run/user/1000/docker.sock We have finally installed docker. And what we have basically done is installed it on the user level.\nAt the end of the installation, you\u0026rsquo;ll be asked to stick some exports in your .bashrc. You need to do that in order to experience a flawless switch.\nThings you should know # Start/Stop Container # To start or stop the docker daemon, you need same old systemctl command. But you\u0026rsquo;re going to pass --user flag after the command.\nsystemctl --user stop docker systemctl --user start docker Connect from Client # There are a few ways you can interact with docker daemon.\nThe most common way is the command line way, another is docker desktop. There are also different SDKs available in different languages.\nWhen you use rootful docker, you do not have to explicitly DOCKER_HOST becuase it is already set. But after switching to rootless docker, you might need to specify this environment variable when you intend to connect to the host.\nValue of this environment variable can be found by using this string:\n1 unix://$XDG_RUNTIME_DIR/docker.sock You may want to expand the XDG_RUNTIME_DIR when you are trying to\nConclusion # ","date":"28 January 2023","externalUrl":null,"permalink":"/posts/2023/exploring-rootless-docker/","section":"Posts","summary":"Introduction # Why rootless docker?\n“Rootless” Docker refers to running Docker containers without granting the Docker daemon full root privileges on the host system. This can provide added security, as it reduces the potential attack surface and damage that could be caused by a malicious or compromised Docker daemon.\nIn normal Docker usage, the Docker daemon runs as the root user, which means it has complete access to the host system. While this is necessary for many Docker features and functionality, it can also represent a significant security risk, especially when running untrusted or unknown containers.\n","title":"Exploring Rootless Docker","type":"posts"},{"content":"","date":"28 January 2023","externalUrl":null,"permalink":"/tags/security/","section":"Tags","summary":"","title":"Security","type":"tags"},{"content":" Introduction # I have always been conscious about privacy and it was the sole reason I wasn\u0026rsquo;t early adapter of smart home devices. This changed when I bought Raspberry Pi. I learned that I can make my Pi hub to all the smart devices at my home.\nAnd this is how I bought 9W Philips Wiz light; the cheapest one I could get in the Indian market.\nTo use this product, you at least need to have a home network (a wifi router will work). The router does not need to have internet connection.\nControl Philips Wiz using App # The most common way to control the Wiz light is using the mobile app. It is available for both Android and iOS.\nI don\u0026rsquo;t want to cover how to operate the bulb using the app. I would leave it upto you. I want you to explore the app completely before proceeding towards the next section.\nAs we are going to control the bulb using code, I want you to practice doing these actions:\nTurn on/off Set colors Set brightness Set temperature Set modes Control Philips Wiz using Go # While there are libraries available to interact with the bulb, I wanted to show you guys what happens internally with the bulb. In a home automation scenario, you\u0026rsquo;d mostly be using such libraries.\nFind the bulb IP # Before we write any piece of code, it is crucial to find out the IP of your bulb. In most latest routers, they show what clients are connected to the router, the device name should appear as wiz_xxxxxx where xxxxxx is the starting mac address of the bulb.\nIf somehow your router does not supports showing the IP of connected devices, you\u0026rsquo;ll have to do some hit and trial.\nGet the IP of your PC/Laptop. You can run ip a if you are on Linux. My PC\u0026rsquo;s IP is 192.168.0.100. Turn off the light from the app and send the Hello Philips Wiz code described in next section to each of the IPs near to your PCs IP. So I\u0026rsquo;d send following message to 192.168.0.95-105. How many machines you have to try depends on how many devices you have on your network.\nWe need the IP because we are going to communicate with bulb by sending and receiving UDP messages.\nHello Philips Wiz using UDP # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 package main import \u0026#34;net\u0026#34; var wizIP = \u0026#34;192.168.0.101\u0026#34; // this is address on my bulb in my network const wizPort = \u0026#34;38899\u0026#34; func main() { c, err := net.Dial(\u0026#34;udp\u0026#34;, fmt.Sprintf(\u0026#34;%s:%s\u0026#34;, wizIp, wizPort) if err != nil { panic(\u0026#34;could not connect to wiz light\u0026#34;) } c.Write([]byte(`{\u0026#34;method\u0026#34;: \u0026#34;setPilot\u0026#34;, \u0026#34;params\u0026#34;:{\u0026#34;state\u0026#34;: true}}`)) } Starting at line 5, first we have to define the IP of the bulb. It is 192.168.0.101 in my case.\nNext we have to define the port on which Wiz light listens on. This port ID is constant and won\u0026rsquo;t change unlike the wizIP.\nLine 14 is the most important. The message is actually a JSON. Let\u0026rsquo;s beautify it before discussing about it.\n1 2 3 4 5 6 { \u0026#34;method\u0026#34;: \u0026#34;setPilot\u0026#34;, \u0026#34;params\u0026#34;: { \u0026#34;state\u0026#34;: true } } \u0026ldquo;method\u0026rdquo;: The first key in the JSON object is method. This is either getPilot or setPilot depending on whether you want to get the state of the bulb, or set a state to the bulb. In our case, we are going to set some state. I have never really used the getPilot method, yet. \u0026ldquo;params\u0026rdquo;: Next in the JSON object is parameters we are sending; it is denoted by params in the JSON body. This is going to complement the method. Now depending on what we are trying to do with the bulb, the params are going to change. In above code, we are just saying to set the state to true, which literally means to start the bulb.\nIf your bulb was off, it would have turned on after running the above code.\nSimilarly, if you want to turn it off, change the state to false and send the message again.\nSet custom colors # Wiz accepts RGB colors values. If you don\u0026rsquo;t know what that means, go to google and type in \u0026ldquo;rgb color picker\u0026rdquo;, then choose a color and then note the RGB value. For example, value for pure red would be 255, 0, 0. Here 255 is the value of red, and 0 is for both green and blue.\nTo set Wiz to red color, this message can be sent:\n1 2 3 4 5 6 7 8 { \u0026#34;method\u0026#34;: \u0026#34;setPilot\u0026#34;, \u0026#34;params\u0026#34;: { \u0026#34;r\u0026#34;: 255, \u0026#34;g\u0026#34;: 0, \u0026#34;b\u0026#34;: 0 } } You have already seen how to send that message.\nSet brightness # In the app UI, you must have seen the brightness slider? If not, consider the following screenshot.\nApp showing temperature and brightness slider First of all, I have warm white light selected. The top slider is for temperature, which we are going to talk about in next section.\nThe lowest value for brightness is 10, and max is 100. You guessed it right, lowest is not 0 because at 0, the bulb will be off.\nYou can club brightness setting with scene settings as well as any color setting. In following message, I\u0026rsquo;m setting the color to green and brightness to 50%.\n1 2 3 4 5 6 7 8 9 { \u0026#34;method\u0026#34;: \u0026#34;setPilot\u0026#34;, \u0026#34;params\u0026#34;: { \u0026#34;r\u0026#34;: 255, \u0026#34;g\u0026#34;: 0, \u0026#34;b\u0026#34;: 0, \u0026#34;dimming\u0026#34;: 50, } } Set temperature # Setting temperature is similarly easy. There is a params called temp. The range is between 2200 to 6200.\nTo the best of my knowledge, temperature can only be clubbed with brightness.\nHere I\u0026rsquo;m presenting you an example payload for the message demonstrating 2200 kelvin temperate with a intensity of 20%.\n1 2 3 4 5 6 7 { \u0026#34;method\u0026#34;: \u0026#34;setPilot\u0026#34;, \u0026#34;params\u0026#34;: { \u0026#34;temp\u0026#34;: 2200 \u0026#34;dimming\u0026#34;: 20, } } Conclusion # When I bought my Philips Wiz, I had no intention of controlling it programatically. It would such an overkill.\nBut shortly after this buy, I also bought a Raspberry Pi. I became aware that this programatic access can be integrated by home automation.\nLibraries already exist in Python. One of them is pywizlight, which seems to be most mature. But there are many more.\nMaybe we could write some similar library in Go.\n","date":"25 January 2023","externalUrl":null,"permalink":"/posts/2023/how-to-control-philips-whiz-bulb-using-go/","section":"Posts","summary":"Introduction # I have always been conscious about privacy and it was the sole reason I wasn’t early adapter of smart home devices. This changed when I bought Raspberry Pi. I learned that I can make my Pi hub to all the smart devices at my home.\nAnd this is how I bought 9W Philips Wiz light; the cheapest one I could get in the Indian market.\n","title":"How to Control Philips Wiz Bulb Using Go","type":"posts"},{"content":"","date":"25 January 2023","externalUrl":null,"permalink":"/tags/philips-wiz/","section":"Tags","summary":"","title":"Philips-Wiz","type":"tags"},{"content":" Introduction # It\u0026rsquo;s no longer than a month since I bought a Raspberry Pi. Pi 4B to be precise. In this post, I\u0026rsquo;m going to log how I spent my 1st month with it.\nSetup the Pi # When I decided to buy Raspi, I decided not to go with the kit. The reason was the model I was looking for, which is 8GB Model B wasn\u0026rsquo;t available in India. And finding a kit for it would have been more challenging.\nSo I bought the Pi itself from OL Electronics. It cost me ₹10,700 INR with taxes and delivery charge.\nAs you might know, Pi does not ship with a power adapter. So I bought one from robu.in. It cost me ₹315 with delivery charges.\nPi also does not comes with a memory card. I tried searching for 64 micro sd cards offline and online. The best deal was from Amazon. It cost me ₹539. You will also need a card reader, which I bought from the local market at a price of ₹30.\nWhile flashing, I choose to use the Lite variant of OS which comes without a desktop environment and has a footprint of 300 MB on disk.\nSetting up the OS # The Pi board, a 5V 3A Type-C power adapter, a micro sd, and a card reader is all you need to boot up a Pi. You don\u0026rsquo;t need a monitor or HDMI adapter if you already have a laptop because you can also SSH to your Pi to do your work remotely and that is how I usually work on my Pi.\nAs you know, I use Ubuntu on my machine. If you are also using Ubuntu, you can install it by:\nsudo apt install rpi-imager Instruction for other operating systems can be found on their GitHub page.\nPut a micro sd card in the card reader and Please consider watching How to use Raspberry Pi Imager.\nBlink Light # I prefer to use the gpiozero package for dealing with GPIO. But there is also an RPi module.\n1 2 3 4 5 6 7 8 9 10 import time from gpiozero import LED led = LED(18) while True: led.on() time.sleep(1) led.off() time.sleep(1) Make sure you attach the long head of the LED to pin 18.\nThere is also an equivalent code using the RPi package:\n1 2 3 4 5 6 7 8 9 10 11 12 import time import RPi.GPIO as GPIO led_pin = 18 GPIO.setmode(GPIO.BCM) GPIO.setup(led_pin, GPIO.OUT) while True: GPIO.output(led_pin, True) time.sleep(1) GPIO.output(led_pin, False) time.sleep(1) I have also short a video I have posted on Instagram. Here it is:\nBlinking LED using GPIO programming Actuate on Proximity # Here I have used both packages to run something when the proximity sensor detects any object near the sensor.\nThe sensor I am using is an IR proximity sensor.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 import time from gpiozero import LED import RPi.GPIO as GPIO GPIO.setmode(GPIO.BOARD) GPIO.setup(8, GPIO.IN) led = LED(18) while True: if GPIO.input(8) == 0: print(\u0026#34;turning light on\u0026#34;) led.on() else: print(\u0026#34;turning light off\u0026#34;) led.off() time.sleep(0.1) IR sensor in action Controlling LED brightness, 0-100 and back # Below is Python code to change the brightness of a LED light using Raspberry Pi.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 import time import RPi.GPIO as GPIO led_pin = 18 GPIO.setmode(GPIO.BCM) GPIO.setup(led_pin, GPIO.OUT) pwm_led = GPIO.PWM(led_pin, 500) pwm_led.start(100) while True: start = 0 end = 100 while start \u0026lt; end: pwm_led.ChangeDutyCycle(start) time.sleep(.025) start += 1 break I don\u0026rsquo;t shoot a video for this one because the intensity was so low that it was very hard to distinguish between different strengths of light.\nControlling an RGB light (failed) # I have also bought a common cathode LED. But somehow, I\u0026rsquo;m not able to turn on the LED. Not sure if the problem is in the LED, my code, or the pins.\nHere is the code I was using.\n1 2 3 4 5 from gpiozero import RGBLED led = RGBLED(22, 27, 17) led.color = (1, 0, 0) And here is the setup I was using.\nFailed attempt to use RGB LED In the above image, the blue, green, and red are placed at 6, 7, and 8 on the left column, and the common cathode is placed at 3rd from the right.\nPlease let me know if you have a solution for this one.\nConclusion # This is not the entirety of the things I have done with my Raspberry Pi in the first month. I have a DHT11 sensor which I mistakenly bought without the IC. And now I need a 4.7K resistor to work with it. I am yet to order the resistor.\nAlongside Raspi accessories, I have also bought some NFC tags to play with. I will post another blog if I make something interesting with it.\nInstalling Home Assistant is another thing that I have done on my Pi. This board is going to be the hub of other smart devices on my network. I\u0026rsquo;m planning to buy some ESF32 to help me with my home automation journey.\nIf you are also keen on home automation and IoT, I\u0026rsquo;d recommend you IoT for Beginners.\nLet me know if you want more articles like this one.\n","date":"3 January 2023","externalUrl":null,"permalink":"/posts/2023/first-month-with-my-raspberry-pi/","section":"Posts","summary":"Introduction # It’s no longer than a month since I bought a Raspberry Pi. Pi 4B to be precise. In this post, I’m going to log how I spent my 1st month with it.\nSetup the Pi # When I decided to buy Raspi, I decided not to go with the kit. The reason was the model I was looking for, which is 8GB Model B wasn’t available in India. And finding a kit for it would have been more challenging.\n","title":"First Month With My Raspberry Pi","type":"posts"},{"content":"","date":"3 January 2023","externalUrl":null,"permalink":"/tags/raspberry-pi/","section":"Tags","summary":"","title":"Raspberry-Pi","type":"tags"},{"content":" Introduction # I have seen many people coming from Python background struggling with Go. What I have obeserved it they write code that is limited to writing functions and calling them. They never leverage design constructs such as abstraction, encapsulation, inheritance, composition etc. I can understand their pain because I\u0026rsquo;ve been in the same boat.\nIn today\u0026rsquo;s post I\u0026rsquo;ll compare how we write classes in Python vs how we do the same thing in a Go way.\nWhat is a class composed of? # Attributes in form of variables Behavior in form of methods Example in Python # Consider the following code in Python that defines a simple Person class with a name attribute and a greet method:\n1 2 3 4 5 6 7 8 9 class Person: def __init__(self, name): self.name = name def greet(self): print(f\u0026#34;Hello, my name is {self.name}\u0026#34;) p = Person(\u0026#34;John\u0026#34;) p.greet() # prints \u0026#34;Hello, my name is John\u0026#34; If you\u0026rsquo;re a Python programmer looking to learn Go, you\u0026rsquo;ll be happy to know that Go has a similar concept to classes called \u0026ldquo;structs\u0026rdquo;. Structs are similar to classes in that they allow you to define a custom data type with its own fields and methods.\nStruct to define the structure and attributes # To define a struct in Go, you use the type keyword followed by the name of the struct and a set of curly braces containing the fields. Here\u0026rsquo;s an example of a simple struct called Person:\n1 2 3 4 type Person struct { name string age int } This defines a struct called Person with two fields: name and age, both of which are of type string and int respectively.\nMethod or behaviour using receiver function # You can also define methods on your structs by using the func keyword and attaching the method to the struct using the (p *Person) syntax, where p is the receiver of the method and Person is the type of the receiver. Here\u0026rsquo;s an example of a method called SayHello that prints a greeting to the console:\n1 2 3 func (p *Person) SayHello() { fmt.Printf(\u0026#34;Hello, my name is %s and I am %d years old\\n\u0026#34;, p.name, p.age) } To create an instance of a struct, you can use the new function or the shorthand syntax :=. Here\u0026rsquo;s an example of both:\n1 2 3 4 5 6 7 // Using the new function p1 := new(Person) p1.name = \u0026#34;John\u0026#34; p1.age = 30 // Using the shorthand syntax p2 := Person{name: \u0026#34;Jane\u0026#34;, age: 25} You can also define a \u0026ldquo;constructor\u0026rdquo; function to create instances of your struct. This can be useful if you want to perform some initialization logic when creating a new instance. Here\u0026rsquo;s an example of a constructor function for the Person struct:\n1 2 3 4 5 6 func NewPerson(name string, age int) *Person { p := new(Person) p.name = name p.age = age return p } You can then use this constructor function to create a new instance of the Person struct like this:\n1 p := NewPerson(\u0026#34;Bob\u0026#34;, 35) I hope this tutorial has helped you understand how to define and use structs in Go as a Python programmer. Structs are a powerful and flexible way to define custom data types in Go, and I encourage you to explore them further.\nThank you for reading!\n","date":"20 December 2022","externalUrl":null,"permalink":"/posts/2022/how-to-write-classes-in-golang/","section":"Posts","summary":"Introduction # I have seen many people coming from Python background struggling with Go. What I have obeserved it they write code that is limited to writing functions and calling them. They never leverage design constructs such as abstraction, encapsulation, inheritance, composition etc. I can understand their pain because I’ve been in the same boat.\nIn today’s post I’ll compare how we write classes in Python vs how we do the same thing in a Go way.\n","title":"How to Write Classes in Golang","type":"posts"},{"content":"","date":"20 December 2022","externalUrl":null,"permalink":"/tags/oops/","section":"Tags","summary":"","title":"Oops","type":"tags"},{"content":"","date":"8 December 2022","externalUrl":null,"permalink":"/tags/api-testing/","section":"Tags","summary":"","title":"Api-Testing","type":"tags"},{"content":"","date":"8 December 2022","externalUrl":null,"permalink":"/tags/postman/","section":"Tags","summary":"","title":"Postman","type":"tags"},{"content":" Introduction # In Postman Premier - The Minimal Postman you need to Learn, I posted the minimum of Postman you need to learn. In this article, I\u0026rsquo;m going to expand on that knowledge and dig more into the response. In this post, I\u0026rsquo;ll be asserting the response on certain values and response codes and more.\nWhat are assertions and how to use it # Assertions are nothing but test cases we give to a particular request. In the previous post, we already have seen that each of the requests has a Tests tab. There we write code in JavaScript, judging specific parameters of the response.\nLet\u0026rsquo;s first test for a specific status code. Just for context, while testing, I\u0026rsquo;m going to use my gingo server.\nLooking for a specific status code # Testing for a specific status code is one of the easiest tasks to do within Postman. I\u0026rsquo;m going to use my GET {{baseUrl}}/books endpoint for this purpose.\nWhen you open the Tests tab, you\u0026rsquo;ll see snippets on the right side column. Click on Status code: Code is 200.\nThe code area will be populated with the following:\n1 2 3 pm.test(\u0026#34;Status code is 200\u0026#34;, function () { pm.response.to.have.status(200); }); When you send the request, you\u0026rsquo;ll see Test Results (1/1) under the response section. That area shows how many of your tests have passed or failed.\nI\u0026rsquo;ve recorded a gif for you.\nTesting for specific status code This was for the positive assertions, you can send wrong data and do a negative assertion as well.\nLooking for a specific string in the body # For this test, I\u0026rsquo;ll be using the GET {{baseUrl}}/books/:isbn endpoint, with isbn\u0026rsquo;s value to be 9781612680194.\nLooking for a specific string in the body is not much different from checking for a status code. You\u0026rsquo;ll find a snippet for it as well. Snippets are a great way to explore assertion in Postman.\nLook for an assertion called Response body: Contains string. It will spit this source code into the test panel.\n1 2 3 pm.test(\u0026#34;Body matches string\u0026#34;, function () { pm.expect(pm.response.text()).to.include(\u0026#34;string_you_want_to_search\u0026#34;); }); I\u0026rsquo;m going to modify the body:\n1 2 3 pm.test(\u0026#34;Body includes Rich Dad Poor Dad\u0026#34;, function () { pm.expect(pm.response.text()).to.include(\u0026#34;Rich Dad Poor Dad\u0026#34;); }); Now I\u0026rsquo;m not going to include a gif for this as this would be very identical to the previous one.\nLooking for a specific header # Another thing we can test in a REST API is headers. Some use cases to look for a header are:\nWhen you are expecting a specific content type When you are expecting an authentication token as a response to a request. For a header, the testing code would look like this:\n1 2 3 pm.test(\u0026#34;Content-Type is application/json\u0026#34;, function () { pm.response.to.have.header(\u0026#34;Content-Type\u0026#34;, \u0026#34;application/json\u0026#34;); }); Adding tests at the collection level # Response time is another thing to test if your API is time critical. This time we don\u0026rsquo;t add this test on a request basis. We are programmers and we DRY. Checking for responses is going to be for each and every request, it\u0026rsquo;s better to add it at the collection level.\nTo add a test at the collection level, click on the collection name on the left panel. As already discussed, you\u0026rsquo;d see Authorization, Pre-request Script, Tests and Variables.\nNavigate to Tests, and put this in the code area:\n1 2 3 pm.test(\u0026#34;Response time is less than 500ms\u0026#34;, function () { pm.expect(pm.response.responseTime).to.be.below(500); }); 500ms is good enough for local testing. If your server is over the internet, you might need to modify it to suit your need.\nThis test will run for every request which is there in the collection and for each new request, you add to the collection.\nRun all tests from a collection # When you click on a collection, you\u0026rsquo;ll see a Run button at the top right corner. When you click that, you\u0026rsquo;ll be asked which tests you want to run.\nThis is the Collection Runner UI. When you are done with all the config. Click on the Run Postman Premier to run all the tests and see the results.\nConclusion # We have learned enough Postman to get our wheels rolling. There\u0026rsquo;s much more to Postman than we have discussed in 2 articles on Fullstack with Santosh. Lots of features keep getting added in each release. You can keep yourself updated by following Postman Docs.\n","date":"8 December 2022","externalUrl":null,"permalink":"/posts/2022/testing-http-requests-with-postman/","section":"Posts","summary":"Introduction # In Postman Premier - The Minimal Postman you need to Learn, I posted the minimum of Postman you need to learn. In this article, I’m going to expand on that knowledge and dig more into the response. In this post, I’ll be asserting the response on certain values and response codes and more.\nWhat are assertions and how to use it # Assertions are nothing but test cases we give to a particular request. In the previous post, we already have seen that each of the requests has a Tests tab. There we write code in JavaScript, judging specific parameters of the response.\n","title":"Testing HTTP Requests with Postman","type":"posts"},{"content":" Introduction # Welcome to my Fullstack with Santosh. Today we are going to learn about how to use Postman as a backend engineer to make our daily life easier. Postman is an API testing tool where you can test your REST APIs. It has now developed into handling many different kinds of interfaces, but today we are going to focus on the REST APIs.\nMake sure you have Postman already installed. I work on Ubuntu and the way I install is sudo snap install postman. You can find instructions about your platform on the downloads page: https://www.postman.com/downloads/\nWhen you open Postman for the first time, you might see a screen similar to this:\nPostman home screen Your home screen might appear a little different because 1. I have already signed in. 2. I have already created some workspaces. 3. I have changed the theme to dark.\nCreate a collection # Before you even start to test endpoints in your server, the first thing you\u0026rsquo;d do is create a collection. Press that little + button on the left column.\nCreate new collection Give it a name. I\u0026rsquo;m going to name it Postman Premier.\nThere are a lot more things that can be done with a Collection. But for now, let\u0026rsquo;s not get deep into it.\nNote: If you don\u0026rsquo;t have an API to test, you can refer to my previous blog posts: Building a Book Store API in Golang With Gin and How to Integrate Swagger UI in Go Backend - Gin Edition where I show how you can develop your simple CRUD server in Go. It\u0026rsquo;s worth a look.\nWorking with Requests # I am going to test my gingo server, which is a CRUD server I have developed before. I already have linked them above.\nCreating a GET Request to get introduced to UI # I have already run my gingo server and I am going to make a GET request at http://localhost:8090/api/v1/books.\nCreating a request is straightforward in Postman. Let\u0026rsquo;s start with creating a GET request.\nWhen you expand your created collection on the left column, you\u0026rsquo;ll see a link called Create a request. Click on that, and a tab will open in the main column.\nIn that window, a text will already be selected. This is the identifier of the request. I like to name my request the same as the path of the endpoint. In my case, I\u0026rsquo;m going to name it /books.\nThe next thing you do is select a verb. There is a dropdown just below the name of the request. This will by default be GET. For this particular request, we don\u0026rsquo;t modify that.\nNext, we\u0026rsquo;ll input the address of the endpoint we need to hit. I\u0026rsquo;ve input http://localhost:8090/api/v1/books.\nWe don\u0026rsquo;t need any other setting for now. Don\u0026rsquo;t be intimidated by the UI if you don\u0026rsquo;t understand it yet.\nI have recorded a gif for your reference. Here it goes.\nCreate a GET request What we covered in this section: Creating a request, request name, request verb, and request path.\nPath Parameters # We can get all books. Let us get a specific book; by ISBN.\nThe quickest way to create our next request is to duplicate the previous one and modify the duplicated one.\nLet us name this new request /books/:isbn. For the path, we\u0026rsquo;re going to use localhost:8090/api/v1/books/:isbn. As soon as you write :isbn in the path input, Params section will be highlighted.\nWhen you navigate to that tab, you\u0026rsquo;d see isbn is already added as KEY in Path Variables section. You need to provide value. You can get ISBN from the response of the previous endpoint we tested. E.g. 9781612680194.\nWhen you do this much, you should see a single book in the response, unlike the previous response, which had all the books.\nBelow is a gif for this section.\nWorking with path params Creating a POST Request and working with Body and Headers # Duplicate the first endpoint we created again, but this time, change the verb to POST.\nIn the previous section Params section was highlighted. This time you need to highlight the Body section.\nWhen you highlight Body, you\u0026rsquo;ll see a group of radio buttons. Click on the one labeled with raw. A dropdown will appear at the right with the text Text. Click on the dropdown and select JSON.\nWhen you select JSON from the dropdown, a new header is added to the request. You can see the header when you highlight the Headers tab. In this tab, you can see many other headers already added by Postman.\nWhen the header is set, use the following body to make a POST request:\n1 2 3 4 5 { \u0026#34;isbn\u0026#34;: \u0026#34;9780137081073\u0026#34;, \u0026#34;title\u0026#34;: \u0026#34;The Clean Coder\u0026#34;, \u0026#34;author\u0026#34;: \u0026#34;Robert C. Martin\u0026#34; } Here is a gif of me doing the same thing.\nWorking with body and header For those who don\u0026rsquo;t know. Headers are not limited to POST requests. It might be required for a GET endpoint to have an auth token in its header.\nMaking requests for other verbs such as DELETE and PUT is no different. I left it over for you to try with those.\nWorking with variables # Using Collection variables # We have different variable scopes available when working with Postman:\nGlobal \u0026gt; Collection \u0026gt; Environment \u0026gt; Local\nThey all have their use case:\nGlobal - Global variables are global to all the collections you have in Postman. This should be mostly avoided. Collection - This one is my favorite. You can have collection-level variables for say credentials, baseurl, etc. Environment - If you are testing a collection with multiple data sets or conditions. Environment variables come to use. Local - Local variables are local to request and will override all other scopes variables. I\u0026rsquo;ll demonstrate Collection variables, but all behave the same way and have similar UI.\nClick on the Postman Premier collection and you\u0026rsquo;ll see 4 tabs. Authorization, Pre-request Script, Tests and Variables. Click on the variables.\nCollection Variables use case: We are doing a lot of repetitions in each of the requests. We are inputting the base URL of the server each time. If we had to somehow modify the port address, we would have to modify it for all.\nLet us change that to use a variable. If we ever need to change the value of the baseurl, we\u0026rsquo;ll only be changing that in a single place.\nWhen you see the variables tab, you\u0026rsquo;ll see a table with 3 columns. Namely VARIABLE, INITIAL VALUE, and CURRENT VALUE. Set VARIABLE = baseUrl, INITIAL VALUE = http://localhost:8090/api/v1, CURRENT VALUE = http://localhost:8090/api/v1. INITIAL_VALUE and CURRENT_VALUE are the same here, but CURRENT_VALUE can be programmatically changed during request-response life cycle.\nWorking with collection variables Now change all the occurrences of http://localhost:8090/api/v1 in the collection\u0026rsquo;s URL section to {{baseUrl}}. This will be a manual process, but this is worth it.\nSaving response data in variables # Why would you want to save response data in a variable?\nThat is a good question to ask. One such use case is user login. When you log in to a service, you\u0026rsquo;d get some kind of token with a certain validity time. That token is needed to make further requests. We can add a header using Pre-request script using the variable we save.\nOur gingo server does not have a user authentication module right now. Let us try saving isbn from a GET request.\nThis may sound a little bit misleading, but anything you want to do after a request goes inside the Tests tab. This also includes setting a variable for a session.\nGo to the Tests section of /books/:isbn and add this javascript code:\n1 2 const body = pm.response.json() pm.collectionVariables.set(\u0026#34;isbn\u0026#34;, body.isbn); The code is straightforward. We are storing the JSON response in a const called body. Then we access the collection variable using pm.collectionVariables. We then set the variable name isbn with the value of body.isbn.\nHere is me doing the same thing.\nSaving response data in variables Similar to collection variables, we can set any scope of variable we discussed above.\nConclusion # In this post, we saw some basic use cases of Postman. In upcoming posts, we\u0026rsquo;ll see more advanced use cases. Until then, subscribe and stay updated.\n","date":"20 November 2022","externalUrl":null,"permalink":"/posts/2022/postman-premier-the-minimal-postman-you-need-to-learn/","section":"Posts","summary":"Introduction # Welcome to my Fullstack with Santosh. Today we are going to learn about how to use Postman as a backend engineer to make our daily life easier. Postman is an API testing tool where you can test your REST APIs. It has now developed into handling many different kinds of interfaces, but today we are going to focus on the REST APIs.\n","title":"Postman Premier - The Minimal Postman you need to Learn","type":"posts"},{"content":"","date":"5 November 2022","externalUrl":null,"permalink":"/tags/database/","section":"Tags","summary":"","title":"Database","type":"tags"},{"content":" Introduction # Database normalization is the process of structuring a database, in accordance with a series of normal forms in order to reduce data redundancy and improve integrity.\nIn other words, database normalization is basically breaking a table into two to prevent repetitive columns aka data redundancy. It entails organizing the columns (attributes) and tables (relations) of a database to ensure that their dependencies are properly enforced by database integrity constraints. It is accomplished by applying some formal rules either by a process of synthesis (creating a new database design) or decomposition (improving an existing database design).\nThat still does not makes sense, right? For this to make sense I would like to define some prerequisites.\nDefinitions # Primary Key: Primary key is the key or column selected by database administrator to uniquely identify tuples (aka records) in a table. There can be only 1 primary key in a table. Foreign Key: Foreign key are primary key of a different table and is used to create relationship between two tables. Composite Key: If there is no single primary key, we select more than one columns whose combintion would act as a primary key. This is called composite key. What are normal forms by the way? # We already defined what normalization is. Now there are a few \u0026lsquo;passes\u0026rsquo; in which data can be normalized. One pass is known as a normal form.\nSo what are different normal forms?\n1. First Normal Form # This is step 1 of the normalization process. At least follow 1st normal form or do not use databases. Scalable Table design which can be extended.\nThere should be a primary key in the table. Each column should contain atomic values. Meaning only one value should be there, not a set or list of values. A column should contain values that are of the same type. Do not inter-mix different types of values in any column. Most RDBMS will throw you an error if you pass another datatype than schema. Each column should have a unique name. Same names lead to confusion at the time of data retrieval.\nOrder in which data is saved doesn\u0026rsquo;t matter. Using SQL query, you can easily fetch data in any order from a table.\nBreak the list of two items into 2 rows. This will create redundancy, but let\u0026rsquo;s wait for the Second Normal Form. But first, let\u0026rsquo;s look at an example. Below is a table without 1st normal form applied. This dataset is of a car rental company where car owners can exibit their car for rent. And other party can book their car for a certain period of time.\nCustomer ID Customer Name Car Number Plate Car Name Date of Transaction Owner ID Owner Name C12 Sachin CarQ1234 CarQ5436 Swift Thar 12/01/2020 18/01/2020 076 078 Dev Irfan C46 Rahul CarQ3421 CarQ6534 CarQ3789 Baleno Honda City Swift 12/01/2020 14/01/2020 15/01/2020 054 065 086 Rohit Shikhar Irfan Above data is not in 1st normal form because each customer has multiple car number plates in a single row.\nCustomer ID Customer Name Car Number Plate Car Name Date of Transaction Owner ID Owner Name C12 Sachin CarQ1234 Swift 12/01/2020 076 Dev C12 Sachin CarQ5436 Thar 18/01/2020 078 Irfan C46 Rahul CarQ3421 Baleno 12/01/2020 054 Rohit C46 Rahul CarQ6534 Honda City 14/01/2020 065 Shikhar C46 Rahul CarQ3789 Swift 15/01/2020 086 Irfan Is it in first normal form now? Definately not. It does not have a primary key.\nIf you look closely, none of the field is unique and every entry is prone to repeat, even Car Number Plate; which will repeat as the car is booked by a different person.\nIn a case where there is no single primary key, we have a select a composite key, which as we know is combination of two column which together can act unique among the row.\nWe can select Car Number Plate + Date of Transation as a composite key. Here we assume that one car can be booked by only one person in a day. By taking these two fields, we can say our table is in first normal form now.\n2. Second Normal Form # To have a table in 2nd normal form:\nThe table to be in 1st Normal Form. There should be no Partial Functional Dependencies. What is Partial Functional Dependency?\nLet us understand this by an example.\nWe have our composite key. By keeping that aside and looking at rest of the table, we can say that Car Name is dependent on one of the field from our composite key; Car Plate Number.\nWhen field is only dependent on one of the composite fields, it is called that a particular field is having partial functional dependency. PFD only happens in the case of composite key.\nOwner ID is also PFD on Car Number Plate. Same with Owner name.\nWe\u0026rsquo;ll have to break our dataset into two tables.\nCar Number Plate Customer ID Customer Name Date of Transaction CarQ1234 C12 Sachin 12/01/2020 CarQ5436 C12 Sachin 18/01/2020 CarQ3421 C46 Rahul 12/01/2020 CarQ6534 C46 Rahul 14/01/2020 CarQ3789 C46 Rahul 15/01/2020 The Car Number Plate in this table acts as a foreign key in the other table.\nAnd this is our another table:\nCar Number Plate Car Name Owner ID Owner Name CarQ1234 Swift 076 Dev CarQ5436 Thar 078 Irfan CarQ3421 Baleno 054 Rohit CarQ6534 Honda City 065 Shikhar CarQ3789 Swift 086 Irfan 3. Third Normal Form # For a table to be in 3rd Normal Form:\nIt should be in 2nd Normal Form There should be no transitive dependencies. What is a Transitive Dependency?\nTransitive Dependency can also be said as Indirect dependencies. Let us take an example again.\nLook at table 2 from previous section. Owner Name depends on Owner ID and Owner ID dependens on Car Number Plate. You can\u0026rsquo;t find Owner Name directly from the Car Number Plate (there can be repeating name). The other two fields, Car Name and Owner ID directly depends on the Car Number Plate.\nWe now have to further split two tables into other 2. First table from previous section is now split into two.\nCar Number Plate Customer ID Date of Transaction CarQ1234 C12 12/01/2020 CarQ5436 C12 18/01/2020 CarQ3421 C46 12/01/2020 CarQ6534 C46 14/01/2020 CarQ3789 C46 15/01/2020 And this\u0026hellip;\nCustomer ID Customer Name C12 Sachin C46 Rahul Now the second table is also split into two:\nCar Number Plate Car Name Owner ID CarQ1234 Swift 076 CarQ5436 Thar 078 CarQ3421 Baleno 054 CarQ6534 Honda City 065 CarQ3789 Swift 086 An this\u0026hellip;\nOwner ID Owner Name 076 Dev 078 Irfan 054 Rohit 065 Shikhar 086 Irfan Why we normalize tables? # Anamolies. There are different type of anamolies.\nWhat are different type of anamolies?\nUpdation Anamolies\nDeletion Anamolies\nInsertion Anamolies\nWe\u0026rsquo;ll get to know more about this with examples.\nUpdation Anamolies # So for setting the stage, let\u0026rsquo;s assume that our table is not normalized. Look at the table from 1st normal form.\nNow assume that Dev has sold his car to Roushan. If you go ahead and replace the Owner Name in first row, you\u0026rsquo;ll still have N number of rows remaining to update.\nWe have to update fields at mulpitle place.\nDeletion Anamolies # Although it\u0026rsquo;s not practicle in the table above. Assume that you have to remove a customer. If you remove a customer Sachin, you\u0026rsquo;ll be deleting all the data with it including Car Number Plate, Car Name, Owner ID/Name.\nThis is known as Deletion Anamolies.\nInsertion Anamolies # In Insertion Anamolies, you can\u0026rsquo;t insert a new row without you have all the other data associated with that row. For example, you want to add a new car to the database. You also need a customer, date of transaction and so on.\nThis is known as Insertion Anamolies.\nWhy not to normalize database # If your system uses non-relational database, you might be against normalization. Although relation database can scale horizontally too, it\u0026rsquo;s not by design efficient at that.\nIf you have multiple table on multiple machines, you will have a network overload with each join. It does not scales well.\nIf you are using databases such as document based database, you\u0026rsquo;d want to have all related data on same document. So that when you shard, you don\u0026rsquo;t have to deal with network overload making the application slower.\nConclusion # This was my first post realted to databases in my blogging history. I have learn from it more than teaching from it. I hope this post was worth of your time too.\nCover image is (c) Integrify.com\n","date":"5 November 2022","externalUrl":null,"permalink":"/posts/2022/database-normalization-scratching-the-database-surface/","section":"Posts","summary":"Introduction # Database normalization is the process of structuring a database, in accordance with a series of normal forms in order to reduce data redundancy and improve integrity.\nIn other words, database normalization is basically breaking a table into two to prevent repetitive columns aka data redundancy. It entails organizing the columns (attributes) and tables (relations) of a database to ensure that their dependencies are properly enforced by database integrity constraints. It is accomplished by applying some formal rules either by a process of synthesis (creating a new database design) or decomposition (improving an existing database design).\n","title":"Database Normalization: Scratching the Database Surface","type":"posts"},{"content":"","date":"5 November 2022","externalUrl":null,"permalink":"/tags/normalization/","section":"Tags","summary":"","title":"Normalization","type":"tags"},{"content":"","date":"5 November 2022","externalUrl":null,"permalink":"/tags/rdbms/","section":"Tags","summary":"","title":"Rdbms","type":"tags"},{"content":"","date":"5 November 2022","externalUrl":null,"permalink":"/tags/sql/","section":"Tags","summary":"","title":"Sql","type":"tags"},{"content":" Introduction # I recently conducted a poll in the Go community regarding which logging library they use in their personal or professional project. The results were interesting. In this post I have jotted down the most favorite logging libraries and here is my opinion about them.\nPoll Results # Here are the poll results which I conducted on r/golang, The Go Programming Language community on Twitter and my LinkedIn following.\nPoll results on r/golang Poll results on Twitter Poll results on LinkedIn Analysis of Results # Selection of the candidates # For choosing the candidates for the poll, I didn\u0026rsquo;t do any thorough research. I was looking for a library to use in my project at work, and I ended up at sirupsen/logrus which was already being used by one of the dependencies in that project.\nAfter reading the README of the logrus, I noted it was in maintenance mode and no new feature was actively being developed. I still decided to use this library because logrus is a mature project and it has all the features which a logging library should have.\nlogrus README recommended using other libraries such as Zerolog, Zap, and Apex.\nI started a poll on r/golang with these four candidates, but also came to know about glog which was a go port of a C++ project by Google. I used that option in the poll conducted on LinkedIn.\nI haven\u0026rsquo;t personally used any one of them yet.\nlogrus # Let\u0026rsquo;s talk about logrus first, it was the first library I landed upon when looking for one. The reason I looked at it first is that logrus was already one of the indirect dependencies of one of the dependencies in my project.\nlogrus stands 2nd in the place. zerolog is faster than logrus but there\u0026rsquo;s a reason it stands seconds among 4. It has been there for a while.\nzerolog # The main reason you\u0026rsquo;d want to use zerolog if you are switching from logrus is the performance. Go is blazingly fast; but if your logger is making it slow, your application gets no benefit from being implemented in Go.\nThis is what zerolog\u0026rsquo;s README says:\nThe zerolog package provides a fast and simple logger dedicated to JSON output.\nZerolog\u0026rsquo;s API is designed to provide both a great developer experience and stunning performance. Its unique chaining API allows zerolog to write JSON (or CBOR) log events by avoiding allocations and reflection.\nzap # zap is probably the most loved logging library for Go. Among 817 people who have voted on r/golang, 315 voted for zap.\nAccording to the stats on their GitHub, zap is the fastest logger available there.\nAlso, zap only supports the two most recent minor versions of Go. So you might be locked with an older version of zap if your project for some reason is not able to update to the latest version of Go.\nConclusion # To have an opinion, I feel that I use them all before coming to a conclusion. But after this poll, I think I\u0026rsquo;m going to start using zap right away. What is your opinion? Why factor was responsible when you were choosing your logging library.\n","date":"22 October 2022","externalUrl":null,"permalink":"/posts/2022/best-logging-library-for-golang/","section":"Posts","summary":"Introduction # I recently conducted a poll in the Go community regarding which logging library they use in their personal or professional project. The results were interesting. In this post I have jotted down the most favorite logging libraries and here is my opinion about them.\nPoll Results # Here are the poll results which I conducted on r/golang, The Go Programming Language community on Twitter and my LinkedIn following.\n","title":"Best Logging Library for Golang","type":"posts"},{"content":"","date":"22 October 2022","externalUrl":null,"permalink":"/tags/logging/","section":"Tags","summary":"","title":"Logging","type":"tags"},{"content":"","date":"3 September 2022","externalUrl":null,"permalink":"/tags/aws/","section":"Tags","summary":"","title":"Aws","type":"tags"},{"content":"","date":"3 September 2022","externalUrl":null,"permalink":"/tags/aws-sdk/","section":"Tags","summary":"","title":"Aws-Sdk","type":"tags"},{"content":"","date":"3 September 2022","externalUrl":null,"permalink":"/tags/lambda/","section":"Tags","summary":"","title":"Lambda","type":"tags"},{"content":" Introduction # Today we are going to do a hands-on on how to kill a dangling EIP or in other words, deallocate an unassigned Elastic IP. But before that, let us know\u0026hellip;\nWhat is an Elastic IP and why do we need it? # Elastic IPs are simply IP addresses provided by Amazon AWS. These are usually attached to internet gateways such as load balancers, or even EC2 instances. I have been using it with EC2 instances directly for my testing use case.\nNormally when you configure your EC2 instance to have a public IP but at the same time, you don\u0026rsquo;t use EIP. The problem is that when you stop and start your instance, your IP address is changed.\nThis is where Elastic IP shines. You attach an EIP to the EC2 instance and when you stop and start your instance, your IP won\u0026rsquo;t be changed.\nThis is extremely important if the EC2 you are running is hosting an internet-facing web server. Web servers shouldn\u0026rsquo;t generally frequently change their IPs.\nBut why do we need an EIP killer? # There is 1 good thing about EIP and 1 bad thing. The good thing is EIP is free to use. You don\u0026rsquo;t need to pay anything for the static IP you are using. How cool!\nThe bad thing is, that when EIP is not attached to an instance, it cost you money. How warm!\nIt won\u0026rsquo;t cost you much if you leave it detached for a few hours. But as a human, we tend to forget if EIP is in use and only get to know about it when we get our monthly bill.\nIn the Asia Pacific (Mumbai) region, if I forget to kill an EIP for a month, I get around a $4 bill. This was my main motivation to write this article.\nWith an EIP killer, I will be able to use Elastic IP with confidence by releasing it when not in use.\nGet Started # What we\u0026rsquo;ll work with:\nAWS Lambda Go AWS Go SDK AWS CLI What we\u0026rsquo;re going to do:\nRun a Lambda on a predefined interval. This Lambda will use AWS SDK to check if any dangling EIP is lying around. If dangling EIP is found, deallocate, or in our language, kill it. Initially, I was going to use Python for the Lambda runtime, but Go is faster than Python. We are going to benefit from this decision as we can run our function for a smaller time span. This is important because we plan to run our Lambda at frequent intervals.\nWith this post, I mostly plan to expand my knowledge in the field of serverless computing. I intend to touch Lambda, EventBridge, and related constructs.\nPhase 1: Provision a Lambda function # The main task in this phase is to manually create a function. In the second phase, we\u0026rsquo;ll see what is the logic for killing the EIPs. Let\u0026rsquo;s get started.\nGo through the web UI and choose to create a function. Keep \u0026ldquo;Author from scratch\u0026rdquo; selected. Fill in the name. I liked to call it \u0026ldquo;eip-killer\u0026rdquo;. Select the runtime to be Go 1.x. Keep everything else on default. Hit Create function. Create Lambda func from Web UI Note: Creating Lambda with default settings also creates an IAM role. This IAM role dictates how this Lambda interacts with other AWS services. By default, this role allows Lambda to talk with CloudWatch to put logs. You can add more policies to this role when working with other AWS services.\nLet\u0026rsquo;s go ahead and do a quick invoke to test if the function is working.\nInvoke result of Lambda (click to zoom) Our EIP-killer is alive. But doesn\u0026rsquo;t know how to kill. Let\u0026rsquo;s teach it in the next phase.\nPhase 2: Write the EIP killing logic # While testing, you might have got a view of the Code tab of the function which says, \u0026ldquo;The code editor does not support the Go 1.x runtime\u0026rdquo;. What this means for us developers is that we have to use our text editor to write the logic and upload the pre-built binary to Lambda.\nThe prime documentation for Go AWS SDK is https://docs.aws.amazon.com/lambda/latest/dg/lambda-golang.html, but don\u0026rsquo;t worry. I\u0026rsquo;ll filter out the required material required for this post.\nPhase 2.1: The initial deployment process # Before we write the actual logic, let\u0026rsquo;s write a simple \u0026ldquo;Hello EIP Killer\u0026rdquo; code to get the awareness of the process required for uploading the code.\nCreate a workspace for our lambda code. 1 2 mkdir eip-killer cd eip-killer Initialize a go module in the folder. 1 go mod init EIP-killer You could also put the GitHub/GitLab URL of the repo where you intend to put your lambda code at.\nCreate a main.go and put this code into it: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 package main import ( \u0026#34;github.com/aws/aws-lambda-go/lambda\u0026#34; ) func hello() (string, error) { return \u0026#34;Hello EIP Killer!\u0026#34;, nil } func main() { // Make the handler available for Remote Procedure Call by AWS Lambda lambda.Start(hello) } As we are using external dependency, get it with this command:\n1 go get github.com/aws/aws-lambda-go/lambda Build a binary out of main.go: 1 GOOS=linux GOARCH=amd64 go build -o main main.go Remember while creating the function we had selected architecture to be x86_64. So we need to reflect that with GOARCH=amd64.\nZip the binary. Upload the zip to function. 1 zip main.zip main Upload using this command:\n1 aws lambda update-function-code --function-name eip-killer --zip-file fileb://main.zip While this could have been enough, the default handler (where lambda begins executing) in function with factory settings is hello. But our code uses main. So we need to update the handler config to match that.\nWe need this update only once.\n1 aws lambda update-function-configuration --function-name eip-killer --handler main After this, when you run the func:\nUpdated function You can roam around this path: https://github.com/aws/aws-lambda-go if you want to dig deeper into Go+Lambda. But for now, I\u0026rsquo;m going to use AWS SDK to do our magic.\nPhase 2.2: Logic to deallocate unused EIPs # Open up the main.go again and write this:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 package main import ( \u0026#34;context\u0026#34; \u0026#34;fmt\u0026#34; \u0026#34;log\u0026#34; \u0026#34;github.com/aws/aws-lambda-go/lambda\u0026#34; \u0026#34;github.com/aws/aws-sdk-go-v2/config\u0026#34; \u0026#34;github.com/aws/aws-sdk-go-v2/service/ec2\u0026#34; \u0026#34;github.com/aws/aws-sdk-go-v2/service/ec2/types\u0026#34; ) type MyEvent struct { Name string `json:\u0026#34;name\u0026#34;` } func Handler(ctx context.Context, name MyEvent) (string, error) { // Load the Shared AWS Configuration (~/.aws/config) cfg, err := config.LoadDefaultConfig(ctx) if err != nil { log.Fatal(err) } // Create an Amazon EC2 service client ec2ServiceClient := ec2.NewFromConfig(cfg) // DescribeAddressesInput with no filter means list all addresses IpListFilter := \u0026amp;ec2.DescribeAddressesInput{} // Ask the EC2 client to list all Elastic IPs. result, err := ec2ServiceClient.DescribeAddresses(context.TODO(), IpListFilter) if err != nil { fmt.Println(\u0026#34;Got an error retrieving information about your Amazon Elastic IPs:\u0026#34;) return \u0026#34;\u0026#34;, err } // Out of all IPs, check if they have an allocation ID. If not, they are a candidate for deletion for _, a := range result.Addresses { fmt.Println(\u0026#34;Allocation ID: \u0026#34; + *a.AllocationId) fmt.Println(\u0026#34;Public IP: \u0026#34; + *a.PublicIp) // if this EIP is not associated with any AWS resources, proceed with killing. if a.AssociationId == nil { // extra step for flexibility if isThisEIPKillable(a.Tags) { fmt.Println(\u0026#34;About to be killed\u0026#34;) // Release/kill the Elastic IP ReleaseAddressFilter := \u0026amp;ec2.ReleaseAddressInput{AllocationId: a.AllocationId} ec2ServiceClient.ReleaseAddress(context.TODO(), ReleaseAddressFilter) fmt.Println(*a.PublicIp + \u0026#34; is not more\u0026#34;) } else { fmt.Println(*a.PublicIp + \u0026#34; has KILLERHOLD enabled and is going to stay\u0026#34;) } } else { fmt.Println(\u0026#34;Association ID: \u0026#34; + *a.AssociationId) fmt.Println(\u0026#34;Instance ID: \u0026#34; + *a.InstanceId) fmt.Println(\u0026#34;PrivateIpAddress: \u0026#34; + *a.PrivateIpAddress) } fmt.Println(\u0026#34;\u0026#34;) } return \u0026#34;Done execution\u0026#34;, nil } // isThisEIPKillable scans to tags of Elastic IP and only returns false when // EIPKILLER is tag is set to true. func isThisEIPKillable(tags []types.Tag) bool { for _, tag := range tags { if *tag.Key == \u0026#34;KILLERHOLD\u0026#34; \u0026amp;\u0026amp; *tag.Value == \u0026#34;true\u0026#34; { return false } } return true } func main() { lambda.Start(Handler) } Quite a large source code. Let us understand it line by line.\nLine 14-16 declares an event struct. This struct is going to be a parameter to the Handler function.\nOn line 18, we see function with signature Handler(ctx context.Context, name MyEvent) (string, error). Handler here is the name of the function.\nThe first parameter to the handler is the context object. If you are working with web services in golang, you must be aware of that.\nNext is our MyEvent which is defined above. This is as per Lambda\u0026rsquo;s spec.\nOur handler function also returns a string and an error. Which is then bubbled to Lambda logs.\nLine 20 deals with configuration. Please note that we are using AWS SDK. And the SDK can run on platforms other than Lambda. On the local system, the default config is located at ~/.aws/config. But on Lambda, it automatically loads the region it\u0026rsquo;s running in.\nOn line 26 we create a new EC2 client to interact with our AWS account. We initialize this client with the config instance we created on line 15.\nOn line 29 we have initialized a filter for IPs. This seems to be an added step. But this variable is going to be a mandatory parameter for our next command.\nOn line 32 we get a list of all Elastic IPs associated with our account. There are no filters to the result as we have not specified any filter on line 29.\nIt\u0026rsquo;s time to look at the isThisEIPKillable function which is from 69 to 76. This is an extra feature for flexibility. This will forgive any EIP which has the tag KILLERHOLD set to true. So when you have an EIP you don\u0026rsquo;t want to kill, you set this tag to it. This is good for the experiment as our Lambda will be running on a 1 hours (configurable by user) interval and it might not be desired outcome to delete them every 1 hour.\nLet us get back to our main flow. We have a result variable with all the addresses in it. On line 40 we check if the IP is associated with any EC2 or not. Then we run the helper function we just discussed. If all is okay, meaning that KIP can be released, we go ahead and release the address back to the AWS pool.\nOn lines 57-59, if EIP is associated with any EC2, we print some metadata about the EIP association.\nThat\u0026rsquo;s it. It was not that hard to understand, isn\u0026rsquo;t it? Before we deploy it, let\u0026rsquo;s create some EIP and not allocate it to any EC2. We\u0026rsquo;ll also have some EIP having KILLERHOLD set to true.\nPhase 2.3: Test our EIP killer Give Lambda permission to release and read EIPs # I hope you have updated your lambda function and pushed the zip file to the function. Review Phase 2.1 if you have not. Continue reading if you did.\nBefore I test it with actual EIPs, I wanted to show something else. Right now if you try to test the function, you\u0026rsquo;d see something like this:\nLambda is not able to access EIPs If you clearly read the error message, it says that API call returned 403. Meaning that Lambda was not authorized to make the call to EIP APIs.\nTo overcome this problem, we need to give access to Lambda to read some metadata from EC2.\nWhen no Lambda Web UI for the function. If you go to Configuration \u0026gt; Permission \u0026gt; Execution role as described in the image below. You\u0026rsquo;ll see a role that was created while we created the function.\nPermission configuration for Lambda Click on that and you\u0026rsquo;ll be taken to the IAM page for that role.\nWhen on that page, look for a dropdown named Add permissions and click on Create inline policy.\nSwitch from Visual editor to JSON. And paste this policy.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 { \u0026#34;Version\u0026#34;: \u0026#34;2012-10-17\u0026#34;, \u0026#34;Statement\u0026#34;: [ { \u0026#34;Sid\u0026#34;: \u0026#34;VisualEditor0\u0026#34;, \u0026#34;Effect\u0026#34;: \u0026#34;Allow\u0026#34;, \u0026#34;Action\u0026#34;: [ \u0026#34;ec2:ReleaseAddress\u0026#34;, \u0026#34;ec2:DescribeAddresses\u0026#34; ], \u0026#34;Resource\u0026#34;: \u0026#34;*\u0026#34; } ] } As you can see, in the Action section, we have added permission for ec2:ReleaseAddress, and ec2:DescribeAddresses.\nReview policy and give it a name, and then Create policy.\nWhen done, you\u0026rsquo;d see this policy in the list of policies attached to the role.\nInline policy attached to role Phase 2.3: Test our EIP killer # We are not set for the actual testing.\nWe are going to allocate 3 Elastic IPs for our testing purpose.\nOne we\u0026rsquo;ll attach to the EC2 instance.\nThe second will be dangling.\nAnd the third one will be also dangling but will have our KILLERHOLD set to true.\nEIP set for testing To actually invoke the function (apart from testing it), one way is to create a public URL. We can also invoke through the AWS CLI, but I\u0026rsquo;m not choosing that for now. You can do it.\nCreate a public URL for lambda Then I went ahead and opened that URL to execute the function.\nLambda invoked by visiting public URL You can configure AWS API Gateway in front of the function. But that\u0026rsquo;s out of the scope of this post.\nNext, we see the result of the invocation.\nResult after EIP Killer execution That\u0026rsquo;s the end of Phase 2. We can kill EIPs now. How wonderful.\nPhase 3: Trigger Lambda at a 1-hour interval # The main part of the EIP killer is already developed. Let\u0026rsquo;s now execute it at a 1-hour interval as planned before.\nPhase 3.1: Configure EventBridge aka CloudWatch Events # We are going to emit an event at our 1-hour interval and we are going to use EventBridge for this.\nI have recorded a screencast gif to preserve some space on this post.\nConfigure EventBridge to emit event on 1 hour Once done, this event should appear in the Trigger section of the Lambda function; as seen in the below image\nTrigger config showing EventBridge rule Phase 3.2: Create an EIP and go to sleep # As I\u0026rsquo;m writing this post at 4 am. I\u0026rsquo;m going to create an EIP and go to sleep. Will check in the morning if our setup is working.\nA few hours later\u0026hellip;\nI woke up this morning with only 1 EIP.\nAnd this is how we gonna achieve this killing at 1 hour. You can configure the interval, but I think this is optimal. You also don\u0026rsquo;t need to have a public IP for this. So if you have, you can delete it now from the web interface.\n","date":"3 September 2022","externalUrl":null,"permalink":"/posts/2022/release-dangling-elastic-ips-using-lambda-and-go-sdk/","section":"Posts","summary":"Introduction # Today we are going to do a hands-on on how to kill a dangling EIP or in other words, deallocate an unassigned Elastic IP. But before that, let us know…\nWhat is an Elastic IP and why do we need it? # Elastic IPs are simply IP addresses provided by Amazon AWS. These are usually attached to internet gateways such as load balancers, or even EC2 instances. I have been using it with EC2 instances directly for my testing use case.\n","title":"Release Dangling Elastic IPs using Lambda and Go SDK","type":"posts"},{"content":"","date":"9 July 2022","externalUrl":null,"permalink":"/series/data-structure-and-algorithms/","section":"Series","summary":"","title":"Data Structure and Algorithms","type":"series"},{"content":"","date":"9 July 2022","externalUrl":null,"permalink":"/tags/datastructures/","section":"Tags","summary":"","title":"Datastructures","type":"tags"},{"content":" Introduction # In the last post, we have already seen that we can classify entire data structure into linear and non-linear data structure. Today we will see how we can implement one of the linear data structures in Java.\nInitial plan was to implement Linked List in Python. But Python is so higher level that we\u0026rsquo;d be actually writing higher order data structure on top of high order data structure. What I try to say with that is Python already has those data structure implemented. Java is more close to C/C++, and gives a balance of features and flexibility. I\u0026rsquo;d have chosen C++ for this post, but I don\u0026rsquo;t want to deal with memory management and pointers for sake of this post.\nProblem with Arrays # Here is one con of array that linked list fixes. See the image below:\nArray using contagious chunk of memory Arrays always uses contagious chunk of memory. In the image above you can see that we have 3 rows of 11 units of memory. In first row, a program has allocated 7 units of memory. Then next 7 units of memory of is empty. Then again 7 unit of memory is occupied with another program.\nNow at this point, you want to allocate 7 or less than 7 unit of memory, you can use the blank space there. But if you have to allocate more than 7 units of memory, you have to look for space somewhere else. In the image above, 11 units of memory is taken from the 3rd row.\nThis represents fragmentation. Meaning that memory is not used efficiently. That 7 units of memory might never be used.\nThis is worse when combined with the fact that you can\u0026rsquo;t change the size of the array after declaration. If you want more space, you have to allocate a different array. If you realized you need less memory than the allocated one, you can\u0026rsquo;t to much about it.\nWith linked list you are not force to use contagious chunk of memory. Overall use of linked list results in better utilization of memory.\nLinked List # A LinkedList is a sequential access linear data structure in which every element is a separate object called a Node, which has two parts:\nThe data The reference (or pointer) Each element of a linked-list has an object (actual data) which can have multiple attributes, and a reference to a similar next node.\nThis post is not about what linked list is, but how to implement it. So let\u0026rsquo;s look at the implementation now.\nWhether singly or doubly, Linked List is a collection of ListNodes.\nListNode # At the very first, we need a ListNode:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 public class ListNode { private int data; private ListNode next; public ListNode(int data) { this.data = data; } @Override public String toString() { return \u0026#34;ListNode(\u0026#34; + \u0026#34;data=\u0026#34; + data + \u0026#39;}\u0026#39;; } public int getData() { return data; } public void setData(int data) { this.data = data; } public ListNode getNext() { return next; } public void setNext(ListNode next) { this.next = next; } } In a minimalistic singly ListNode, we\u0026rsquo;d have 2 fields. First is the data itself, and the pointer to the next node.\nI have defined other methods for convinience, like the constructor, getters \u0026amp; setters for data and next pointer. I also have a toString method to represent node as a string.\nLinkedList # Linked List is composed (as a chain) of these Nodes. By the naming convension, the first node is called head and the last node is called tail.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 public class LinkedList { public static void main(String[] args) { ListNode node1 = new ListNode(1); ListNode node2 = new ListNode(2); ListNode node3 = new ListNode(3); ListNode node4 = new ListNode(4); ListNode node5 = new ListNode(5); node1.setNext(node2); node2.setNext(node3); node3.setNext(node4); node4.setNext(node5); } } We right now have no methods implemented in our Linked List. We don\u0026rsquo;t even have a method to add a node to linked list. We\u0026rsquo;ll do this as we go.\nMethods of a Linked List # Traverse a Linked List # 1 2 3 4 5 6 7 8 9 10 11 12 public static void traverse(ListNode head) { // base condition if (head == null) { return; } // logic System.out.print(head.getData() + \u0026#34; -\u0026gt; \u0026#34;); // recursive call traverse(head.getNext()); } Explanation:\nThe above code has 3 section denoted by comments. These are the minimal things which needs to be in a recursive function.\nLet\u0026rsquo;s do the dry run. Suppose we put node1 to this function. The function would check if it is null or not, if not, it will go ahead and print the data and the -\u0026gt; bit. Next it calls the same function will the next node in the list.\nWhen the recursion reaches at the last node, the head == null would become true, and recursion would end there. Function calls would pop out of stack.\nRunner code:\nYou can put the runner code inside the main method:\n1 2 3 4 public static void main(String[] args) { // ... traverse(node1); } Output:\n1 1 -\u0026gt; 2 -\u0026gt; 3 -\u0026gt; 4 -\u0026gt; 5 -\u0026gt; 6 -\u0026gt; 7 -\u0026gt; Excuse the -\u0026gt; at the end, and let\u0026rsquo;s focus on the logic. Fixing that extra bit is not in the scope of this post.\nLength of a Linked List # 1 2 3 4 5 6 7 8 public static int length(ListNode head) { int counter = 0; while (head != null) { counter++; head = head.getNext(); } return counter; } Explanation:\nFunction is passed a node, typically the first/head node. When the function executes, first we initialize a counter. Rest of the code is almost of the traverse code, just that we are incrementing counter variable on each iteration. At last we are returning the coutter.\nRunner code:\nYou can put the runner code inside the main method:\n1 2 3 4 public static void main(String[] args) { // ... System.out.println(length(node1)); } Output:\n1 5 Bonus:\nThe recursive version of length would look like this:\n1 2 3 public static int length(ListNode head) { return head == null ? 0 : 1 + length(head.getNext()); } Very briefly explaining, the function goes deep into recursion until the head == null becomes true. While returning back from recursion, it starts adding +1 to the result. The last result is returned after recursino is finished.\nBoth of the approach takes O(n) time. For O(1) time, you should consider maintaining a length variable and increment/decrement it at the time of adding/removing a node.\nCheck if a node is prerent # 1 2 3 4 5 6 7 public static boolean isPresentRecursive(ListNode head, int data) { if (head == null) { return false; } return head.getData() == data || isPresentRecursive(head.getNext(), data); } Explanation # This is a recursive approach to check if a Linked List has a data node. The code is identical to that of traversing. The only exception is we return true if we find the data node.\nThe return statement can be split into multiple lines for understanding.\n1 2 3 4 5 if (head.getData() == data) { return true; } return isPresentRecursive(head.getNext(), data); Runner code # You can put the runner code inside the main method:\n1 2 3 4 public static void main(String[] args) { // ... System.out.println(isPresentRecursive(node1, 3)); } Output # 1 true Insert at Kth position # This is the sole insertion method we are going to implement. This same method can be used to insert at head, as well as tail. For head, you\u0026rsquo;d pass K to be 0. For tail, you\u0026rsquo;d pass K to be length(head).\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 public static ListNode insertAtK(ListNode head, int data, int k) { if (head == null \u0026amp;\u0026amp; k != 0) { return head; } if (k \u0026lt; 0 || k \u0026gt; length(head)) { return head; } ListNode newNode = new ListNode(data); if (k == 0) { newNode.setNext(head); head = newNode; } else { int count = 0; ListNode temp = head; while (count \u0026lt; k-1) { temp = temp.getNext(); count++; } newNode.setNext(temp.getNext()); temp.setNext(newNode); } return head; } Explanation # At first we have some guard conditions. If head is null, it\u0026rsquo;s natural that new node should be at head. If that\u0026rsquo;s not the case, it returns the passed head node.\nIn next guard condition, we return head if K is beyond the length of the Linked List.\nIf that is not the case, we go ahead and create a ListNode with the given data. If k is set to be 0 meaning that node goes at the head. We set the new node\u0026rsquo;s next to the head, and then set the head value to new node because we also have to return the new head.\nRunner code # You can put the runner code inside the main method:\n1 2 3 4 public static void main(String[] args) { // ... System.out.println(isPresentRecursive(node1, 3)); } Output # 1 true Conclusion # In this post I have implemented some most common operations on a Linked List. In upcoming posts, we\u0026rsquo;d see some more operations on the same data structures. We\u0026rsquo;ll also come up with data structures problems to solve and will discuss the approach.\n","date":"9 July 2022","externalUrl":null,"permalink":"/posts/2022/implementing-linked-list-in-java-part-1/","section":"Posts","summary":"Introduction # In the last post, we have already seen that we can classify entire data structure into linear and non-linear data structure. Today we will see how we can implement one of the linear data structures in Java.\nInitial plan was to implement Linked List in Python. But Python is so higher level that we’d be actually writing higher order data structure on top of high order data structure. What I try to say with that is Python already has those data structure implemented. Java is more close to C/C++, and gives a balance of features and flexibility. I’d have chosen C++ for this post, but I don’t want to deal with memory management and pointers for sake of this post.\n","title":"Implementing Linked List in Java: Part 1","type":"posts"},{"content":"","date":"9 July 2022","externalUrl":null,"permalink":"/tags/linkedlist/","section":"Tags","summary":"","title":"Linkedlist","type":"tags"},{"content":"","date":"28 June 2022","externalUrl":null,"permalink":"/tags/ad-block/","section":"Tags","summary":"","title":"Ad-Block","type":"tags"},{"content":"","date":"28 June 2022","externalUrl":null,"permalink":"/tags/dns/","section":"Tags","summary":"","title":"Dns","type":"tags"},{"content":"","date":"28 June 2022","externalUrl":null,"permalink":"/tags/privacy/","section":"Tags","summary":"","title":"Privacy","type":"tags"},{"content":" Introduction # We have previously seen how you can Spin up your own VPN on Ubuntu 20.04. Among many benefits, one of the benefits of the VPN is geographical anonymity.\nToday we are going to discuss and set up yet another benefit. We are going to tweak our VPN server set up a little bit to enable ad-blocker out of the box. Meaning that you won\u0026rsquo;t have to install an ad-blocker on your browser. It will be enabled connection-wide. You won\u0026rsquo;t be seeing any ads on devices that you can\u0026rsquo;t control ads on, like smart TV, etc.\nSo without further ado, let\u0026rsquo;s get started.\nProvision OpenVPN on Ubuntu 20.04 # Let me go briefly on how to get the OpenVPN installed on a VM on AWS. I am going to use another installer this time. But for the rest of the configuration, I\u0026rsquo;ll anyway ask you to refer to my last post. It\u0026rsquo;s a gem.\nStep 1. Download and run the installation script from GitHub.\n1 2 curl -sL https://git.io/vpn -o openvpn-install.sh sudo bash openvpn-install.sh Step 2. Go through the instruction and enter the details as you go.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 Welcome to this OpenVPN road warrior installer! This server is behind NAT. What is the public IPv4 address or hostname? Public IPv4 address / hostname [xxx.xxx.xxx.xxx]: Which protocol should OpenVPN use? 1) UDP (recommended) 2) TCP Protocol [1]: What port should OpenVPN listen to? Port [1194]: Select a DNS server for the clients: 1) Current system resolvers 2) Google 3) 1.1.1.1 4) OpenDNS 5) Quad9 6) AdGuard DNS server [1]: Enter a name for the first client: Name [client]: ubuntu On the script is done running, you\u0026rsquo;d get an ovpn file at the /root/ directory. Download the file to your local system and verify by connecting to the server. Please find the instruction in my last post on how to retrieve this .ovpn configuration file and how to run it.\nConfirm the VPN connection # Before VPN connection IP (redacted for privacy):\nBefore VPN Connection IP After VPN connection IP (redacted for privacy):\nAfter VPN Connection IP Do not proceed from here if you are not able to connect to the VPN.\nWhen you can see your IP changed, proceed to the next section of installing the ad-blocker.\nHow to install Ad Blocker # First confirm that you see ads # It\u0026rsquo;s better to first disable/remove all the ad-block related plugins you might be using on your browser. If you are using Brave or another privacy-focused browser, please disable them for this testing and confirm that you can see ads everywhere you go.\nBest place to see ads is on Google\u0026rsquo;s first page As you can see in the above Google search result page, the entries starting with Ad are indeed the ads. We are tasked to make those links disappear.\nMake note of the newly created tunnel network interface # With the installation of OpenVPN, you\u0026rsquo;d see a new network interface added to your system. You can see this by:\nip a show tun0 Note: If you have more than 1 tunnel interface, you might have to find an appropriate name. E.g. tun1, tun2 etc.\nAs the above command run, it will show output like this:\n1 2 3 4 5 6 3: tun0: \u0026lt;POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP\u0026gt; mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 500 link/none inet 10.8.0.1/24 scope global tun0 valid_lft forever preferred_lft forever inet6 fe80::2d29:1edf:7724:149d/64 scope link stable-privacy valid_lft forever preferred_lft forever 10.8.0.1/24. Remember this address. We\u0026rsquo;re gonna need this at a later stage.\nInstall Pi-hole # What is Pi-hole? Pi-hole is a Linux-based network-level advertisement and Internet tracker blocking application that acts as a DNS sinkhole and optionally a DHCP server. You can find more at https://pi-hole.net/.\nSo it blocks the DNS lookups for ad/tracking domains. The name includes Pi because it was intended to be used on Pi, but was made available to the entire Linux community.\nWhy choose Pi-hole? There were other options available for ad/tracker blocking. But I chose Pi-hole because of its web interface where you can see stats. You can also add your domain to the block.\nNow enter these commands to install:\n1 2 curl -sL https://install.pi-hole.net -o basic-install.sh sudo bash basic-install.sh It should output something like this:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 [✓] Root user check .;;,. .ccccc:,. :cccclll:. ..,, :ccccclll. ;ooodc \u0026#39;ccll:;ll .oooodc .;cll.;;looo:. .. \u0026#39;,\u0026#39;. .\u0026#39;,,,,,,\u0026#39;. .\u0026#39;,,,,,,,,,,. .\u0026#39;,,,,,,,,,,,,.... ....\u0026#39;\u0026#39;\u0026#39;,,,,,,,\u0026#39;....... ......... .... ......... .......... .......... .......... .......... ......... .... ......... ........,,,,,,,\u0026#39;...... ....\u0026#39;,,,,,,,,,,,,. .\u0026#39;,,,,,,,,,\u0026#39;. .\u0026#39;,,,,,,\u0026#39;. ..\u0026#39;\u0026#39;\u0026#39;. [✓] Update local cache of available packages [✓] Checking apt-get for upgraded packages... 7 updates available [i] It is recommended to update your OS after installing the Pi-hole! [i] Checking for / installing Required dependencies for OS Check... [✓] Checking for grep [i] Checking for dnsutils (will be installed) [i] Waiting for package manager to finish (up to 30 seconds) [i] Processing apt-get install(s) for: dnsutils, please wait... And then continue to install packages.\nThe installer will also ask some questions. I\u0026rsquo;m going to guide you through it.\nScreen 1:\nFirst screen of the Pi-hole installer Nothing much to do on this screen. Pretty self-explanatory.\nScreen 2:\nSecond screen of the Pi-hole installer Donate if you like the software.\nScreen 3:\nHost should have static IP This is a fair warning here. You might be installing Pi-hole in a home environment. Or on the wild internet. In both cases, please make sure IP is not changed as the device loses power. Otherwise, Pi-hole might not function properly.\nScreen 4:\nChoose tunnel we created in prior section On this screen, choose the tunnel we created as part of the OpenVPN installation.\nScreen 5:\nChoose DNS Provider for Pi-hole Next, we choose the DNS Provider. Choose anyone you like.\nScreen 6:\nSelect a block list Pi-hole only comes with one. But we can add another after the installation has been done.\nScreen 7:\nSelect whether you want to install web admin interface How can I not turn the web admin interface On? This was the reason I chose Pi-hole.\nScreen 8:\nInstall packages required by web admin interface I have none of those PHP packages as well as any HTTP server on my system. I will tick the On option here as well.\nScreen 9:\nWant to log queries? In this particular setup I\u0026rsquo;m doing, I\u0026rsquo;m the only one who is going to use the VPN server. If you are planning to connect multiple devices or want to start a company around this, please be aware that logs will need more storage.\nScreen 10:\nHow much info you want to store on the server? Once again, I\u0026rsquo;m the only one using the server. So I\u0026rsquo;ll store everything.\nScreen 11:\nConfiguration complete window Please make note of the admin webpage password here. I think it won\u0026rsquo;t be shown again. Not sure.\nConfirm Pi-hole is working # At this point, if you try to access any domain which is listed on one of the block lists we have opted for. It should loop back to your localhost.\nYou can confirm this by running ping on one of the ad serving domains:\n1 2 3 4 5 6 $ ping pagead2.googlesyndication.com PING pagead2.googlesyndication.com (127.0.0.1) 56(84) bytes of data. 64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.020 ms 64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.044 ms 64 bytes from localhost (127.0.0.1): icmp_seq=3 ttl=64 time=0.046 ms 64 bytes from localhost (127.0.0.1): icmp_seq=4 ttl=64 time=0.052 ms As you can see, the response is coming from the localhost instead of the actual originating server.\nConfigure OpenVPN to use Pi-hole DNS # Update DNS Server # You need to locate the OpenVPN server config file and open it in an editor.\nsudo vim /etc/openvpn/server/server.conf You have to look for line(s) saying dhcp-option DNS and comment on all of them. Prepend # to start the line to comment.\nNow, paste this line:\npush \u0026quot;dhcp-option DNS 10.8.0.1\u0026quot; Please note that 10.8.0.1 is the tunnel address we fetched at the start of this post. If your tunnel has any address other than this, please use that here.\nNow it\u0026rsquo;s time to restart the OpenVPN service.\nsudo systemctl restart openvpn-server@server unable to resolve host my-hostname: No address associated with hostname # This might not be relevant to you. But I have been getting this error and preventing me to restart the openvpn service mentioned above. So I had to add the following line to my /etc/hosts file.\n127.0.0.1 \u0026lt;my-hostname\u0026gt; After this, I was able to restart the openvpn service.\nOpen ports for OpenVPN subnet # If you have set up the system till now, there is one last step remaining. Right now the connection is blocked between the Pi-hole and the OpenVPN clients. At this point, if you try to access any site, you\u0026rsquo;d still see ads.\nI\u0026rsquo;m using Ubuntu, so the firewall is managed with ufw, I had to issue these commands:\n1 2 3 ufw allow proto tcp from 10.8.0.0/24 to 10.8.0.1 port 80 ufw allow proto tcp from 10.8.0.0/24 to 10.8.0.1 port 53 ufw allow proto udp from 10.8.0.0/24 to 10.8.0.1 port 53 Please note that port 53 is used for DNS.\nRestart your VPN connection from your client. And you\u0026rsquo;d see no ads:\nOpenVPN with Pi-hole enabled == no ads Web Admin # Well, I don\u0026rsquo;t have much to talk about the web admin right now. I\u0026rsquo;ll have to sit down and traverse through all the settings and info available through the web admin area.\nAll I can do is share a screenshot of my web admin.\nPi-hole dashboard Conclusion # Today we learned how we can use Pi-hole to enable ad-blocking on our VPN connection. This VPN connection can be used by clients like our phones and TVs for which ad blockers are not available system wide.\nRelated reading:\nhttps://docs.pi-hole.net/guides/vpn/openvpn/setup-openvpn-server/ ","date":"28 June 2022","externalUrl":null,"permalink":"/posts/2022/add-ad-blocker-to-openvpn-enabled-vpn/","section":"Posts","summary":"Introduction # We have previously seen how you can Spin up your own VPN on Ubuntu 20.04. Among many benefits, one of the benefits of the VPN is geographical anonymity.\nToday we are going to discuss and set up yet another benefit. We are going to tweak our VPN server set up a little bit to enable ad-blocker out of the box. Meaning that you won’t have to install an ad-blocker on your browser. It will be enabled connection-wide. You won’t be seeing any ads on devices that you can’t control ads on, like smart TV, etc.\n","title":"This is how you add Ad Blocker to OpenVPN enabled VPN","type":"posts"},{"content":"","date":"28 June 2022","externalUrl":null,"permalink":"/tags/vpn/","section":"Tags","summary":"","title":"Vpn","type":"tags"},{"content":"","date":"26 May 2022","externalUrl":null,"permalink":"/tags/api/","section":"Tags","summary":"","title":"Api","type":"tags"},{"content":"","date":"26 May 2022","externalUrl":null,"permalink":"/tags/gin/","section":"Tags","summary":"","title":"Gin","type":"tags"},{"content":" Introduction # Unlike FastAPI, Gin does not have OpenAPI integration built in. With FastAPI when you add a route, documentation is already generated. With Gin, this is not the case. But recently I integrated Swagger UI in one of my Go backends and I wanted to document that process.\nPrerequisites # An already existing Gin server You might consider Building a Book Store API in Golang With Gin if you are starting from scratch. I\u0026rsquo;ll be using the same book store and extending over it to integrate Swagger UI.\nHow Swagger works with Gin # The way Swagger works with other Go backend is not different they all have the same mechanism. But how does it really work and what do we have to really do?\nSwagger uses the Go comment system which is very well integrated with documentation already as we know. We write comments in a pre-defined way, details of which we will see further ahead in the post. But mostly it is divided into 2 parts. The server itself and the routes.\nSwagger has a CLI binary which when runs converts these comment documents into OpenAPI compliant documentation. The resultant file also includes OpenAPI server specs in JSON and YAML format.\nLastly, we route the generated content via a handler in our backend.\nHow do we do that? Let\u0026rsquo;s see.\nSetup Swagger CLI # The prime repository you should keep under your pillow is https://github.com/swaggo/swag. Both in terms of CLI and documentation.\nWe\u0026rsquo;ll install a CLI application called swag. Here\u0026rsquo;s how:\n1 2 3 4 go get -u github.com/swaggo/swag/cmd/swag # 1.16 or newer go install github.com/swaggo/swag/cmd/swag@latest The first command will download the dependencies to integrate in the server application.\nThe second one is where we install the CLI.\nNow you should be able to run the swag command:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 $ swag -h NAME: swag - Automatically generate RESTful API documentation with Swagger 2.0 for Go. USAGE: swag [global options] command [command options] [arguments...] VERSION: v1.8.1 COMMANDS: init, i Create docs.go fmt, f format swag comments help, h Shows a list of commands or help for one command GLOBAL OPTIONS: --help, -h show help (default: false) --version, -v print the version (default: false) Documenting the Gin Server # When you look at Swagger UI documentation for any OpenAPI enabled website, you\u0026rsquo;d see something like this:\nPetstore Swagger UI Docs You might be familiar with the low half part of the UI. This is what Swagger docs are for i.e. to document the routes.\nBut before we deal with the lower 50% part I want you to notice in the above picture is the top 50% part. Right from the Swagger Petstore header to Find out more about the Swagger anchor link.\nThis part shows the metadata about the API server itself. In this section, we are going to build that part. You can find the code to start with here. We\u0026rsquo;ll be continuing on that code.\nFirst of all, we need to go get the packages we need to work with:\ngo get -u github.com/swaggo/files go get -u github.com/swaggo/gin-swagger Now after we have done that, let\u0026rsquo;s look at our main.go file. This is the main.go at the moment:\n1 2 3 4 5 6 7 8 9 package main import \u0026#34;github.com/santosh/gingo/routes\u0026#34; func main() { router := routes.SetupRouter() router.Run(\u0026#34;:8080\u0026#34;) } Add a route for Swagger Docs # We add a new router in the main function:\nrouter.GET(\u0026quot;/docs/*any\u0026quot;, ginSwagger.WrapHandler(swaggerFiles.Handler)) The modified file would look like this:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 -import \u0026#34;github.com/santosh/gingo/routes\u0026#34; +import ( + \u0026#34;github.com/santosh/gingo/routes\u0026#34; + swaggerFiles \u0026#34;github.com/swaggo/files\u0026#34; + ginSwagger \u0026#34;github.com/swaggo/gin-swagger\u0026#34; +) func main() { router := routes.SetupRouter() + router.GET(\u0026#34;/docs/*any\u0026#34;, ginSwagger.WrapHandler(swaggerFiles.Handler)) + router.Run(\u0026#34;:8080\u0026#34;) } swaggerFiles.Handler here is the handler which has all the assets embedded in it. Assets like the HTML, CSS, and javascript files are shown on the documentation end.\nginSwagger.WrapHandler is a wrapper around http.Handler, but for Gin.\nThis should be enough for testing. Let\u0026rsquo;s see how the docs render at \u0026lt;server address\u0026gt;/docs/index.html.\nDocs loaded, API definition didn\u0026rsquo;t load The good news is API documentation site is up. The bad news is it does not look like a normal API documentation site. What did we miss?\nGenerate swagger docs every time you modify doc string # Swagger documentation is stored in Go\u0026rsquo;s own docstring. We have a special syntax that we follow. More on that later. But first, let us learn how to generate docs.\n1 2 3 4 5 6 $ swag init 2022/05/25 23:59:16 Generate swagger docs.... 2022/05/25 23:59:16 Generate general API Info, search dir:./ 2022/05/25 23:59:16 create docs.go at docs/docs.go 2022/05/25 23:59:16 create swagger.json at docs/swagger.json 2022/05/25 23:59:16 create swagger.yaml at docs/swagger.yaml We run swag init every time we update docs for our API. This generates 3 files inside a sub directory called docs/.\n1 2 3 4 5 6 7 $ tree docs docs ├── docs.go ├── swagger.json └── swagger.yaml 0 directories, 3 files swagger.json and swagger.yaml are the actual specification which you can upload to services like AWS API Gateway and similar services. docs.go is a glue code that we need to import into our server.\nLet us now import the documentation to our main.go.\n1 2 3 4 5 6 7 8 9 10 11 12 package main import ( + _ \u0026#34;github.com/santosh/gingo/docs\u0026#34; \u0026#34;github.com/santosh/gingo/routes\u0026#34; swaggerFiles \u0026#34;github.com/swaggo/files\u0026#34; ginSwagger \u0026#34;github.com/swaggo/gin-swagger\u0026#34; ) +// @title Gingo Bookstore API func main() { router := routes.SetupRouter() As you can see we have imported docs the module by giving the full path of the module. We also have to prepend this import using _ because it is not explicitly used in main.go file.\nYou might also have noticed // @title Gingo Bookstore API line. Please note its position as this is important. It is just above the main() func. Also @title is not random here. It is one of the keywords which is documented under General API Info. We\u0026rsquo;ll see more of these as we go, but for now, let\u0026rsquo;s regenerate our docs and review the API docs.\nWorking API Docs Looks like we are getting somewhere. :D\nAdd General API Info to Gin API Server # We saw @title annotation in the last section. It is used to set the title for the API server. But it is not the only annotation available. There is a wide array of annotation in Swagger. You can find them in use in real life here.\nI\u0026rsquo;m going to use some of them to construct metadata on my doc site.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 // @title Gin Book Service // @version 1.0 // @description A book management service API in Go using Gin framework. // @termsOfService https://tos.santoshk.dev // @contact.name Santosh Kumar // @contact.url https://twitter.com/sntshk // @contact.email sntshkmr60@gmail.com // @license.name Apache 2.0 // @license.url http://www.apache.org/licenses/LICENSE-2.0.html // @host localhost:8080 // @BasePath /api/v1 Run swag init. Re-run the server. Check for update: Swagger API Server with Metadata That\u0026rsquo;s some metadata in there.\nYou might also have noticed that the API spec inside docs/ directory has changed. For example, here is the swagger.yaml file from the docs/ dir.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 { \u0026#34;swagger\u0026#34;: \u0026#34;2.0\u0026#34;, \u0026#34;info\u0026#34;: { - \u0026#34;title\u0026#34;: \u0026#34;Gingo Bookstore API\u0026#34;, - \u0026#34;contact\u0026#34;: {} + \u0026#34;description\u0026#34;: \u0026#34;A book management service API in Go using Gin framework.\u0026#34;, + \u0026#34;title\u0026#34;: \u0026#34;Gin Book Service\u0026#34;, + \u0026#34;termsOfService\u0026#34;: \u0026#34;https://tos.santoshk.dev\u0026#34;, + \u0026#34;contact\u0026#34;: { + \u0026#34;name\u0026#34;: \u0026#34;Santosh Kumar\u0026#34;, + \u0026#34;url\u0026#34;: \u0026#34;https://twitter.com/sntshk\u0026#34;, + \u0026#34;email\u0026#34;: \u0026#34;sntshkmr60@gmail.com\u0026#34; + }, + \u0026#34;license\u0026#34;: { + \u0026#34;name\u0026#34;: \u0026#34;Apache 2.0\u0026#34;, + \u0026#34;url\u0026#34;: \u0026#34;http://www.apache.org/licenses/LICENSE-2.0.html\u0026#34; + }, + \u0026#34;version\u0026#34;: \u0026#34;1.0\u0026#34; }, + \u0026#34;host\u0026#34;: \u0026#34;localhost:8080\u0026#34;, + \u0026#34;basePath\u0026#34;: \u0026#34;/api/v1\u0026#34;, \u0026#34;paths\u0026#34;: {} } Now it\u0026rsquo;s time to add docs for API endpoints.\nAdd API Operation Info to API Endpoints # Just like General API Info for the main server, there also is API Operation for individual routes/endpoints handlers.\nThey are more diverse than what I\u0026rsquo;m going to use here. The ones I\u0026rsquo;m going to use are just a subset of them. If you want to check out some real examples, you can find them here on each of the route handlers.\nHere is modified handlers/books.go:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 \u0026#34;github.com/santosh/gingo/models\u0026#34; ) -// GetBooks responds with the list of all books as JSON. +// GetBooks godoc +// @Summary Get books array +// @Description Responds with the list of all books as JSON. +// @Tags books +// @Produce json +// @Success 200 {array} models.Book +// @Router /books [get] func GetBooks(c *gin.Context) { c.JSON(http.StatusOK, db.Books) } -// PostBook takes a book JSON and store in DB. +// PostBook godoc +// @Summary Store a new book +// @Description Takes a book JSON and store in DB. Return saved JSON. +// @Tags books +// @Produce json +// @Param book body models.Book true \u0026#34;Book JSON\u0026#34; +// @Success 200 {object} models.Book +// @Router /books [post] func PostBook(c *gin.Context) { var newBook models.Book @@ -28,7 +41,14 @@ func PostBook(c *gin.Context) { c.JSON(http.StatusCreated, newBook) } -// GetBookByISBN locates the book whose ISBN value matches the isbn +// GetBookByISBN godoc +// @Summary Get single book by isbn +// @Description Returns the book whose ISBN value matches the isbn. +// @Tags books +// @Produce json +// @Param isbn path string true \u0026#34;search book by isbn\u0026#34; +// @Success 200 {object} models.Book +// @Router /books/{isbn} [get] func GetBookByISBN(c *gin.Context) { isbn := c.Param(\u0026#34;isbn\u0026#34;) Run swag init. Re-run the server. Check for updates: Swagger UI; with documented routes And here is the updated swagger.yaml:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 basePath: /api/v1 +definitions: + models.Book: + properties: + author: + type: string + isbn: + type: string + title: + type: string + type: object host: localhost:8080 info: contact: @@ -12,5 +22,58 @@ info: termsOfService: https://tos.santoshk.dev title: Gin Book Service version: \u0026#34;1.0\u0026#34; -paths: {} +paths: + /books: + get: + description: Responds with the list of all books as JSON. + produces: + - application/json + responses: + \u0026#34;200\u0026#34;: + description: OK + schema: + items: + $ref: \u0026#39;#/definitions/models.Book\u0026#39; + type: array + summary: Get books array + tags: + - books + post: + description: Takes a book JSON and store in DB. Return saved JSON. + parameters: + - description: Book JSON + in: body + name: book + required: true + schema: + $ref: \u0026#39;#/definitions/models.Book\u0026#39; + produces: + - application/json + responses: + \u0026#34;200\u0026#34;: + description: OK + schema: + $ref: \u0026#39;#/definitions/models.Book\u0026#39; + summary: Store a new book + tags: + - books + /books/{isbn}: + get: + description: Returns the book whose ISBN value matches the isbn. + parameters: + - description: search book by isbn + in: path + name: isbn + required: true + type: string + produces: + - application/json + responses: + \u0026#34;200\u0026#34;: + description: OK + schema: + $ref: \u0026#39;#/definitions/models.Book\u0026#39; + summary: Get single book by isbn + tags: + - books swagger: \u0026#34;2.0\u0026#34; Notice the new definitions and paths section added. It is for models and the endpoints respectively.\nBonus: Router Group for /api/v1 # It is always a good idea to prepend your routes with api/v1. v1 bit has a logic behind it. It is for the time when you have to introduce a breaking change which is not backward compatible. Maybe the input or the output has changes that might break millions of dependent client.\nIn these situations, you increment the version to something like api/v2 and let the older API server serve old from the old handler.\nRight now all of the routes in gingo server start with /book. We are going to change that.\nHere is the modified routes/routes.go.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 func SetupRouter() *gin.Engine { router := gin.Default() - router.GET(\u0026#34;/books\u0026#34;, handlers.GetBooks) - router.GET(\u0026#34;/books/:isbn\u0026#34;, handlers.GetBookByISBN) - // router.DELETE(\u0026#34;/books/:isbn\u0026#34;, handlers.DeleteBookByISBN) - // router.PUT(\u0026#34;/books/:isbn\u0026#34;, handlers.UpdateBookByISBN) - router.POST(\u0026#34;/books\u0026#34;, handlers.PostBook) + + v1 := router.Group(\u0026#34;/api/v1\u0026#34;) + { + v1.GET(\u0026#34;/books\u0026#34;, handlers.GetBooks) + v1.GET(\u0026#34;/books/:isbn\u0026#34;, handlers.GetBookByISBN) + // router.DELETE(\u0026#34;/books/:isbn\u0026#34;, handlers.DeleteBookByISBN) + // router.PUT(\u0026#34;/books/:isbn\u0026#34;, handlers.UpdateBookByISBN) + v1.POST(\u0026#34;/books\u0026#34;, handlers.PostBook) + } return router } Here we use router.Group to create a group with a path of /api/v1. We then move all the route definitions into the group and surround it with braces. That is how we create a path route in Gin.\nConclusion # API documentation is an essential part of API documentation. Instead of documenting the endpoints anywhere else, we can document the routes right in the code. That way we only have 1 single source of truth. No need to maintain code and documentation separately. In turn, we get always up-to-date documentation.\nEvery backend server has some sort of support for Swagger UI. I have covered the basics for Gin, but if you use any other framework, I encourage you to look for your own framework.\n","date":"26 May 2022","externalUrl":null,"permalink":"/posts/2022/how-to-integrate-swagger-ui-in-go-backend-gin-edition/","section":"Posts","summary":"Introduction # Unlike FastAPI, Gin does not have OpenAPI integration built in. With FastAPI when you add a route, documentation is already generated. With Gin, this is not the case. But recently I integrated Swagger UI in one of my Go backends and I wanted to document that process.\nPrerequisites # An already existing Gin server You might consider Building a Book Store API in Golang With Gin if you are starting from scratch. I’ll be using the same book store and extending over it to integrate Swagger UI.\n","title":"How to Integrate Swagger UI in Go Backend - Gin Edition","type":"posts"},{"content":"","date":"26 May 2022","externalUrl":null,"permalink":"/tags/openapi/","section":"Tags","summary":"","title":"Openapi","type":"tags"},{"content":"","date":"26 May 2022","externalUrl":null,"permalink":"/tags/swagger/","section":"Tags","summary":"","title":"Swagger","type":"tags"},{"content":"","date":"21 May 2022","externalUrl":null,"permalink":"/tags/cloud/","section":"Tags","summary":"","title":"Cloud","type":"tags"},{"content":" Introduction # In last post we learned to install and configure Portainer to maintain your local Docker environment. Although I\u0026rsquo;m not against command line, Portainer is a great companion if you don\u0026rsquo;t prefer to remember long commands to manager your docker. In this post I\u0026rsquo;m going to extend on last post and show how can you manage your VPS docker environment from your local Portainer installation.\nPrerequisites # Portainer configured on your local machine If you don\u0026rsquo;t have portainer configured on your local machine, please read my last post regarding installing and configuring Portainer on local machine.\nInstall Portainer Agent on your VPS # Well, this post is even shorter than the last one. Just make sure you are ssh\u0026rsquo;ed into your VPS and run Portainer Agent with the following command:\ndocker run -d -p 9001:9001 --name portainer_agent --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/docker/volumes:/var/lib/docker/volumes portainer/agent:latest This command is very similar to the one we used for installing the Portainer server. What extra we are doing is we are also binding /var/lib/docker/volumes inside the container. The second difference is that this time we are using the portainer/agent image rather than portainer/portainer-ce.\nNote:\nAlthough the installation is finished. You also need to expose port 9001 of your VPS so that you can connect to it over TCP. For example, I am using AWS and I have created a new security group and allowed only my home IP to connect to it.\nIf your VPS OS has some OS level firewall, please allow port 9001 to open as well.\nConfigure Portainer Agent on your local system # We are back to our local installation. Here are the steps your should follow to add a new environment.\nFrom the sidebar, choose Environments. On the screen which appears, click on Add environment button. You shall see something like this: Portainer add agent environment screen The first field Name is for identification purpose among multiple environment. You can keep this anything you want.\nSecond one is Environment URL, here you need to put public IP of your VPS appended by :9001.\nWhen you are done with it. Click on Add environment.\nNow when you visit the home screen. You\u0026rsquo;d see both the environment listed to you.\nTwo Portainer environment added Conclusion # This was a quick post I wanted to share. I hope it was helpful to you. If you like this post, please subscribe to the newsletter, connect with me on LinkedIn and follow me on Twitter.\n","date":"21 May 2022","externalUrl":null,"permalink":"/posts/2022/manage-your-vps-docker-environment-with-your-local-portainer/","section":"Posts","summary":"Introduction # In last post we learned to install and configure Portainer to maintain your local Docker environment. Although I’m not against command line, Portainer is a great companion if you don’t prefer to remember long commands to manager your docker. In this post I’m going to extend on last post and show how can you manage your VPS docker environment from your local Portainer installation.\n","title":"Manage Docker Environment on VPS with your Local Portainer","type":"posts"},{"content":" Introduction # This is a quick post where I demonstrate how to install Portainer on Ubuntu. Regardless this tutorial is demonstrated on Ubuntu, you can run Portainer on whatever Docker host you are using including Windows.\nMost awaited Docker Desktop is available for Linux now. Still, it is a desktop application. There is also an alternative that you already might have heard of if you have worked with Kubernetes. It is called Portainer.\nIf you have not heard of it, you are at the right place. We are going to install Portainer on our local machine to monitor and manage the Docker environment on our local\nBenefits of Portainer # You can manage multiple instances of Docker, including one on your local machine and your servers. Provider UI to manage stacks (docker-compose.yml) Install Portainer on your local machine # You need to have Docker already installed on your system. Head over to installation instruction if you are doubtful.\nIf you are done with the above setup. You can proceed by running the following command; which is a Portainer image.\ndocker run -d -p 8000:8000 -p 9000:9000 --name=portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce I hope you can understand the command pretty well as there is no rocket science in it. The important part is /var/run/docker.sock of the host machine should be available to the portainer container to make changes on behalf of the host machine. We are also using a separate volume called portainer_data and binding it to /data in the container.\nWait for the image to download and the container to run. After the process is finished, you\u0026rsquo;d be returned with the hash of the container.\n77d5143780c6e3369dc26e6d6393f6c2a131b2fdb59a5c1256c42972ae552a0f Configure Portainer # Use the hash obtained in the last step to find where exactly the container is running in the bridge network.\ndocker inspect 77d5143780c6e3369dc26e6d6393f6c2a131b2fdb59a5c1256c42972ae552a0f | grep IPAddress You\u0026rsquo;d see a something like this:\n1 2 3 \u0026#34;SecondaryIPAddresses\u0026#34;: null, \u0026#34;IPAddress\u0026#34;: \u0026#34;172.17.0.2\u0026#34;, \u0026#34;IPAddress\u0026#34;: \u0026#34;172.17.0.2\u0026#34;, Use this address to construct a socket. It will be 172.17.0.2:9000 in my case as Portainer runs on port 9000.\nUser Setup When you reach that address, you\u0026rsquo;d see this screen:\nPortainer first screen Environment Setup If you proceed from that screen, you\u0026rsquo;d reach here:\nEnvironment setup wizard As we are configuring the local environment, we\u0026rsquo;d just click on Get Started and proceed.\nThat is all. You can see your images and containers right on your browser.\nPortainer walkthrough Now you can roam around and explore portainer. In a future post, I\u0026rsquo;ll demonstrate setting up Prometheus and Grafana with Portainer.\nConclusion # Portainer is not only for managing your local Docker workflow. You can manage Kubernetes, Nomad, and whatnot. Please head over to Portainer site for more info.\nIn next post, we are going to see how we can configure and manage a remote machine\u0026rsquo;s docker locally. See you later.\n","date":"18 May 2022","externalUrl":null,"permalink":"/posts/2022/install-and-configure-portainer-on-your-local-machine/","section":"Posts","summary":"Introduction # This is a quick post where I demonstrate how to install Portainer on Ubuntu. Regardless this tutorial is demonstrated on Ubuntu, you can run Portainer on whatever Docker host you are using including Windows.\nMost awaited Docker Desktop is available for Linux now. Still, it is a desktop application. There is also an alternative that you already might have heard of if you have worked with Kubernetes. It is called Portainer.\n","title":"Install and Configure Portainer on your Local Machine","type":"posts"},{"content":"","date":"16 May 2022","externalUrl":null,"permalink":"/tags/grpc/","section":"Tags","summary":"","title":"Grpc","type":"tags"},{"content":" Introduction # The Internet has evolved in the last 2 decades a lot. HTTP/1.1 was not enough so we have HTTP2 now. The specification we used to transfer data between the client and the server has also evolved. From XML to JSON, now we have Protocol Buffer, which is a binary spec. Let\u0026rsquo;s dive deeper.\nThe picture below from Wikipedia shows how exponential data is growing.\nGrowth of data Prerequisites # Go HTTP2 Enough HTTP2 to know gRPC # During the early days of the web, we only had static content, hardly text files, and HTML files. After then, the web grew a bit and the web server and clients started to deal with rich media like rich texts, images, and videos. Servers also started to serve dynamic content based on what clients request.\nThis was the start of the RPC era. A client would hit a certain endpoint with certain data, and the server\u0026rsquo;s work was to respond to that request. In the early time of RPC, XML was extensively used. We still use XML in some systems. But most of the world has moved to JSON as a communication format between the server and the client.\nNow, over time, everything has evolved. One of the things which evolved is HTTP spec. Now we have HTTP2. HTTP2 is the base of gRPC. Certain properties make HTTP2 a nurturing ground for gRPC.\nHTTP/1.1 HTTP/2 Header is plaintext and not compressed Header is compressed and binary Spawns a new TCP connection on each request uses already existing TCP connection TLS is not required Requires TLS by default, enhanced security Let\u0026rsquo;s learn about Protocol Buffer first # We have seen how HTTP2 is essential for gRPC, now let\u0026rsquo;s see where Protocol Buffer stands.\nAlthough you can use gRPC with JSON, Protocol Buffer brings new things to the table. We saw that for a long time, JSON was used to send data back and forth between servers and clients. Now, what has happened is that we have taken yet another step to move from JSON to its successor.\nProtocol buffers are building block of gRPC and is a replacement for JSON. Protocol Buffer inherits a lot from HTTP/2.\nJSON Protocol Buffer Plaintext Binary Larger payload over wire Smaller payload over wire Write your server .proto files can create server stub As we already know, HTTP/2 is binary. So are protocol buffers. Now you don\u0026rsquo;t need to pass a date as a string in JSON. You can pass BSON over the wire. It is easy on a network as we only use the space we need to use. Say int32 only uses 4 bytes of data. The same data in JSON (string) could have used multiple bytes for a longer integer value. We write .proto files. the proto compiler generates stub files which can be used to write the server as well as the client. The key takeaway here is we can use the same proto files to generate clients and servers in many different languages. Protocol Buffers are agnostic to the language you are working with to develop your server or the client. .proto files have their syntax and data types which convert to specific data types in a destination programming language. This is my favorite reason to use gRPC in my projects. I recommend you to read through the proto syntax, keywords, and data types here: https://developers.google.com/protocol-buffers/docs/proto3\nWhat is gRPC? # Ever heard of RPC? It stands for Remote Procedure Call. It\u0026rsquo;s an old way of running a remote procedure on a remote machine. Let me make it a little familiar for you. When you hit an endpoint from a frontend to a backend, you are making a remote call or a remote procedure call.\nSOAP and REST both are an example of RPC. You can send data in the body and hit an API on the other end. That\u0026rsquo;s how it has been happening since the start.\ngRPC is a continuation of that SOAP and REST. gRPC brings all the advancements that their ancestors can\u0026rsquo;t. With gRPC, a client application can directly call a method on a server application on a different machine as if it were a local object.\nTypes of Service Methods in gRPC # We\u0026rsquo;ll practically see what service methods are when we do hands-on with code. But right now I want to specify that in gRPC we have 4 kinds of method definitions.\nTypes of Service Methods Unary - Unary is similar to a normal REST call. A client initiates a TCP connection, sends a message, waits for the server to respond, and finally, the server responds.\nServer Streaming - Server streaming RPCs where the client sends a request to the server and gets a stream to read a sequence of messages back. For example, you searched for a keyword. Instead of returning a static page, Twitter returns a stream of tweets, including whatever is being tweeted in real time.\nClient Streaming - Client streaming RPCs where the client writes a sequence of messages and sends them to the server. Once the client has finished writing the messages, it waits for the server to read them and return its response. Again gRPC guarantees message ordering within an individual RPC call. An example of it would be an IoT device (e.g. a car) streaming its device location to the central server (e.g. Uber).\nBidirectional Streaming - Bidirectional streaming RPCs where both sides send a sequence of messages using a read-write stream. The two streams operate independently, so clients and servers can read and write in whatever order they like. An example of this would be a chat application. But I know there are more complex use cases available in the wild. Please let me know in the comments if you do so.\nFor a detailed explanation of what happens when a gRPC client calls a gRPC server method, please consider reading RPC life cycle.\nDeveloping with gRPC # We have had enough of theories. Let\u0026rsquo;s develop some code to see this thing in action.\nWe are going to have an example in Go. Although as we already know, the proto files have a language of their own which is used to generate code in multiple languages.\nWe are going to write a simple calculator app.\nWrite the proto file # .proto files are contracts between the server and the client. This is similar to the REST API you have already experienced with.\nBefore I proceed further, I\u0026rsquo;d like to inform you that having the whole code into a module will make your life easy. That\u0026rsquo;s the reason, I\u0026rsquo;ve created a go module at the root of the directory named github.com/santosh/example.\ncalculator/calculator.proto # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 syntax = \u0026#34;proto3\u0026#34;; option go_package = \u0026#34;github.com/santosh/example/calculator\u0026#34;; package calculator; // The calculator service definition. service Calculator { // Adds two number rpc Add(Input) returns (Output); } // Input message containing two operands message Input { int32 Operand1 = 1; int32 Operand2 = 2; } // Output message containing result of operation message Output { int32 Result = 1; } Explanation # Protocol buffer data is structured as messages, where each message is a small logical record of information containing a series of name-value pairs called fields. Line 14-17 is an example of a message. So is 20-22. Messages are comprised of data types, identifiers, and index positions.\nYou\u0026rsquo;d also note messages are composed inside Services. Service is simply what this web service does. We currently have a Calculator service from lines 8-11. Calculator service comprises of a method called Add. Add takes 2 parameters as defined in Input message and emits an Output message.\nGenerate code from a proto file # You\u0026rsquo;d need a protoc compiler to generate code from proto files. If you are on Debian based system, you can use sudo apt install protobuf-compiler. If you are on any other OS, please read Protocol Buffer Compiler Installation\nI am going to use Go for this tutorial, so I\u0026rsquo;m going to install the Go plugin for the protoc compiler.\n1 2 go install google.golang.org/protobuf/cmd/protoc-gen-go@v1.28 go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@v1.2 Now with everything in place, I\u0026rsquo;d issue this command from the root of the module:\nprotoc --go_out=. --go_opt=paths=source_relative --go-grpc_out=. --go-grpc_opt=paths=source_relative calculator/calculator.proto If this command is successful, this would generate 2 more go files.\n1 2 3 4 5 6 7 $ tree calculator calculator ├── calculator_grpc.pb.go ├── calculator.pb.go └── calculator.proto 0 directories, 3 files If this is your first time with gRPC, and you look at the generated code now, you\u0026rsquo;d probably be lost.\nWe are going to use both files to create our server and the client. The generated code will start to make sense.\nImplement calculator server and the client # We all know that the output generated by the protoc compiler is a stub. We need to use that stub as a guide to implementing/writing our client and the server code. The stub work as a guideline for us.\nRight now, the generated code lives inside calculator package in calculator.pb.go which mostly deals with data part (i.e. Input, Output, Operand1, Operand2, their getters etc) and calculator_grpc.pb.go which mostly deals with method implementation (i.e Add in both server and client). The first thing in both server and client code is to import this package.\nI will keep referencing the proto file as we define our client/server code.\nWrite gRPC calculator server # If you look into calculator_grpc.pb.go, you\u0026rsquo;d find that there is a struct called UnimplementedCalculatorServer. This struct represents our server. Now there is a reason it is named Unimplemented. If you look at methods attached to this struct, you\u0026rsquo;d see a method named Add. This is the same method we defined in our proto file. Here is a refresher:\n// Adds two number rpc Add(Input) returns (Output); What we are going to do is we are going to take this UnimplementedCalculatorServer and implement the Add method.\n18 19 20 21 22 23 24 25 26 27 28 // server is used to implement calculator.CalculatorServer. type server struct { pb.UnimplementedCalculatorServer } // Add implements calculator.CalculatorServer func (s *server) Add(ctx context.Context, in *pb.Input) (*pb.Output, error) { log.Printf(\u0026#34;Received: %v %v\u0026#34;, in.GetOperand1(), in.GetOperand2()) result := in.GetOperand1() + in.GetOperand2() return \u0026amp;pb.Output{Result: result}, nil } Pay attention to the signature of the method. We are taking Input in form of *pb.Input and returning Output in form of *pb.Output. This is very the same as we declared in the proto file.\nThe Add implementation is incomplete without Add logic. On line 26 we are using GetOperand1 and GetOperand2 which is available from the calculator.pb.go file. At last, we use the Output struct to return the result.\nImplementing the method is not enough, we also need to start the gRPC server and start listening.\n30 31 32 33 34 35 36 37 38 39 40 41 42 func main() { flag.Parse() lis, err := net.Listen(\u0026#34;tcp\u0026#34;, fmt.Sprintf(\u0026#34;:%d\u0026#34;, *port)) if err != nil { log.Fatalf(\u0026#34;failed to listen: %v\u0026#34;, err) } s := grpc.NewServer() pb.RegisterCalculatorServer(s, \u0026amp;server{}) log.Printf(\u0026#34;server listening at %v\u0026#34;, lis.Addr()) if err := s.Serve(lis); err != nil { log.Fatalf(\u0026#34;failed to serve: %v\u0026#34;, err) } } On line 32 I\u0026rsquo;m starting to listen to TCP connection on a given port. grpc.NewServer() calls the internal library function to create a new gRPC server, this call returns a grpc.ServiceRegistrar object. This object is then passed to RegisterCalculatorServer along with our implemented methods.\nThe whole code looks like this:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 package main import ( \u0026#34;context\u0026#34; \u0026#34;flag\u0026#34; \u0026#34;fmt\u0026#34; \u0026#34;log\u0026#34; \u0026#34;net\u0026#34; pb \u0026#34;github.com/santosh/example/calculator\u0026#34; \u0026#34;google.golang.org/grpc\u0026#34; ) var ( port = flag.Int(\u0026#34;port\u0026#34;, 50051, \u0026#34;The server port\u0026#34;) ) // server is used to implement calculator.CalculatorServer. type server struct { pb.UnimplementedCalculatorServer } // Add implements calculator.CalculatorServer func (s *server) Add(ctx context.Context, in *pb.Input) (*pb.Output, error) { log.Printf(\u0026#34;Received: %v %v\u0026#34;, in.GetOperand1(), in.GetOperand2()) result := in.GetOperand1() + in.GetOperand2() return \u0026amp;pb.Output{Result: result}, nil } func main() { flag.Parse() lis, err := net.Listen(\u0026#34;tcp\u0026#34;, fmt.Sprintf(\u0026#34;:%d\u0026#34;, *port)) if err != nil { log.Fatalf(\u0026#34;failed to listen: %v\u0026#34;, err) } s := grpc.NewServer() pb.RegisterCalculatorServer(s, \u0026amp;server{}) log.Printf(\u0026#34;server listening at %v\u0026#34;, lis.Addr()) if err := s.Serve(lis); err != nil { log.Fatalf(\u0026#34;failed to serve: %v\u0026#34;, err) } } In above implementation, we have used flag library to pass port from command line.\nWrite gRPC calculator client # Like we have done with server code, we\u0026rsquo;ll start by defining command line flags.\n14 15 16 17 18 19 20 21 var ( addr = flag.String(\u0026#34;addr\u0026#34;, \u0026#34;localhost:50051\u0026#34;, \u0026#34;the address to connect to\u0026#34;) operand1 = flag.Int(\u0026#34;op1\u0026#34;, 2, \u0026#34;1st operand\u0026#34;) operand2 = flag.Int(\u0026#34;op2\u0026#34;, 2, \u0026#34;2nd operand\u0026#34;) operand1int32 = int32(*operand1) operand2int32 = int32(*operand2) ) Line 19-20 here are kind of a hack as flag module does not have a way to accept int32 which is requirement for our Output.Result.\nNext is connecting with the server part:\n25 26 27 28 29 30 // Set up a connection to the server. conn, err := grpc.Dial(*addr, grpc.WithTransportCredentials(insecure.NewCredentials())) if err != nil { log.Fatalf(\u0026#34;did not connect: %v\u0026#34;, err) } defer conn.Close() grpc.Dial takes address of the server as well as other variadic parameters. As gRPC works over HTTP2 which is by default requires TLS, we are using insecure credentials as we have not configured our server to use certificates.\nOn the next line, we are going to create a new gRPC client:\n31 c := pb.NewCalculatorClient(conn) On preceding lines, we are going to invoke the Add endpoint:\n33 34 35 36 37 38 39 40 // Contact the server and print out its response. ctx, cancel := context.WithTimeout(context.Background(), time.Second) defer cancel() r, err := c.Add(ctx, \u0026amp;pb.Input{Operand1: operand1int32, Operand2: operand2int32}) if err != nil { log.Fatalf(\u0026#34;could not add: %v\u0026#34;, err) } log.Printf(\u0026#34;Add result: %v\u0026#34;, r.GetResult()) The entirity of the code looks like this:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 package main import ( \u0026#34;context\u0026#34; \u0026#34;flag\u0026#34; \u0026#34;log\u0026#34; \u0026#34;time\u0026#34; pb \u0026#34;github.com/santosh/example/calculator\u0026#34; \u0026#34;google.golang.org/grpc\u0026#34; \u0026#34;google.golang.org/grpc/credentials/insecure\u0026#34; ) var ( addr = flag.String(\u0026#34;addr\u0026#34;, \u0026#34;localhost:50051\u0026#34;, \u0026#34;the address to connect to\u0026#34;) operand1 = flag.Int(\u0026#34;op1\u0026#34;, 2, \u0026#34;1st operand\u0026#34;) operand2 = flag.Int(\u0026#34;op2\u0026#34;, 2, \u0026#34;2nd operand\u0026#34;) operand1int32 = int32(*operand1) operand2int32 = int32(*operand2) ) func main() { flag.Parse() // Set up a connection to the server. conn, err := grpc.Dial(*addr, grpc.WithTransportCredentials(insecure.NewCredentials())) if err != nil { log.Fatalf(\u0026#34;did not connect: %v\u0026#34;, err) } defer conn.Close() c := pb.NewCalculatorClient(conn) // Contact the server and print out its response. ctx, cancel := context.WithTimeout(context.Background(), time.Second) defer cancel() r, err := c.Add(ctx, \u0026amp;pb.Input{Operand1: operand1int32, Operand2: operand2int32}) if err != nil { log.Fatalf(\u0026#34;could not add: %v\u0026#34;, err) } log.Printf(\u0026#34;Add result: %v\u0026#34;, r.GetResult()) } Let\u0026rsquo;s go ahead and test our service.\nDemo our server and the client # I have recorded a gif for the demo.\ngRPC calculator demo Here I have demonstrated with the default flag value, but you can override -op1 and -op2 flag to see different results.\nConclusion # This was merely an introduction to gRPC and protocol buffer. What I\u0026rsquo;ve found is different than a conventional REST API is the endpoints. In REST, we have a predefined endpoint to hit. Such as /calculation/add. We would then pass JSON with the first and the second operand.\nThis is not the case with gRPC. We get the stub files as the protoc artifact. The only communication is from there.\nIn this post, I have only covered the Unary service method types. I\u0026rsquo;ll leave streaming for some other time. If you\u0026rsquo;d like to read and practice more gRPC, I\u0026rsquo;d found this series very helpful.\nExternal Links # Protocol Buffers Proto3 documentation GRPC official Documentation What does multiplexing mean in HTTP/2 ","date":"16 May 2022","externalUrl":null,"permalink":"/posts/2022/grpc-for-absolute-beginners-in-go/","section":"Posts","summary":"Introduction # The Internet has evolved in the last 2 decades a lot. HTTP/1.1 was not enough so we have HTTP2 now. The specification we used to transfer data between the client and the server has also evolved. From XML to JSON, now we have Protocol Buffer, which is a binary spec. Let’s dive deeper.\nThe picture below from Wikipedia shows how exponential data is growing.\n","title":"gRPC for Absolute Beginners, in Go","type":"posts"},{"content":"","date":"16 May 2022","externalUrl":null,"permalink":"/tags/http2/","section":"Tags","summary":"","title":"Http2","type":"tags"},{"content":"","date":"16 May 2022","externalUrl":null,"permalink":"/tags/protobuf/","section":"Tags","summary":"","title":"Protobuf","type":"tags"},{"content":"","date":"30 April 2022","externalUrl":null,"permalink":"/tags/authentication/","section":"Tags","summary":"","title":"Authentication","type":"tags"},{"content":"","date":"30 April 2022","externalUrl":null,"permalink":"/tags/jwt/","section":"Tags","summary":"","title":"Jwt","type":"tags"},{"content":" Introduction # We have been using FastAPI along with the Test-Driven Development process to come up with an authentication system. Till now we can register to own tiny application using a username and password. We are also able to generate a JWT token to be classified as a logged-in user. But there is no point in logging in until the logged-in user can do something privileged. In this iteration of the series, we\u0026rsquo;d see how to access a protected endpoint using FastAPI and JWT. Given that we already have a JWT token.\nSeries Index # We have already seen:\nProject Setup and FastAPI introduction Database Setup and User Registration Fix the failing test with mocking and dependency injection Authentication Premier and Login Endpoint Token Validation and Accessing Protected Endpoints (you are here) What happens when you access protected routes? # I would like to start this section by saying that HTTP is stateless, meaning that applications written on top of HTTP won\u0026rsquo;t be able to identify a client if they have authorized. Meaning that if you have sent the server the correct username and password, they\u0026rsquo;d send you an authentication token. This token has to be sent with each request made to be identified as logged in.\nWhat happens when you have successfully logged in? # Let us start where we left off in the previous post, take a look at the below workflow.\nSuccessful login attempt On successful login, you are given a Bearer token. In the above gif, you can see that server responds with a JSON with access_token and token_type. The former contains the encoded JWT token.\nNow when you don\u0026rsquo;t send this token with every request, you\u0026rsquo;d be considered logged out. Here is what I\u0026rsquo;m trying to say in an image.\nTrying to access protected route with and without Bearer token Points to remember:\nOn successful login, the server sends a JWT token to the client. This token needs to be sent to the server in every request when trying to access a protected route. Access Protected Endpoints # For sake of contrast, we are going to have two endpoints, we will reuse our /ping endpoint, which we can access without being signed up or logged in. And then we\u0026rsquo;ll have a dummy /protected endpoint which will return a dummy JSON { \u0026quot;message\u0026quot;: \u0026quot;protected resource\u0026quot; }.\nLet\u0026rsquo;s first write down the requirements.\nRequirement for /protected endpoint # GET request to /protected without JWT token returns 403. GET request to /protected with JWT token returns 200. Requirements turned into tests # I am going to put this test inside tests/test_users.py along with other user-related tests.\ntests/test_users.py # 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 diff --git a/tests/test_users.py b/tests/test_users.py index 2239b5b..dbd5750 100644 --- a/tests/test_users.py +++ b/tests/test_users.py @@ -84,3 +84,31 @@ class TestUserLogin: ) assert response.status_code == 200 assert len(response.json()) == 2 + + +class TestProtectedEndpoints: + def test_get_request_without_JWT_token_returns_403(self): + \u0026#34;\u0026#34;\u0026#34;sends GET request to /protected without JWT token\u0026#34;\u0026#34;\u0026#34; + + response = client.get(\u0026#34;/protected\u0026#34;) + assert response.status_code == 403 + + def test_get_request_without_JWT_token_returns_200_and_body(self): + \u0026#34;\u0026#34;\u0026#34;sends GET request to /protected\u0026#34;\u0026#34;\u0026#34; + + response = client.post( + \u0026#34;/users/auth\u0026#34;, + json={\u0026#34;username\u0026#34;: \u0026#34;santosh\u0026#34;, \u0026#34;password\u0026#34;: \u0026#34;sntsh\u0026#34;} + ) + + login_response = response.json() + access_token = login_response[\u0026#39;access_token\u0026#39;] + token_type = login_response[\u0026#39;token_type\u0026#39;] + + response = client.get( + \u0026#34;/protected\u0026#34;, + headers={\u0026#39;Authorization\u0026#39;: \u0026#39;{} {}\u0026#39;.format(token_type, access_token)} + ) + + assert response.status_code == 200 + assert len(response.json()) == 1 Before we try to implement or even run the test, I would like to go through the definition of the tests to tell what they are doing.\nThe test_get_request_without_JWT_token_returns_403 route tries to access a protected route (route which we are yet to define). It returns a status code of 403 in return. This happens because we don\u0026rsquo;t have the Bearer token in the Authorization header.\nIf you don\u0026rsquo;t know about the Bearer token in Authorization, I\u0026rsquo;d recommend you read the previous post in this series.\nThe test_get_request_without_JWT_token_returns_200_and_body does exactly what the first test does. But this time, it first fetches the JWT token by sending a login request. Thereafter, it uses the token and token type to put inside the Authorization header. And when we send the GET request again, we see 200 in response.\nSo without further ado, let\u0026rsquo;s run the tests.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 $ pytest -k protected ========================== test session starts ========================== platform linux -- Python 3.8.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 rootdir: /efs/repos/fastauth plugins: anyio-3.4.0 collected 11 items / 9 deselected / 2 selected tests/test_users.py FF [100%] =============================== FAILURES ================================ _ TestProtectedEndpoints.test_get_request_without_JWT_token_returns_403 _ self = \u0026lt;tests.test_users.TestProtectedEndpoints object at 0x7f1401a20880\u0026gt; def test_get_request_without_JWT_token_returns_403(self): \u0026#34;\u0026#34;\u0026#34;sends GET request to /protected without JWT token\u0026#34;\u0026#34;\u0026#34; response = client.get(\u0026#34;/protected\u0026#34;) \u0026gt; assert response.status_code == 403 E assert 404 == 403 E + where 404 = \u0026lt;Response [404]\u0026gt;.status_code tests/test_users.py:94: AssertionError _ TestProtectedEndpoints.test_get_request_without_JWT_token_returns_200_and_body _ self = \u0026lt;tests.test_users.TestProtectedEndpoints object at 0x7f14019e6460\u0026gt; def test_get_request_without_JWT_token_returns_200_and_body(self): \u0026#34;\u0026#34;\u0026#34;sends GET request to /protected\u0026#34;\u0026#34;\u0026#34; response = client.post( \u0026#34;/users/auth\u0026#34;, json={\u0026#34;username\u0026#34;: \u0026#34;santosh\u0026#34;, \u0026#34;password\u0026#34;: \u0026#34;sntsh\u0026#34;} ) login_response = response.json() access_token = login_response[\u0026#39;access_token\u0026#39;] token_type = login_response[\u0026#39;token_type\u0026#39;] response = client.get( \u0026#34;/protected\u0026#34;, headers={\u0026#39;Authorization\u0026#39;: \u0026#39;{} {}\u0026#39;.format(token_type, access_token)} ) \u0026gt; assert response.status_code == 200 E assert 404 == 200 E + where 404 = \u0026lt;Response [404]\u0026gt;.status_code tests/test_users.py:113: AssertionError ======================== short test summary info ======================== FAILED tests/test_users.py::TestProtectedEndpoints::test_get_request_without_JWT_token_returns_403 FAILED tests/test_users.py::TestProtectedEndpoints::test_get_request_without_JWT_token_returns_200_and_body ==================== 2 failed, 9 deselected in 1.44s ==================== As we can see, we are getting 404 as the status code usual instead of 403 and 200 respectively. This is obvious because as usual, we haven\u0026rsquo;t implemented the route.\nWrite code to fix failing test # Before we write the actual route, we need to have some helper functions. In the last post, we saw how to encode some data, along with the expiry time of a token. Now we are going to do the reverse of that.\nfastauth/auth.py # 34 35 36 37 def decode_jwt_token(token: str): decoded_token = jwt.decode(token, SECRET_KEY, algorithms=JWT_ALGORITHM) print(decoded_token) return decoded_token if decoded_token[\u0026#39;exp\u0026#39;] \u0026gt;= time.time() else None This function takes in the token and returns decoded version if the expiry is remaining, otherwise returns None.\nNow in crude words, this decoded token needs to be shown to FastAPI with every request to identify the request as authorized; then only FastAPI is letting the client access the protected route.\nWe could have written that part manually, but we have a certain structure we have to follow with FastAPI. So we are going to write a class that inherits from HTTPBearer.\nBelow I\u0026rsquo;m going to post the entire diff along with the function I wrote above.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 diff --git a/fastauth/auth.py b/fastauth/auth.py index ead9785..6363f3c 100644 --- a/fastauth/auth.py +++ b/fastauth/auth.py @@ -1,6 +1,9 @@ import datetime +import time import bcrypt +from fastapi import Request, HTTPException +from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials import jwt from sqlalchemy.orm import Session @@ -26,3 +29,37 @@ def encode_jwt_token(*, data: dict, expires_delta: datetime.timedelta = None): encoded_jwt = jwt.encode(to_encode, SECRET_KEY, algorithm=JWT_ALGORITHM) return encoded_jwt + + +def decode_jwt_token(token: str): + decoded_token = jwt.decode(token, SECRET_KEY, algorithms=JWT_ALGORITHM) + print(decoded_token) + return decoded_token if decoded_token[\u0026#39;exp\u0026#39;] \u0026gt;= time.time() else None + + +class JWTBearer(HTTPBearer): + def __init__(self, auto_error: bool = True): + super(JWTBearer, self).__init__(auto_error=auto_error)+ + async def __call__(self, request: Request): + credentials : HTTPAuthorizationCredentials = await super(JWTBearer, self).__call__(request) + if credentials:+ if not credentials.scheme == \u0026#34;Bearer\u0026#34;: + raise HTTPException(status_code=401, detail=\u0026#34;Invalid authentication scheme.\u0026#34;) + if not self.verify_jwt(credentials.credentials): + raise HTTPException(status_code=401, detail=\u0026#34;Invalid token or expired token.\u0026#34;) + return credentials.credentials + else: + raise HTTPException(status_code=401, detail=\u0026#34;Invalid authentication code.\u0026#34;) + + def verify_jwt(self, jwtToken: str) -\u0026gt; bool: + isTokenValid: bool = False + + try: + payload = decode_jwt_token(jwtToken) + except: + payload = None + + if payload: + isTokenValid = True + return isTokenValid main.py # Let us come to the file with our route definition.\nRemember I told you there is a certain way we handle token validation in FastAPI. Yes. The class we wrote, we can use that class as a dependency to the route in the following way.\n49 50 51 52 @app.get(\u0026#34;/protected\u0026#34;, dependencies=[Depends(auth.JWTBearer())]) def get_protected_resource(): return { \u0026#34;message\u0026#34;: \u0026#34;protected resource\u0026#34; } The thing to note in the above code is that the dependencies=[Depends(auth.JWTBearer())]. dependencies array takes multiple dependencies of the route. We have our single JWTBearer class marked as a dependency to the /protected route.\nYou can assume dependencies similar to a middleware. What a middleware does is adds or removes more data into the request and then pass the data to the next middleware.\nI would ask you to read fastapi dependency vs middleware\nConfirm if tests are passing # Let us run only the required tests.\n1 2 3 4 5 6 7 8 9 10 $ pytest -k protected =================================== test session starts ==================================== platform linux -- Python 3.8.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 rootdir: /efs/repos/fastauth plugins: anyio-3.4.0 collected 11 items / 9 deselected / 2 selected tests/test_users.py .. [100%] ============================= 2 passed, 9 deselected in 1.66s ============================== Test manually # If you wish to, you are free to test it manually by not passing JWT token while accessing the protected route.\nThen in the next iteration, obtain a token, and put it inside the Authorization header with the authorization type as Bearer.\nBonus # I would like to give out the complete code we have written till now in a git repository.\nhttps://github.com/santosh/fastauth\nI have also gone ahead and included a Postman collection of endpoints to test. Feel free to play around with data and values to get a better understanding of what\u0026rsquo;s going on.\nConclusion # This post is a full stop to the 5 post series on this topic. I tried to cover the basics of Test Driven Development, Dependency Injection, and Mocking. I also had this itch to dig into information security. I did that by writing my first authentication system. I learned concepts related to authorization and authentication concerning HTTP while writing this series. I plan to keep digging into infosec space.\nThank you all of you if you have been following this post till now. I appreciate it and respect your time. I hope you have learned something new here. You can connect with on on LinkedIn, or say Hi on Twitter. You can also put your email into the below box to stay tuned with similar posts around the tech and programming space.\nHave a nice time ahead.\n","date":"30 April 2022","externalUrl":null,"permalink":"/posts/2022/tdd-approach-to-create-an-authentication-system-with-fastapi-part-5/","section":"Posts","summary":"Introduction # We have been using FastAPI along with the Test-Driven Development process to come up with an authentication system. Till now we can register to own tiny application using a username and password. We are also able to generate a JWT token to be classified as a logged-in user. But there is no point in logging in until the logged-in user can do something privileged. In this iteration of the series, we’d see how to access a protected endpoint using FastAPI and JWT. Given that we already have a JWT token.\n","title":"TDD Approach to Create an Authentication System With FastAPI Part 5","type":"posts"},{"content":"","date":"30 April 2022","externalUrl":null,"permalink":"/series/tdd-auth-with-fastapi/","section":"Series","summary":"","title":"TDD Auth With FastAPI","type":"series"},{"content":" Introduction # In the last couple of posts in TDD Auth with FastAPI we\u0026rsquo;ve been sustainably moved towards a web service that can let users register with the service. Now what?\nWe have already done the easy part. The next part is to look at the authorization. This involves letting the user log in. And only give access to what they are authorized for. Let us look at the login part first.\nBefore talking into login, I\u0026rsquo;d like to tell you that I have always found authentication and authorization challenging. There are so many terminologies involved. A few to name are OAuth, Basic Authentication, JWT, Cookies, SAML, and whatnot. I can\u0026rsquo;t explain all of them in this single post, but I\u0026rsquo;ll try to cover some basic building blocks to get you started.\nSeries Index # We have already seen:\nProject Setup and FastAPI introduction Database Setup and User Registration Fix the failing test with mocking and dependency injection Authentication Premier and Login Endpoint (you are here) Authentication Premier # What is Authentication? # Authentication in most basic terms is the process of validating an identity to ascertain that they are who they claim to be. By the way, authentication can be achieved using passwords, OTPs, biometrics, authentication apps, access tokens, certificates, and more. We\u0026rsquo;ll be authenticating using passwords and tokens.\nFollowing is a sequence diagram that roughly describes what happens in a successful login process.\nA sequence diagram for login Once you are authenticated, the server provides a token, this token then needs to be sent to the server in each request by the client. you are authorized to do certain activities on the application.\nDifferent kinds of authentication (and scenario) # There are mainly two kinds of application authenticating.\nUser to application authentication. When I say user to application authentication, I refer to a scenario where a user (using a client, like a web browser) is logging into an application.\nApplication to application authentication. App to app authentication is when you use one service (say Google) to sign in to another application (say appX). The \u0026ldquo;Signin with \u0026rdquo; button represents an example of app-to-app authentication.\nHere you can argue that at the end of the day user is going to be authenticated so this is a user to app authentication. But my point is, that we are being authenticated to an application, using the data from another application.\nWe are not talking about an app to app authentication in this post. For this post, I\u0026rsquo;ll only be exploring simple authentication where a user uses their username and password. And when username and password match, an authentication token is sent to the frontend. Their token is required to be identified as logged in and to be able to access restricted resources.\nIn this section we talk about some terminologies and how are they related to each other.\nHTTP and the Authorization Header # You must be knowing about different request and response headers when dealing with HTTP in general. Some of the common ones which are on top of my mind are:\nRequest header: User-Agent, Referer, Origin, Host, Content-Type, Content-Length, Accept. Response header: Status, Location, Content-Type, Cache-Control. Among those request headers, there is one more header called Authorization. If you are dealing with auth, the first thing you must do is learn more about this header. I will cover some of the basic info here and give you some links at the end of this section for further reading.\nAn Authorization header looks like this:\nAuthorization: \u0026lt;scheme\u0026gt; \u0026lt;credentials\u0026gt; \u0026lt;scheme\u0026gt; is something we\u0026rsquo;ll see in the next section. \u0026lt;credentials\u0026gt; are a piece of string, the value of which will depend on what the scheme is.\nhttps://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Authorization\nAuthentication Schemes # There are over half a dozen of authentication schemes, but these ones are common.\nBasic Bearer Digest Basic # Basic authentication scheme deals with username and password. In basic HTTP authentication, a request contains a header field in the form of:\nAuthorization: Basic username:password where credentials are base64 encoded so the actual header will look like this:\nAuthorization: Basic dXNlcm5hbWU6cGFzc3dvcmQ= Some final points:\nThe security of the Basic scheme is only dependent on the transport layer i.e. HTTPS. HTTP does not provide a method for a web server to instruct the client to \u0026ldquo;log out\u0026rdquo; the user. You should not be using the Basic scheme. There is a similar approach taken in the Digest scheme which you might consider. Bearer # Originally taken from OAuth, Bearer deals with tokens. The name \u0026ldquo;Bearer authentication\u0026rdquo; can be understood as \u0026ldquo;give access to the bearer of this token\u0026rdquo;. In terms of the header, the Bearer is not much different from Basic. It looks like\nAuthorization: Bearer \u0026lt;credentials\u0026gt; But instead of passing bare username:password in credentials, they are signed and encrypted. They are more secure than Basic because these tokens can have an expiry time set to them. When the server reads the token passed from a client and finds that it has expired, it marks the request as unauthorized. In that case, the client again has to generate the token.\nFinal points:\nAlthough Bearer is safer than Basic, it is highly advised to use them over HTTPS. We are going to discuss Bearer tokens only in this post. We\u0026rsquo;ll use JSON Web Token. If you are feeling crazy, read the spec of JWT to know about it in-depth. Bearer scheme in my opinion is the most widely used scheme on the internet. https://stackoverflow.com/questions/34013299/web-api-authentication-basic-vs-bearer\nDigest # Simply speaking, Digest is a complex and secure version of the Basic scheme. I have not used Digest at the moment and thus I\u0026rsquo;m not going to talk about it in this post. But if you want to read about it, please follow https://en.wikipedia.org/wiki/Digest_access_authentication.\nLogin Endpoint # Define requirement for authentication (/users/auth) endpoint # We have already seen in the above diagram that username and password go from the frontend application to the backend service endpoint. The client sending this password could be a web browser like Firefox, a CLI tool like curl, or even an API testing tool like Postman.\nNow let us describe the functional requirement of our /users/auth endpoint. Thinks about what our endpoints will look like in a finished product. Here is what I can think of:\nThe consumer hits /users/auth with the GET method and should get Method Not Allowed.\nThe API consumer hits /users/auth with a POST method:\nWithout body = returns unauthorized With body: If user not found OR username exists but the password is wrong, returns unauthorized Should check in the database if the user exists, if so, creates and returns the JWT token Functional requirements turned into test # Below are the test cases for the spec. This is a slightly modified version of the registration cases.\ntests/test_users.py # 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 @@ -54,3 +54,32 @@ class TestUserRegistration: json={\u0026#34;username\u0026#34;: \u0026#34;santosh\u0026#34;, \u0026#34;password\u0026#34;: \u0026#34;sntsh\u0026#34;, \u0026#34;fullname\u0026#34;: \u0026#34;Santosh Kumar\u0026#34;} ) assert response.status_code == 201 + + +class TestUserLogin: + \u0026#34;\u0026#34;\u0026#34;TestUserLogin tests /users/auth\u0026#34;\u0026#34;\u0026#34; + + def test_get_request_returns_405(self): + \u0026#34;\u0026#34;\u0026#34;login endpoint does only expect a post request\u0026#34;\u0026#34;\u0026#34; + response = client.get(\u0026#34;/users/auth\u0026#34;) + assert response.status_code == 405 + + def test_post_request_without_body_returns_422(self): + \u0026#34;\u0026#34;\u0026#34;body should have username, password and fullname\u0026#34;\u0026#34;\u0026#34; + response = client.post(\u0026#34;/users/auth\u0026#34;) + assert response.status_code == 422 + + def test_post_request_with_improper_body_returns_422(self): + \u0026#34;\u0026#34;\u0026#34;both username and password is required\u0026#34;\u0026#34;\u0026#34; + response = client.post( + \u0026#34;/users/auth\u0026#34;, + json={\u0026#34;username\u0026#34;: \u0026#34;santosh\u0026#34;} + ) + assert response.status_code == 422 + + def test_post_request_with_proper_body_returns_200_with_jwt_token(self): + response = client.post( + \u0026#34;/users/auth\u0026#34;, + json={\u0026#34;username\u0026#34;: \u0026#34;santosh\u0026#34;, \u0026#34;password\u0026#34;: \u0026#34;sntsh\u0026#34;} + ) + assert response.status_code == 200 + assert len(response.json()) == 2 And when you run these tests, they\u0026rsquo;re obviously gonna throw 404.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 $ pytest --tb=line ============================== test session starts ============================== platform linux -- Python 3.8.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 rootdir: /efs/repos/fastauth plugins: anyio-3.4.0 collected 9 items tests/test_main.py . [ 11%] tests/test_users.py ....FFFF [100%] ==================================== FAILURES =================================== /efs/repos/fastauth/tests/test_users.py:65: assert 404 == 405 /efs/repos/fastauth/tests/test_users.py:70: assert 404 == 422 /efs/repos/fastauth/tests/test_users.py:78: assert 404 == 422 /efs/repos/fastauth/tests/test_users.py:85: assert 404 == 201 Our tests are red. We\u0026rsquo;ll have to do something to turn them green.\nWrite code to fix failing test # Add new models for credential data and token # User has to send their username and password to be able to log in to the system and get a token. Here are the changes:\nfastauth/schemas.py # 19 20 21 22 23 24 25 26 27 28 + + +class UserAuthenticate(BaseModel): + username: str + password: str + + +class Token(BaseModel): + access_token: str + token_type: str UserAuthenticate indicates a structure of JSON that we are supposed to receive in a request from a user.\nToken denotes a structure that the server will return.\nDefine /users/auth route # Let\u0026rsquo;s have a route as per our spec to let a user log in. This route, after receiving proper credentials, returns a JWT token\nfastauth/main.py # 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 from fastapi import FastAPI, Depends, HTTPException from sqlalchemy.orm import Session -from fastauth import schemas, crud +from fastauth import schemas, crud, auth from fastauth.database import DBInit +# These constants go in a specific config file +ACCESS_TOKEN_EXPIRE_MINUTES=15 def get_db(): session = None @@ -20,9 +22,26 @@ app = FastAPI() async def ping(): return {\u0026#39;msg\u0026#39;: \u0026#39;pong\u0026#39;} -@app.post(\u0026#34;/users/register\u0026#34;, status_code=201, response_model=schemas.UserInfo) +@app.post(\u0026#34;/users/register\u0026#34;, status_code=201, response_model=schemas.UserInfo, tags=[\u0026#34;users\u0026#34;]) def register_user(user: schemas.UserCreate, db: Session = Depends(get_db)): db_user = crud.get_user_by_username(db, username=user.username) if db_user: raise HTTPException(status_code=409, detail=\u0026#34;Username already registered\u0026#34;) return crud.create_user(db=db, user=user) + + +@app.post(\u0026#34;/users/auth\u0026#34;, response_model=schemas.Token, tags=[\u0026#34;users\u0026#34;]) +def authenticate_user(user: schemas.UserAuthenticate, db: Session = Depends(get_db)): + db_user = crud.get_user_by_username(db, username=user.username) + if db_user is None: + raise HTTPException(status_code=403, detail=\u0026#34;Username or password is incorrect\u0026#34;) + else: + is_password_correct = auth.check_username_password(db, user) + if is_password_correct is False: + raise HTTPException(status_code=403, detail=\u0026#34;Username or password is incorrect\u0026#34;) + else: + from datetime import timedelta + access_token_expires = timedelta(minutes=ACCESS_TOKEN_EXPIRE_MINUTES) + access_token = auth.encode_jwt_token( + data={\u0026#34;sub\u0026#34;: user.username}, expires_delta=access_token_expires) + return {\u0026#34;access_token\u0026#34;: access_token, \u0026#34;token_type\u0026#34;: \u0026#34;Bearer\u0026#34;} Let us start from line 33 where we define the route /users/auth. response_model says that this endpoint in good terms returns a schemas.Token. tags, which is also on 25 is to group endpoint on Swagger UI.\nOn line 34, we see that authenticate_user accepts user which comes from a request, and db which is a DB session.\nOn lines 35-37, we check if the passed user exists or not, if not, HTTP 403 is returned. If that is not the case, the function continues to run.\nIf the user exists, we check if the password matches with the help of a helper function check_username_password. If it does not, we again throw a 403.\nThe only case that remained now is to return a token. On lines 43-47, we create a token with an expiry set to ACCESS_TOKEN_EXPIRE_MINUTES minutes. After this much time, the token expires. We are again using a helper function called encode_jwt_token.\nAuth helper function # Now let us see what helper function we have for aiding in the authentication.\nBefore writing this file, please install PyJWT.\npip install PyJWT fastauth/auth.py # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 import datetime import bcrypt import jwt from sqlalchemy.orm import Session from . import models, schemas, crud SECRET_KEY = \u0026#34;09d25e094faa6ca2556c818166b7a9563b93f7099f6f0f4caa6cf63b88e8d3e7\u0026#34; JWT_ALGORITHM = \u0026#34;HS256\u0026#34; def check_username_password(db: Session, user: schemas.UserAuthenticate): db_user_info: models.UserInfo = crud.get_user_by_username(db, username=user.username) db_pass = db_user_info.password.encode(\u0026#39;utf8\u0026#39;) request_pass = user.password.encode(\u0026#39;utf8\u0026#39;) return bcrypt.checkpw(request_pass, db_pass) def encode_jwt_token(*, data: dict, expires_delta: datetime.timedelta = None): to_encode = data.copy() if expires_delta: expire = datetime.datetime.utcnow() + expires_delta else: expire = datetime.datetime.utcnow() + datetime.timedelta(minutes=15) to_encode.update({\u0026#34;exp\u0026#34;: expire}) encoded_jwt = jwt.encode(to_encode, SECRET_KEY, algorithm=JWT_ALGORITHM) return encoded_jwt In check_username_password, we are checking the received password with the password stored in the database. Of course, the database has the hashed version of the password store. This is one of the best practices we must incorporate. Please don\u0026rsquo;t store passwords in plain text.\nIn encode_jwt_token, we use the PyJWT module to create a JWT token with 15 minutes of expiry. The code is pretty much self-explanatory.\nFixing create_user # I just noticed that I need to decode the output of bcrypt.hashpw before saving it into the database.\n1 2 3 4 5 6 7 def create_user(db: Session, user: schemas.UserCreate): - hashed_password = bcrypt.hashpw(user.password.encode(\u0026#39;utf-8\u0026#39;), bcrypt.gensalt()) + hashed_password = bcrypt.hashpw(user.password.encode(\u0026#39;utf8\u0026#39;), bcrypt.gensalt()) + hashed_password = hashed_password.decode(\u0026#39;utf8\u0026#39;) db_user = models.UserInfo(username=user.username, password=hashed_password, fullname=user.fullname) db.add(db_user) db.commit() Confirm if tests are passing # Automated tests # Let us run our test suite\n1 2 3 4 5 6 7 8 9 10 11 $ pytest ==================== test session starts ===================== platform linux -- Python 3.8.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 rootdir: /efs/repos/fastauth plugins: anyio-3.4.0 collected 9 items tests/test_main.py . [ 11%] tests/test_users.py ........ [100%] ===================== 9 passed in 2.17s ====================== Automated tests are in a happy state.\nTest manually using SwaggerUI # Let\u0026rsquo;s test our API manually also.\nManually test login endpoint With this, let\u0026rsquo;s end this post here. In the next post, we shall implement a protected route and shall see how we can use our so-called JWT token to access those routes. Until then, have a nice day.\nConclusion # In this post, we saw some of the very basic concepts to get you started with a login system. We saw how HTTP and Authorization header is involved in this. We also saw a different types of logins. We talked about Authentication Schemes.\nAt last, we came up with a login endpoint spec and turned it into tests, next we implemented the test. Kudos to you!\nPlease join my subscriber list to get updates on my blog. Please enter your email below.\n","date":"16 April 2022","externalUrl":null,"permalink":"/posts/2022/tdd-approach-to-create-an-authentication-system-with-fastapi-part-4/","section":"Posts","summary":"Introduction # In the last couple of posts in TDD Auth with FastAPI we’ve been sustainably moved towards a web service that can let users register with the service. Now what?\nWe have already done the easy part. The next part is to look at the authorization. This involves letting the user log in. And only give access to what they are authorized for. Let us look at the login part first.\n","title":"TDD Approach to Create an Authentication System With FastAPI Part 4","type":"posts"},{"content":" Introduction # Go community believes that we need no framework to develop web services. And I agree with that. When you work without using any framework, you learn the ins and outs of development. But once you learn how things work, you must reside in a community and don\u0026rsquo;t reinvent the wheel.\nI have created POCs in Go before and I had to deal with HTTP headers, serializing/deserializing, error handling, database connections, and whatnot. But now I\u0026rsquo;ve decided to join the Gin community as it is one of the widely accepted in the software development community.\nAlthough I\u0026rsquo;m writing this post as a standalone article and would keep things simple here. But I want to continue building on these examples to have authentication, authorization, databases (including Postgres, ORM), swagger, and GraphQL covered. Will be interlinking the posts when I create them.\nWhy Gin # There are numerous reasons you may want to use Gin. If you ask me, I\u0026rsquo;m a big fan of Gin\u0026rsquo;s sensible defaults.\nAnother thing I like about Gin is that it\u0026rsquo;s an entire framework. You don\u0026rsquo;t need a separate multiplexer and a separate middleware library and so on. On top of that, there are many common things already available that you don\u0026rsquo;t have to reinvent. It does enhance our productivity. Although I\u0026rsquo;m not using it in production, I already have started to feel it.\nHello World in Gin # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 package main import ( \u0026#34;net/http\u0026#34; \u0026#34;github.com/gin-gonic/gin\u0026#34; ) func main() { router := gin.New() router.GET(\u0026#34;/ping\u0026#34;, func(ctx *gin.Context) { ctx.JSON(http.StatusOK, gin.H{ \u0026#34;message\u0026#34;: \u0026#34;pong\u0026#34;, }) }) router.Run() } Let\u0026rsquo;s get familiar with Gin a little bit.\n1 router := gin.New() This creates an instance of Engine. gin.Engine is the core of gin. It also acts as a router and that is why we have put that Engine instance into a variable called router.\n1 router.GET(\u0026#34;/ping\u0026#34;, func(ctx *gin.Context) { This binds a route /ping to a handler. In the example above, I\u0026rsquo;ve created an anonymous function, but it could be a separate function as well. The thing to note here is the parameter to this function. *gin.Context. Context is yet another important construct besides Engine. Context has almost 100 methods attached to it. A newcomer should be spending most of their time understanding this Context struct and their methods.\nLet\u0026rsquo;s now look at the next few lines:\n1 2 3 ctx.JSON(http.StatusFound, gin.H{ \u0026#34;message\u0026#34;: \u0026#34;pong\u0026#34;, }) One of the *gin.Context method is JSON. This method is used to send a JSON response to the client. Meaning that it automatically sets response\u0026rsquo;s Content-Type to application/json. JSON method takes an HTTP status code and a map of response. gin.H is an alias to map[string]interface{}. So basically we can create an object which can have string key and whatever value we want.\nNext is:\n1 router.Run() Engine.Run simply takes our router along with the route handler and binds it to http.Server. The default port is 8080, but if you want, you can have another address passed here.\nThe Book Store API # I\u0026rsquo;ve already done a POC on bookstore before, at that time, I wanted to prototype a connection between MongoDB and Go. But this time my goal is to have Postgres and GraphQL incorporated.\nSo first of all, I\u0026rsquo;d like you to set up a directory structure like this:\n1 2 3 4 5 6 7 8 9 10 11 $ tree . ├── db │ └── db.go ├── go.mod ├── go.sum ├── handlers │ └── books.go ├── main.go └── models └── book.go And let\u0026rsquo;s start filling up those files.\ndb/db.go # 1 2 3 4 5 6 7 8 9 10 package db import \u0026#34;github.com/santosh/gingo/models\u0026#34; // Books slice to seed book data. var Books = []models.Book{ {ISBN: \u0026#34;9781612680194\u0026#34;, Title: \u0026#34;Rich Dad Poor Dad\u0026#34;, Author: \u0026#34;Robert Kiyosaki\u0026#34;}, {ISBN: \u0026#34;9781781257654\u0026#34;, Title: \u0026#34;The Daily Stotic\u0026#34;, Author: \u0026#34;Ryan Holiday\u0026#34;}, {ISBN: \u0026#34;9780593419052\u0026#34;, Title: \u0026#34;A Mind for Numbers\u0026#34;, Author: \u0026#34;Barbara Oklay\u0026#34;}, } Instead of going into the complexity of setting up a database right now, I\u0026rsquo;ve decided to use an in-memory database for this post. In this file, I\u0026rsquo;ve seeded db.Books slice with some books.\nIf models.Book makes, you curious, the next file is that only.\nmodels/book.go # 1 2 3 4 5 6 7 8 package models // Book represents data about a book. type Book struct { ISBN string `json:\u0026#34;isbn\u0026#34;` Title string `json:\u0026#34;title\u0026#34;` Author string `json:\u0026#34;author\u0026#34;` } Nothing fancy here, we only have 3 fields as of now. All of them are strings and with struct tags.\nLet us see our main.go before we go onto handlers.go.\nmain.go # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 package main import ( \u0026#34;github.com/gin-gonic/gin\u0026#34; \u0026#34;github.com/santosh/gingo/handlers\u0026#34; ) func setupRouter() *gin.Engine { router := gin.Default() router.GET(\u0026#34;/books\u0026#34;, handlers.GetBooks) router.GET(\u0026#34;/books/:isbn\u0026#34;, handlers.GetBookByISBN) // router.DELETE(\u0026#34;/books/:isbn\u0026#34;, handlers.DeleteBookByISBN) // router.PUT(\u0026#34;/books/:isbn\u0026#34;, handlers.UpdateBookByISBN) router.POST(\u0026#34;/books\u0026#34;, handlers.PostBook) return router } func main() { router := setupRouter() router.Run(\u0026#34;:8080\u0026#34;) } Almost similar to the hello world example we saw above. But this time we have gin.Default() instead of gin.New(). The Default comes with defaults which most of us would like to have. Like logging middleware.\nFrankly speaking, I haven\u0026rsquo;t used much of Gin\u0026rsquo;s middleware yet. But it\u0026rsquo;s damn simple to create your middlewares. I\u0026rsquo;ll put some links at the bottom of the post for your exploration. But for now, let\u0026rsquo;s look at our handlers.\nhandlers/books.go # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 package handlers import ( \u0026#34;net/http\u0026#34; \u0026#34;github.com/gin-gonic/gin\u0026#34; \u0026#34;github.com/santosh/gingo/db\u0026#34; \u0026#34;github.com/santosh/gingo/models\u0026#34; ) // GetBooks responds with the list of all books as JSON. func GetBooks(c *gin.Context) { c.JSON(http.StatusOK, db.Books) } // PostBook takes a book JSON and store in DB. func PostBook(c *gin.Context) { var newBook models.Book // Call BindJSON to bind the received JSON to // newBook. if err := c.BindJSON(\u0026amp;newBook); err != nil { return } // Add the new book to the slice. db.Books = append(db.Books, newBook) c.JSON(http.StatusCreated, newBook) } // GetBookByISBN locates the book whose ISBN value matches the isbn func GetBookByISBN(c *gin.Context) { isbn := c.Param(\u0026#34;isbn\u0026#34;) // Loop over the list of books, look for // an book whose ISBN value matches the parameter. for _, a := range db.Books { if a.ISBN == isbn { c.JSON(http.StatusOK, a) return } } c.JSON(http.StatusNotFound, gin.H{\u0026#34;message\u0026#34;: \u0026#34;book not found\u0026#34;}) } // func DeleteBookByISBN(c *gin.Context) {} // func UpdateBookByISBN(c *gin.Context) {} The real juice is in this handlers file. This might need some explanation.\nhandlers.GetBooks, which is bound to GET /books dumps the entire book slice.\nhandlers.GetBookByISBN, which is bound to GET /books/:isbn does the same thing, but it also accepts isbn as a URL parameter. This handler scans the entire slice and returns the matched book. Scanning a large slice would not be the most optimal solution, but don\u0026rsquo;t forget that we\u0026rsquo;ll be implementing a real database while we continue to develop this bookstore.\nThe most interesting one here is handlers.PostBook, which is bound to POST /books. c.BindJSON is the magic method, which takes in the JSON from the request and stores it into previously created newBook struct. Later on\nTests # We need a little change here at the moment. We need to remove these contents from main.go:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 @@ -1,17 +1,9 @@ package main -import ( - \u0026#34;github.com/gin-gonic/gin\u0026#34; - \u0026#34;github.com/santosh/gingo/handlers\u0026#34; -) +import \u0026#34;github.com/santosh/gingo/routes\u0026#34; func main() { - router := gin.Default() - router.GET(\u0026#34;/books\u0026#34;, handlers.GetBooks) - router.GET(\u0026#34;/books/:isbn\u0026#34;, handlers.GetBookByISBN) - // router.DELETE(\u0026#34;/books/:isbn\u0026#34;, handlers.DeleteBookByISBN) - // router.PUT(\u0026#34;/books/:isbn\u0026#34;, handlers.UpdateBookByISBN) - router.POST(\u0026#34;/books\u0026#34;, handlers.PostBook) + router := routes.SetupRouter() router.Run(\u0026#34;:8080\u0026#34;) } And put it into a new file.\nroutes/roures.go # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 package routes import ( \u0026#34;github.com/gin-gonic/gin\u0026#34; \u0026#34;github.com/santosh/gingo/handlers\u0026#34; ) func SetupRouter() *gin.Engine { router := gin.Default() router.GET(\u0026#34;/books\u0026#34;, handlers.GetBooks) router.GET(\u0026#34;/books/:isbn\u0026#34;, handlers.GetBookByISBN) // router.DELETE(\u0026#34;/books/:isbn\u0026#34;, handlers.DeleteBookByISBN) // router.PUT(\u0026#34;/books/:isbn\u0026#34;, handlers.UpdateBookByISBN) router.POST(\u0026#34;/books\u0026#34;, handlers.PostBook) return router } I have changes that make sense to you. We did this because we need to start the server from our tests.\nNext, we create a books_test.go in handlers.\nhandlers/books_test.go # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 package handlers_test import ( \u0026#34;bytes\u0026#34; \u0026#34;encoding/json\u0026#34; \u0026#34;net/http\u0026#34; \u0026#34;net/http/httptest\u0026#34; \u0026#34;testing\u0026#34; \u0026#34;github.com/santosh/gingo/models\u0026#34; \u0026#34;github.com/santosh/gingo/routes\u0026#34; \u0026#34;github.com/stretchr/testify/assert\u0026#34; ) func TestBooksRoute(t *testing.T) { router := routes.SetupRouter() w := httptest.NewRecorder() req, _ := http.NewRequest(\u0026#34;GET\u0026#34;, \u0026#34;/books\u0026#34;, nil) router.ServeHTTP(w, req) assert.Equal(t, 200, w.Code) assert.Contains(t, w.Body.String(), \u0026#34;9781612680194\u0026#34;) assert.Contains(t, w.Body.String(), \u0026#34;9781781257654\u0026#34;) assert.Contains(t, w.Body.String(), \u0026#34;9780593419052\u0026#34;) } func TestBooksbyISBNRoute(t *testing.T) { router := routes.SetupRouter() w := httptest.NewRecorder() req, _ := http.NewRequest(\u0026#34;GET\u0026#34;, \u0026#34;/books/9781612680194\u0026#34;, nil) router.ServeHTTP(w, req) assert.Equal(t, 200, w.Code) assert.Contains(t, w.Body.String(), \u0026#34;Rich Dad Poor Dad\u0026#34;) } func TestPostBookRoute(t *testing.T) { router := routes.SetupRouter() book := models.Book{ ISBN: \u0026#34;1234567891012\u0026#34;, Author: \u0026#34;Santosh Kumar\u0026#34;, Title: \u0026#34;Hello World\u0026#34;, } body, _ := json.Marshal(book) w := httptest.NewRecorder() req, _ := http.NewRequest(\u0026#34;POST\u0026#34;, \u0026#34;/books\u0026#34;, bytes.NewReader(body)) router.ServeHTTP(w, req) assert.Equal(t, 201, w.Code) assert.Contains(t, w.Body.String(), \u0026#34;Hello World\u0026#34;) } Also, again, pretty much self-explanatory. I don\u0026rsquo;t think the above code needs any explanation. We are testing for response codes and response bodies for a specific string.\nLet\u0026rsquo;s also run the tests and check how it goes:\n1 2 3 4 5 6 $ go test ./... -cover ? github.com/santosh/gingo [no test files] ? github.com/santosh/gingo/db [no test files] ok github.com/santosh/gingo/handlers (cached) coverage: 83.3% of statements ? github.com/santosh/gingo/models [no test files] ? github.com/santosh/gingo/routes [no test files] Exercise # Yeah, let\u0026rsquo;s this blog post more interesting by adding some interactivity. I have some tasks for you, which you need to solve on your own. Please have a try on them. They are:\nImplement DeleteBookByISBN and UpdateBookByISBN handlers and enable them. Write tests for handlers mentioned above. Our tests are very basic. So are our handlers. We are not doing any error handling. Add error handling to handlers and write tests to validate them. Conclusion # We have seen how simple is it to create a hello world application in Gin. But this journey does not end here. I\u0026rsquo;ll come back with more tutorials next time. Until then, goodbye.\nRelated Link\nhttps://sosedoff.com/2014/12/21/gin-middleware.html https://santoshk.dev/posts/2020/sending-post-request-in-go-with-a-body/ ","date":"17 March 2022","externalUrl":null,"permalink":"/posts/2022/building-a-bookstore-api-in-golang-with-gin/","section":"Posts","summary":"Introduction # Go community believes that we need no framework to develop web services. And I agree with that. When you work without using any framework, you learn the ins and outs of development. But once you learn how things work, you must reside in a community and don’t reinvent the wheel.\nI have created POCs in Go before and I had to deal with HTTP headers, serializing/deserializing, error handling, database connections, and whatnot. But now I’ve decided to join the Gin community as it is one of the widely accepted in the software development community.\n","title":"Building a Book Store API in Golang With Gin","type":"posts"},{"content":" Introduction # I recently got asked to me what did __init__ dunder method did in Python. And I was like, \u0026ldquo;it is used to initialize variables inside a class\u0026rdquo;. And just after that, the follow-up question was, \u0026ldquo;then what is __new__ used for. And I was completely blank and was not able to answer that.\nI was not able to answer that question because there are not many tutorials out there that talk about __new__. I didn\u0026rsquo;t want to happen this with you. And that is why I came up with this blog post for you.\nSimilarities # Let\u0026rsquo;s start with similarities.\nBoth of them are called/invoked during the creation of the instance. Differences # Let\u0026rsquo;s start with differences.\nnew init 1 Called before init Called after new 2 Accepts a type as the first argument Accepts an instance as the first argument 3 Is supposed to return an instance of the type received Is not supposed to return anything 4 Used to control instance creation Used to initialize instance variables Talking about the first point. __new__ is called when the instance is first created. This happens before the initialization of the class.\nBy the way, did you note that the first argument to __init__ is always self? This self is the instance of the class. self is what __new__ returns.\nComing to the third point, __new__ is supposed to return an instance of the class. Note that if __new__ does not returns anything, __init__ is not called.\nWhich one of them is a constructor? # If you are coming from another language, you might be surprised that there are two similar things doing the same kind of work. Most languages you might have worked with would have something called a constructor.\nIn Python, that concept is broken down into constructor and initializer. And you might have guessed, __new__ is the constructor and __init__ is the initializer.\nPlease note that __new__ is implicit. Meaning that if you don\u0026rsquo;t actually need to modify the creation of an instance of the class, you don\u0026rsquo;t need to have a new method.\nOne more thing I\u0026rsquo;d like to add is\u0026hellip; instance variables are local to an instance. So anything you are doing in init is local to that instance only. But anything you are doing in new will be affecting anything created for that type.\nExecution flow with an example # I am going to add some code to make this more engaging. Consider this example:\n1 2 3 4 5 6 7 8 9 class Demo: def __new__(cls, *args): print(\u0026#34;__new__ called\u0026#34;) return object.__new__(cls) def __init__(self): print(\u0026#34;__init__ called\u0026#34;) d = Demo() This is the simplest example of both __new__ and __init__ in action. If you save the above code in a file and run it, you\u0026rsquo;d see something like this:\n1 2 3 $ python3 in_it.py __new__ called __init__ called As you can see, the new method is called first and then execution is passed to the init method.\nUse Cases # Use case for __new__ # One of the best use cases I can take an example of is when creating a Singleton. As we know, Singleton ensures a class only has one instance and provides a global point of access to it.\nSome of the places I have seen singleton in action are in game programming where there is only one instance of the player. Another place, if you have used frontend libraries like Vuex (or Redux), there is only one global instance of the store. Doesn\u0026rsquo;t matter how many instances you create, you\u0026rsquo;ll end up having only one.\nLet\u0026rsquo;s see how to achieve similar behavior in Python:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 class Singleton: __instance = None def __new__(cls): if cls.__instance is None: print(\u0026#34;creating...\u0026#34;) cls.__instance = object.__new__(cls) return cls.__instance s1 = Singleton() s2 = Singleton() print(s1) print(s2) Output:\n1 2 3 4 $ python3 singleton.py creating... \u0026lt;__main__.Singleton object at 0x7f943301d350\u0026gt; \u0026lt;__main__.Singleton object at 0x7f943301d350\u0026gt; As you can see, creating\u0026hellip; is printed only once. they both point to the same memory location. In my case, it\u0026rsquo;s 0x7f943301d350.\nYou might be guessing that can\u0026rsquo;t we do the same thing with __init__? No! That\u0026rsquo;s because __init__ does not return anything. We\u0026rsquo;ll see in the next section what __init__ is well suited for.\nBut first, I wanted to show you another use case of new.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 class Animal: def __new__(cls, legs): if legs == 2: return Biped() else: return Quadruped() class Biped: def __init__(self): print(\u0026#34;Initializing 2-legged animal\u0026#34;) class Quadruped: def __init__(self): print(\u0026#34;Initializing 4-legged animal\u0026#34;) anim1 = Animal(legs=4) anim1 = Animal(legs=2) Output:\n1 2 3 $ python3 newexample.py Initializing 4-legged animal Initializing 2-legged animal I am by no means a zoologist. But you get my point here. You can use __new__ to conditionally create an instance from a class.\nUse case for __init__ # As we have already seen previously. init is there to initialize an instance variable. These instance variables can later be used in different methods of the instance.\nI have extensively used __init__ when I used to work with Qt framework. Qt is a framework for desktop-based UI development. When initializing UI objects, you can set how wide or long the window could be. You can also read preferences from a file and apply that during the initialization phase of an application. Setting the window title could be another example.\nHere I\u0026rsquo;ll demonstrate one such example:\n1 2 3 4 5 class Window(QWidget): def __init__(self, parent = None): super(Window, self).__init__(parent) self.resize(200, 100) self.setWindowTitle(\u0026#34;My App\u0026#34;) The above example is by no means a complete example, but when set up correctly, it will show a window similar to this.\nSample PySide window Key Takeaways # In most cases, you don\u0026rsquo;t need __new__ is called before __init__. __new__ returns a instance of class. __init__ receives the instances of the class returned by __new__. Use __init__ to initilize value. The same concept can be used to answer the question of abstraction vs encapsulation.\nConclusion # That is all I know about __init__ vs __new__. If you have something in your mind that I missed. Please let me know.\nI\u0026rsquo;ll also list some references I look to write this blog post so that you can really the real source of information.\nhttps://dev.to/pila/constructors-in-python-init-vs-new-2f9j https://www.reddit.com/r/learnpython/comments/2s3pms/what_is_the_difference_between_init_and_new/ https://www.pythonprogramming.in/how-to-use-new-and-init-in-python.html ","date":"6 March 2022","externalUrl":null,"permalink":"/posts/2022/__init__-vs-__new__-and-when-to-use-them/","section":"Posts","summary":"Introduction # I recently got asked to me what did __init__ dunder method did in Python. And I was like, “it is used to initialize variables inside a class”. And just after that, the follow-up question was, “then what is __new__ used for. And I was completely blank and was not able to answer that.\nI was not able to answer that question because there are not many tutorials out there that talk about __new__. I didn’t want to happen this with you. And that is why I came up with this blog post for you.\n","title":"__init__ vs __new__ and When to Use Them","type":"posts"},{"content":"","date":"27 February 2022","externalUrl":null,"permalink":"/categories/","section":"Categories","summary":"","title":"Categories","type":"categories"},{"content":"","date":"27 February 2022","externalUrl":null,"permalink":"/tags/ci/","section":"Tags","summary":"","title":"Ci","type":"tags"},{"content":"","date":"27 February 2022","externalUrl":null,"permalink":"/categories/devops/","section":"Categories","summary":"","title":"Devops","type":"categories"},{"content":" Introduction # We have previously seen how to set up a GitHub to Jenkins pipeline with webhooks, this time we are going to continue on that lesson and learn how to configure Jenkins to build docker images and push Docker images to DockerHub.\nWe\u0026rsquo;ll also see how to create images for both arm64 as well as amd64 architectures machines. Keep on reading.\nWhile you read this post, take a moment to connect with me on LinkedIn.\nIndex # Setup dummy project Build images, test and delete the docker image Setup registry credentials Push images to docker hub Build and push multi-architecture images Install docker on Jenkins hosts # By now, you should have the Jenkins server already running. You should have a repo with a working Jenkinsfile. Your Jenkins server should have the repo linked to each other and each push to GitHub should trigger a build on the Jenkins server. If you are not able to do any of those, please look at How to Setup a GitHub to Jenkins Pipeline with WebHooks.\nComing back to the present, our Jenkins pipeline will be using commands like docker build and docker push. Jenkins does not come with docker capabilities on its own. We\u0026rsquo;ll have to install it on our own on the machine that Jenkins is running on (if your build is running on another machine than the master node of Jenkins, adjust accordingly).\nSomeone already has written about Setting Up Docker on Ubuntu 20.04 (Arm64) so I won\u0026rsquo;t replicate it here. Please refer to that post for installing Docker on Ubuntu 20.04.\nI have also written a docker role for installing Docker Engine. Please note that my Jenkins host runs a Ubuntu 20.04 on an arm64 machine. If you are using other OS or architecture, please let\u0026rsquo;s deal with multi-platform, multi-architecture install of Docker together. It\u0026rsquo;s a good first issue for someone who\u0026rsquo;s learning Ansible and want to do some action.\nAnyway, if you are taking a manual way of installing Docker, please make sure jenkins user is added in docker group. Otherwise, the Jenkins build might fail.\nSetup dummy files to work with # Let\u0026rsquo;s write a tiny hello world python application. This will be a command-line application. When this script is invoked, it will print a \u0026lsquo;hello world\u0026rsquo; to the terminal. Then we\u0026rsquo;ll wrap this script in a docker image.\nHere are the minimal files I came up with.\nDockerfile # 1 2 3 4 5 6 7 8 9 FROM python:3.8-alpine WORKDIR /app RUN pip3 install pytest COPY . . CMD [ \u0026#34;python3\u0026#34;, \u0026#34;cotu.py\u0026#34;] cotu.py # 1 2 3 4 5 6 def hello_world(): print(\u0026#34;hello world\u0026#34;) if __name__ == \u0026#34;__main__\u0026#34;: hello_world() script_test.py # 1 2 3 4 5 6 7 8 import script def test_hello_world(capsys): script.hello_world() out, err = capsys.readouterr() assert out == \u0026#39;hello world\\n\u0026#39; assert err == \u0026#39;\u0026#39; With these 3 files in place (along with our Jenkinsfile), you may want to build an image locally.\nBuild, run, test, and remove locally # The command to build is docker build -t sntshk/cotu:latest .. I hope you already know what they are. In case you don\u0026rsquo;t, build is the docker\u0026rsquo;s subcommand which is used to build/create images out of Dockerfile. This command is going to read the Dockerfile we have written above as a recipe to create images.\nWith -t sntshk/cotu:latest, you tell the build command to tag your image. Here you usually put \u0026lt;your dockerhub user id\u0026gt;/\u0026lt;image name\u0026gt;:\u0026lt;image versione\u0026gt;. And at last, there is a . (dot) which tells the build command that the Dockerfile is in the current directory itself.\n1 2 3 4 5 6 7 8 9 10 11 12 $ docker build -t sntshk/cotu:latest . Sending build context to Docker daemon 169.5kB Step 1/7 : FROM python:3.8-alpine 3.8-alpine: Pulling from library/python [...output trimmed...] ---\u0026gt; Running in 6c59633bf78c Removing intermediate container 6c59633bf78c ---\u0026gt; b3814bc48ae9 Successfully built b3814bc48ae9 Successfully tagged sntshk/cotu:latest You can check if they are created, and then run the container made with this image.\n1 2 3 4 5 6 $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE sntshk/cotu latest b3814bc48ae9 9 minutes ago 58.9MB $ docker run --rm -it sntshk/cotu hello world Don\u0026rsquo;t panic about the image size just yet. It can be reduced. But my focus is not there right now.\nIf you run the test, it will pass as well. You can confirm that it has passed by checking if the last command resulted in 0 exit status.\n1 2 3 4 5 6 7 8 9 10 11 12 $ docker run --rm -it sntshk/cotu pytest ==================================== test session starts ==================================== platform linux -- Python 3.8.12, pytest-7.0.1, pluggy-1.0.0 rootdir: /app collected 1 item test_cotu.py . [100%] ===================================== 1 passed in 0.01s ===================================== $ echo $? 0 Finally, let\u0026rsquo;s remove the images which we have built.\n1 2 3 4 5 6 7 8 9 10 11 12 13 $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE sntshk/cotu latest b3814bc48ae9 9 minutes ago 58.9MB $ docker rmi $(docker images -qa) Untagged: sntshk/cotu:latest [...output trimmed...] Deleted: sha256:8d3ac3489996423f53d6087c81180006263b79f206d3fdec9e66f0e27ceb8759 $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE We have done the major part here. We know how to create images, run them, run test commands, and remove images. We have to do the same thing on the Jenkins server (minus the running part). We\u0026rsquo;ll do that in the next section.\nModify Jenkinsfile to build, test and delete Docker image # Let\u0026rsquo;s look at how we left our Jenkinsfile in the last article.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 pipeline { agent any stages { stage(\u0026#39;Init\u0026#39;) { steps { echo \u0026#39;Initializing..\u0026#39; echo \u0026#34;Running ${env.BUILD_ID} on ${env.JENKINS_URL}\u0026#34; } } stage(\u0026#39;Test\u0026#39;) { steps { echo \u0026#39;Testing..\u0026#39; echo \u0026#39;Running pytest..\u0026#39; } } stage(\u0026#39;Build\u0026#39;) { steps { echo \u0026#39;Building..\u0026#39; echo \u0026#39;Running docker build -t sntshk/cotu .\u0026#39; } } stage(\u0026#39;Publish\u0026#39;) { steps { echo \u0026#39;Publishing..\u0026#39; echo \u0026#39;Running docker push..\u0026#39; } } stage(\u0026#39;Cleanup\u0026#39;) { steps { echo \u0026#39;Cleaning..\u0026#39; echo \u0026#39;Running docker rmi..\u0026#39; } } } } They are merely placeholder stages. We\u0026rsquo;re going to put life into it.\nJust like Jenkinsfile have an echo directive to echo whatever text is in front of them, there is a sh directive to run shell commands.\nBelow is the diff of the Jenkinsfile after modification.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 - stage(\u0026#39;Test\u0026#39;) { + stage(\u0026#39;Build\u0026#39;) { steps { - echo \u0026#39;Testing..\u0026#39; - echo \u0026#39;Running pytest..\u0026#39; + echo \u0026#39;Running docker build -t sntshk/cotu:latest .\u0026#39; + sh \u0026#39;docker build -t sntshk/cotu:latest .\u0026#39; } } - stage(\u0026#39;Build\u0026#39;) { + stage(\u0026#39;Test\u0026#39;) { steps { - echo \u0026#39;Building..\u0026#39; - echo \u0026#39;Running docker build -t sntshk/cotu .\u0026#39; + echo \u0026#39;Testing..\u0026#39; + sh \u0026#39;docker run --rm -e CI=true sntshk/cotu pytest\u0026#39; } } stage(\u0026#39;Cleanup\u0026#39;) { steps { - echo \u0026#39;Cleaning..\u0026#39; - echo \u0026#39;Running docker rmi..\u0026#39; + echo \u0026#39;Removing unused docker images..\u0026#39; + // keep intermediate images as cache, only delete the final image + sh \u0026#39;docker images -q | xargs --no-run-if-empty docker rmi\u0026#39; } } } A few things to notice here:\nI have swapped the position of the Build and Test stages. I have decided to build the docker image first, instead of testing. This seems counter-intuitive, isn\u0026rsquo;t it? You might be thinking that if the test failed, the building was useless. I know it costs the time of building an image if the test is supposed to fail. But it helped me to bear the pain of setting up a test environment.\nSome of the echo statements are now sh commands. This includes all of the build, test, and cleanup stages.\nAt this point when you run your build (by simply pushing this update to GitHub), your build should not fail. Please let me know if it did.\nNext, we\u0026rsquo;ll see what we need to do to push this built image to a registry. See you in the next section.\nSetup Docker Registry (DockerHub) credentials with Jenkins # Hello again, the first thing you must do to push to a docker registry is to log in to it. I\u0026rsquo;m using DockerHub as an example, but it could be any registry. E.g. AWS ECR, Google Cloud Container Registry, Azure Container Registry, or your self-hosted registry.\nTo set up DockerHub as a registry in Jenkins, I need DockerHub\u0026rsquo;s user ID and password. If you don\u0026rsquo;t know how to set up credentials with Jenkins, please follow these steps:\nHead over to \u0026lt;JENKINS_URL\u0026gt;/credentials/. Add Jenkins credential screen On the table on the right, when you hover over the text (global), you\u0026rsquo;ll see a caret will appear, click and that and select Add Credentials.\nOn the page that appears: 1. Select the Kind to be Secret text. 2. Scope to Global. 3. Secret to be your DockerHub password. 4. ID to be whatever you\u0026rsquo;d like to refer to this password as. Finally, click the OK button.\nSteps to setup a cred You\u0026rsquo;ll finally see something like this. Cred listing Similarly, I have also created a secret for DOCKER_ID. These credentials will be available to each build as an environment variable. We\u0026rsquo;ll set it up in upcoming sections.\nLogin to Registry and Publish # With credentials registered with Jenkins, we need to use them in our recipe file to push the built image to the registry. But first, we need to\u0026hellip;\nModify Jenkinsfile to initialize credentials # Setting credentials in itself is not enough. We also have to use it in our pipeline. Some modifications are needed in the Jenkinsfile.\n1 2 3 4 5 6 7 8 9 pipeline { agent any + environment { + DOCKER_ID = credentials(\u0026#39;DOCKER_ID\u0026#39;) + DOCKER_PASSWORD = credentials(\u0026#39;DOCKER_PASSWORD\u0026#39;) + } + stages { On lines 4-7 in the above diff, I\u0026rsquo;m storing the credentials in an environment variable that will be available for the entire pipeline.\nLogin to Docker Registry # 1 2 3 4 5 6 7 8 stage(\u0026#39;Init\u0026#39;) { steps { echo \u0026#39;Initializing..\u0026#39; echo \u0026#34;Running ${env.BUILD_ID} on ${env.JENKINS_URL}\u0026#34; + echo \u0026#34;Current branch: ${env.BRANCH_NAME}\u0026#34; + sh \u0026#39;echo $DOCKER_PASSWORD | docker login -u $DOCKER_ID --password-stdin\u0026#39; } } On line 6, I\u0026rsquo;m using a traditional way to log in to the docker hub in a CI environment.\nPublish to Docker Registry # Finally, our Jenkinsfile needs a final bit of modification to push the built image to DockerHub.\n1 2 3 4 5 6 7 8 stage(\u0026#39;Publish\u0026#39;) { steps { - echo \u0026#39;Publishing..\u0026#39; - echo \u0026#39;Running docker push..\u0026#39; + echo \u0026#39;Publishing image to DockerHub..\u0026#39; + sh \u0026#39;docker push $DOCKER_ID/cotu:latest\u0026#39; } } After this stage, the image will be deleted in the cleanup stage to prevent storage in our Jenkins host.\nFinal Jenkinsfile # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 pipeline { agent any environment { DOCKER_ID = credentials(\u0026#39;DOCKER_ID\u0026#39;) DOCKER_PASSWORD = credentials(\u0026#39;DOCKER_PASSWORD\u0026#39;) } stages { stage(\u0026#39;Init\u0026#39;) { steps { echo \u0026#39;Initializing..\u0026#39; echo \u0026#34;Running ${env.BUILD_ID} on ${env.JENKINS_URL}\u0026#34; echo \u0026#34;Current branch: ${env.BRANCH_NAME}\u0026#34; sh \u0026#39;echo $DOCKER_PASSWORD | docker login -u $DOCKER_ID --password-stdin\u0026#39; } } stage(\u0026#39;Build\u0026#39;) { steps { echo \u0026#39;Building image..\u0026#39; sh \u0026#39;docker build -t $DOCKER_ID/cotu:latest .\u0026#39; } } stage(\u0026#39;Test\u0026#39;) { steps { echo \u0026#39;Testing..\u0026#39; sh \u0026#39;docker run --rm -e CI=true $DOCKER_ID/cotu pytest\u0026#39; } } stage(\u0026#39;Publish\u0026#39;) { steps { echo \u0026#39;Publishing image to DockerHub..\u0026#39; sh \u0026#39;docker push $DOCKER_ID/cotu:latest\u0026#39; } } stage(\u0026#39;Cleanup\u0026#39;) { steps { echo \u0026#39;Removing unused docker containers and images..\u0026#39; sh \u0026#39;docker ps -aq | xargs --no-run-if-empty docker rm\u0026#39; // keep intermediate images as cache, only delete the final image sh \u0026#39;docker images -q | xargs --no-run-if-empty docker rmi\u0026#39; } } } } The pipeline is pushing images.\nPassing builds and publishing of images Pushed image to DockerHub Mission accomplished. If you\u0026rsquo;d like to know about the multi-arch builds, please refer to the bonus section.\nProblems you might run into # Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock This is a typical error that is not related to Jenkins. When you see this error, it can have 2 meanings.\na) jenkins user is not in the docker group.\nb) User is added into the group but hasn\u0026rsquo;t re-logged in.\nPlease see post installation guide for Linux on Docker Docs.\nIn Jenkins\u0026rsquo; context, you might need to restart the Jenkins service.\nBonus: Multi-Arch images with docker manifest # As you can see in the last image, the image which is pushed is for the arm64 machine. You can\u0026rsquo;t run this image on an amd64 machine and expect it to work. This happened because the image is built on an arm64. And by default, when you do docker pull or docker push, docker knows which arch you are on, and pulls/pushes accordingly.\nBut I usually work on an amd64 machine locally. And this motivated me to build multiple architectures of images simultaneously.\nFor this, there is a Docker plugin called Docker Buildx. Luckily, buildx is available on Linux installations by default. The next step would be to create a new builder instance.\nIt is important to run the below command as a Jenkins user in the Jenkins server.\n1 $ docker buildx create --use --name multiarch Read the last line carefully. If you miss this, your builds will fail. I have not been able to incorporate this into my ansible config yet, so this step is manual for now. Let\u0026rsquo;s also inspect this builder instance to see the architectures it supports.\n1 2 3 4 5 6 7 8 9 $ docker buildx inspect --bootstrap Name: multiarch Driver: docker-container Nodes: Name: multiarch0 Endpoint: unix:///var/run/docker.sock Status: running Platforms: linux/arm64, linux/arm/v7, linux/arm/v6 After you have run the above command, and don\u0026rsquo;t see your target architecture in the Platforms section, you need to install appropriate emulators. I was using a Ubuntu machine so I installed qemu-user-static from apt. I restarted my docker service and ran docker buildx inspect --bootstrap again.\n1 2 3 4 5 6 7 8 9 $ docker buildx inspect --bootstrap Name: multiarch Driver: docker-container Nodes: Name: multiarch0 Endpoint: unix:///var/run/docker.sock Status: running Platforms: linux/arm64, linux/amd64, linux/riscv64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6 Now my build machine is capable of building amd64 images.\nIf you still see any problem at this stage, please feel free to ping me on @sntshk. Also, have a read of Building Multi-Architecture Docker Images With Buildx by Artur Klauser.\nModify Dockerfile and Jenkinsfile for multi-arch build # We have added --platform=$TARGETPLATFORM in the FROM directive.\nDockerfile # 1 2 3 4 -FROM python:3.8-alpine +FROM --platform=$TARGETPLATFORM python:3.8-alpine WORKDIR /app This is important as we\u0026rsquo;ll be passing --platform to build command now. And this image will be built whatever number of the platform we pass.\nJenkinsfile # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 @@ -18,7 +18,7 @@ pipeline { stage(\u0026#39;Build\u0026#39;) { steps { echo \u0026#39;Building image..\u0026#39; - sh \u0026#39;docker build -t $DOCKER_ID/cotu:latest .\u0026#39; + sh \u0026#39;docker buildx build -t $DOCKER_ID/cotu:latest .\u0026#39; } } stage(\u0026#39;Test\u0026#39;) { @@ -29,16 +29,15 @@ pipeline { } stage(\u0026#39;Publish\u0026#39;) { steps { - echo \u0026#39;Publishing image to DockerHub..\u0026#39; - sh \u0026#39;docker push $DOCKER_ID/cotu:latest\u0026#39; + echo \u0026#39;Building and publishing multi-arch image to DockerHub..\u0026#39; + sh \u0026#39;docker buildx build --push --platform linux/amd64,linux/arm64 -t $DOCKER_ID/cotu:latest .\u0026#39; } } Some notable changes are:\ndocker build is replaced with docker buildx build. buildx command also has a --push flag, which tells docker to push as soon as the build is succeeded. buildx takes several --platforms. This is a comma-separated value of arches. When the build is successful. You should see something like this on your Docker Hub side.\nDockerHub listing of multi-arch image For people who are reading their article on a Mac. MacOS\u0026rsquo; irony is that, you don\u0026rsquo;t need a darwin/amd64 image to run on Mac, you can use linux/amd64. This is because Docker VM on Mac runs in a Linux VM rather than native on the machine itself.\nEpilogue # That\u0026rsquo;s how we create an image for a CLI application. And that\u0026rsquo;s how we can build it for multiple architectures of the machine. The docker client will request relevant images according to their host architecture.\nWith that said, if you are planning to use methods described in this post in production. I\u0026rsquo;d highly recommend looking more at the security aspect. I didn\u0026rsquo;t cover much of security on the Jenkins side as it was not in the scope of this article. One of the things which can be improved is\u0026hellip; notice that when you go through the console output of the build, you\u0026rsquo;d see something like this:\n1 2 3 WARNING! Your password will be stored unencrypted in /var/lib/jenkins/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store This is one place you can improve at. There is also a Docker Pipeline plugin which can be used to write Jenkins files with their syntax. I haven\u0026rsquo;t used that personally, so can\u0026rsquo;t say much.\nPlease let me know what do you think about this post. Subscribe to the below newsletter.\n","date":"27 February 2022","externalUrl":null,"permalink":"/posts/2022/how-to-setup-a-jenkins-to-dockerhub-pipeline-with-multi-arch-images/","section":"Posts","summary":"Introduction # We have previously seen how to set up a GitHub to Jenkins pipeline with webhooks, this time we are going to continue on that lesson and learn how to configure Jenkins to build docker images and push Docker images to DockerHub.\nWe’ll also see how to create images for both arm64 as well as amd64 architectures machines. Keep on reading.\n","title":"How to Setup a Jenkins to DockerHub Pipeline with Multi-Arch Images","type":"posts"},{"content":"","date":"27 February 2022","externalUrl":null,"permalink":"/tags/jenkins/","section":"Tags","summary":"","title":"Jenkins","type":"tags"},{"content":" Introduction # In a typical CI/CD pipeline, what happens is when you push your code to a code repository (remote) like GitHub/GitLab/BitBucket, an event is triggered by the git host provider. This trigger is known as WebHook. This webhook can be used by some third-party service to provide an array of integrations. Do you want to tweet on every git push? You can do that.\nBut today we are not going to tweet on each push. But we are going to run a build that creates a docker image and pushes it to Docker Hub. From there one can pull the images to further use them for further consumption. You wanna use that in your Kubernetes cluster? Why not!\nIndex # Setup Jenkins server Setup repo with Jenkinsfile Setup Multibranch Jenkins pipeline Setup WebHook on GitHub repo Test Multibranch Jenkins pipeline Publish to DockerHub (in next post) While you read this post, take a moment to connect with me on LinkedIn.\nInstall Jenkins on AWS Cloud # Leasing a machine from AWS will cost you money. But if you are doing this for learning purposes and you pledge to delete the installation after the tutorial has been done then the cost should be very nominal.\nIf you are completely new to AWS, you can leverage AWS Free Tier. Just so that you are not in dark, this tutorial makes use of EC2 and EBS.\nI\u0026rsquo;m not exactly going to show how to setup Jenkins server because I have already covered that in past. Starting with this post would be a good start if you are setting your own Jenkins server.\nI also have some ansible playbooks which can help you provision your Jenkins and Nginx server.\nI hope you have the Jenkins server already running by now. Let it be on your local server or on a cloud. I am following this tutorial on AWS cloud. If you are trying this on your local and you have any issues, please let me know in the comments at the end of the article.\nInstall Multibranch Scan Webhook Trigger # This is one extension that we\u0026rsquo;ll use with our Jenkins installation.\nTo install this extension, head over your Jenkins instance and go to Dashboard \u0026gt; Manage Jenkins \u0026gt; Manage Plugin.\nNow search for Multibranch Scan Webhook Trigger and install it.\nYou might have to restart the Jenkins server to take the effect.\nSetup a repo with mock Jekinsfile # After installing Jenkins and the Multibranch Scan Webhook Trigger plugin, the next step is to prepare our repo which is hosted on GitHub.\nI suppose you have nothing right now.\nTo start with, let\u0026rsquo;s go with this simple pipeline file. This pipeline file is going to print statements to the console on the Jenkins server instead of doing any actual work. But in a typical production environment, they are replaced with real code or command.\nLet\u0026rsquo;s set up a Jenkinsfile:\nJenkinsfile # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 pipeline { agent any stages { stage(\u0026#39;Init\u0026#39;) { steps { echo \u0026#39;Initializing..\u0026#39; echo \u0026#34;Running ${env.BUILD_ID} on ${env.JENKINS_URL}\u0026#34; } } stage(\u0026#39;Test\u0026#39;) { steps { echo \u0026#39;Testing..\u0026#39; echo \u0026#39;Running pytest..\u0026#39; } } stage(\u0026#39;Build\u0026#39;) { steps { echo \u0026#39;Building..\u0026#39; echo \u0026#39;Running docker build -t sntshk/cotu .\u0026#39; } } stage(\u0026#39;Publish\u0026#39;) { steps { echo \u0026#39;Publishing..\u0026#39; echo \u0026#39;Running docker push..\u0026#39; } } stage(\u0026#39;Cleanup\u0026#39;) { steps { echo \u0026#39;Cleaning..\u0026#39; echo \u0026#39;Running docker rmi..\u0026#39; } } } } Every Jenkinsfile must have a pipeline section. This section instructs the Jenkins application to commence actions defined in different stages.\nEach stage must have an agent. If agent in all the stages are the same, it can go one scope above inside the pipeline. agents are basically hosts which can run a specific stage. So in a typical environment, 3 different stages can be running on 3 different compute devices.\nLet\u0026rsquo;s talk a little bit more about the stages. Each stage defined here is a stage/step in CI/CD workflow. In the above pipeline:\nInit: This is the initializing phase. In this phase, I plan to set up the build environment for the build. Things like setting up environment variables etc. Test: The most probable reason anyone would want to establish a CI pipeline is to run unit test. So in this phase, I plan to run the unit test. Build: Build phase is for creating the artifact out of the repository. This could be a docker image or .deb file depending on the what the repository is for. Publish: Publish is for publishing the artifact to some repository. Once again, it could be Docker Hub, PYPI or npm, or anything else. Cleanup: In cleanup phase, I want to clean anything which is will build upon progressively and which can make my Jenkins server out of space. Once again, we are not doing anything real yet. But this is sufficient enough to proceed to the next step. Above is not an exhaustive list of stages, you can have as many stage as you want. They will be executed in chronological order.\nCommit this file to the repo and push it to your GitHub. Once you do that, note the project URL. We\u0026rsquo;ll need that in upcoming sections.\nIt will look like something like this: https://github.com/yourusername/yourrepo\nSetup Multibranch Jenkins Pipeline (and manual build) # After you are running your Jenkins server successfully. You need to create a Pipeline for the project you are setting up CI/CD for. We are talking about the same project for which we set up a GitHub in the last section. Every stage in that pipeline is executed after every push of the commit on GitHub. This execution is triggered by something called a WebHook which we\u0026rsquo;ll study in the next section.\nOn the Dashboard, select New Item. Enter a name for the pipeline and select Multibranch Pipeline. Press OK. Once you\u0026rsquo;ve done the above, now let\u0026rsquo;s proceed with the most important fields only\u0026hellip;\nFill in a Display Name. Under Branch Sources, click Add source and select Git.\n4a. Under Project Repository, input https://github.com/yourusername/yourrepo. Now at this point, if your repo is a private repo, you need to do some additional step here to configure credentials. I am not doing it here as it is out of the scope of this article.\nUnder Build Configuration, select Mode to be by Jenkinsfile.\n5a. Select Script Path to be Jenkinsfile. If Jenkinsfile was in another directory, we would have put the relative from the root of the repository here. Next up.. under Scan Multibranch Pipeline Triggers, check Scan by webhook.\n6a. Under Trigger Token, input a random string. Note: make this token strong enough that it is hard to guess. Now when you go back to your dashboard. You\u0026rsquo;ll see your project listed here. When you click on this project, you\u0026rsquo;ll see several branches here (for now, you\u0026rsquo;d only see main/master).\nAt this point, we have our pipeline setup. At this point, whenever you make an update to the GitHub repository, you can come to the job\u0026rsquo;s page and click on \u0026ldquo;Scan Multibranch Pipeline Now\u0026rdquo;. This would trigger a new build which would run through a defined pipeline.\nIn the next section, we\u0026rsquo;ll see how we can automate this build.\nAutomating the manual build with WebHooks # What is a WebHook? # If you are familiar with RESTful APIs, then webhooks are noshing more than a POST request made by a web service to another service. In our case, the web service which is creating this request is GitHub and the service where this request is made to is our Jenkins server running on EC2.\nFor a more formal definition from GeeksForGeeks:\nWebhooks allow interaction between web-based applications through the use of custom callbacks. The use of webhooks allows web applications to automatically communicate with other web apps. Unlike traditional systems where one system (subject) keeps polling another system (observer) for some data, Webhooks allow the observer to push data into the subject’s system automatically whenever some event occurs.\nThis eliminates the need for constant checking to be done by the subject. Webhooks operate entirely over the internet and hence, all communication between systems must be in the form of HTTP messages.\nBy this definition, I got to understand that instead of our Jenkins polling GitHub for updates at intervals, GitHub notifies Jenkins when a new update comes in. This is more efficent.\nSetup Webhook on GitHub repo # As the title says, we are going to make a connection between Github and Jenkins. Go to your repo\u0026rsquo;s settings and look for Webhooks in the left panel.\nhttps://github.com/\u0026lt;user\u0026gt;/\u0026lt;repo\u0026gt;/settings/hooks. For example, when I go to https://github.com/santosh/cotu/settings/hooks, I see this:\nAdd webhook screen GitHub (click to zoom) Click on \u0026ldquo;Add webhook\u0026rdquo; to add a new hook. And now you\u0026rsquo;ll be presented with a form. Input as directed below:\nPayload URL: The payload URL is divided into 4 parts. Let\u0026rsquo;s take an example of my Jenkins server. [JENKINS_URL][/multibranch-webhook-trigger][/invoke][?token=mysecrettoken]\na. JENKINS_URL: This is the host the Jenkins is installed on. It will be different for you. For me, it\u0026rsquo;s https://ci.santoshk.dev.\nb. /multibranch-webhook-trigger: This comes from Multibranch Scan Webhook Trigger we installed. You can read about this on their page.\nc. /invoke: This is the endpoint GitHub will hit when there are any changes registered on their side.\nd. ?token=mysecrettoken: For sake of security, you can set up a security token. So that only the one who knows the token can trigger a build.\nContent type: Set it to application/json.\nWhich events would you like to trigger this webhook?: Choose Let me select individual events. This will show some more options. Choose Pushes and Pull requests.\nAdd webhook details Keep everything else on the default options and click Add webhook at the bottom.\nTest Multibranch Jenkins pipeline # When you push a new update on the repo now, you\u0026rsquo;ll the builds being taken up.\nBuilds:\nOn this page, you\u0026rsquo;ll see that one build has already been run. This happens are part of the setup. When you click on that branch, you\u0026rsquo;d see something like this:\nJenkins build This page, also shows how much time each stage took.\nLogs:\nIn your lower left that there is a green checkmark and besides that, there is #1. If you click on that, and then select Console Output from the left sidebar. You\u0026rsquo;d see the actual console output generated by the Jenkins when running this build pipeline.\nIt will look something like this.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 Branch indexing \u0026gt; git rev-parse --resolve-git-dir /var/lib/jenkins/caches/git-e5842b23d97cc4456ddd15d1a300b011/.git # timeout=10 Setting origin to https://github.com/santosh/cotu \u0026gt; git config remote.origin.url https://github.com/santosh/cotu # timeout=10 Fetching origin... Fetching upstream changes from origin \u0026gt; git --version # timeout=10 \u0026gt; git --version # \u0026#39;git version 2.25.1\u0026#39; \u0026gt; git config --get remote.origin.url # timeout=10 \u0026gt; git fetch --tags --force --progress -- origin +refs/heads/*:refs/remotes/origin/* # timeout=10 Seen branch in repository origin/master Seen branch in repository origin/setup-cicd-pipeline Seen 2 remote branches Obtained Jenkinsfile from 094a548654f1ee686b034557f2197de945c4ba22 [Pipeline] Start of Pipeline [Pipeline] node Running on Jenkins in /var/lib/jenkins/workspace/example_setup-cicd-pipeline [Pipeline] { [Pipeline] stage [Pipeline] { (Declarative: Checkout SCM) [Pipeline] checkout Selected Git installation does not exist. Using Default The recommended git tool is: NONE No credentials specified Cloning the remote Git repository Cloning with configured refspecs honoured and without tags Cloning repository https://github.com/santosh/cotu \u0026gt; git init /var/lib/jenkins/workspace/example_setup-cicd-pipeline # timeout=10 Fetching upstream changes from https://github.com/santosh/cotu \u0026gt; git --version # timeout=10 \u0026gt; git --version # \u0026#39;git version 2.25.1\u0026#39; \u0026gt; git fetch --no-tags --force --progress -- https://github.com/santosh/cotu +refs/heads/*:refs/remotes/origin/* # timeout=10 \u0026gt; git config remote.origin.url https://github.com/santosh/cotu # timeout=10 \u0026gt; git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10 Avoid second fetch Checking out Revision 094a548654f1ee686b034557f2197de945c4ba22 (setup-cicd-pipeline) \u0026gt; git config core.sparsecheckout # timeout=10 \u0026gt; git checkout -f 094a548654f1ee686b034557f2197de945c4ba22 # timeout=10 Commit message: \u0026#34;Add Cleanup stage in Jenkinsfile\u0026#34; First time build. Skipping changelog. [Pipeline] } [Pipeline] // stage [Pipeline] withEnv [Pipeline] { [Pipeline] stage [Pipeline] { (Init) [Pipeline] echo Initializing.. [Pipeline] echo Running 1 on https://ci.santoshk.dev/ [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Test) [Pipeline] echo Testing.. [Pipeline] echo Running pytest.. [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Build) [Pipeline] echo Building.. [Pipeline] echo Running docker build -t sntshk/cotu . [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Publish) [Pipeline] echo Publishing.. [Pipeline] echo Running docker push.. [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Cleanup) [Pipeline] echo Cleaning.. [Pipeline] echo Running docker rmi.. [Pipeline] } [Pipeline] // stage [Pipeline] } [Pipeline] // withEnv [Pipeline] } [Pipeline] // node [Pipeline] End of Pipeline Finished: SUCCESS To summarize, whenever this pipeline is run, Jenkins clones the repo, then runs the pipeline.\nEpilogue # We have not seen pushing to DockerHub in this post. But that post is already is in my draft. Stay tuned, and share this post with your circle.\n","date":"20 February 2022","externalUrl":null,"permalink":"/posts/2022/how-to-setup-a-github-to-jenkins-pipeline-with-webhook/","section":"Posts","summary":"Introduction # In a typical CI/CD pipeline, what happens is when you push your code to a code repository (remote) like GitHub/GitLab/BitBucket, an event is triggered by the git host provider. This trigger is known as WebHook. This webhook can be used by some third-party service to provide an array of integrations. Do you want to tweet on every git push? You can do that.\n","title":"How to Setup a GitHub to Jenkins Pipeline with WebHooks","type":"posts"},{"content":"","date":"4 February 2022","externalUrl":null,"permalink":"/tags/ansible/","section":"Tags","summary":"","title":"Ansible","type":"tags"},{"content":" Recap # We have already seen how to enable HTTPS on a domain if you have a VPS and you are using Nginx. In one other post we have also talked about getting started with Ansible in which I wrote about basic concepts in Ansible such as inventory, module, playbook. We have also talked about Ansible Roles which are about directory structures.\nWhat we\u0026rsquo;ll cover # In last post we saw how I went through steps of enabling HTTPS on all of the subdomains on a website. I did it manually, not with Ansible. This post is continuation of that where I am going to automate this process.\nIn last post I also refrained to add anything to the public repo as I found it creepy to store my security related data openly on the web. So instead of putting chuck of configuration openly on the tasks or templates, what we can use is something called variables. Good thing about vairables are that we can put it in a separate file and use it from there. And if we keep the variables separately, we can also override variables from command line, and use ansible-vaults to encrypt them. Variables are also used in substituting placeholder text in template files.\nThis is third post in the series of my Ansible learning and in this post I\u0026rsquo;ll talk about:\nHow to make use of Ansible in this automation process of fetching HTTPS certificates. On the Ansible side, I\u0026rsquo;ll cover templating, variables. We\u0026rsquo;ll also see how we can override variables from command line. While you read this post, take a moment to connect with me on LinkedIn.\nPick up where we left off # Just so that everyone is on the same track (assuming you are following this on your own). I have this setup right now.\n1 VPC running Ubuntu 20.04. I have leased from AWS. You can lease from anywhere you want. Make sure you can ssh to it and have sudo access to install stuff. 1 domain. HTTPS certificates are typically issued to domain names, not IP addresses. v0.1.0 of my Ansible configuration. A side note. You can\u0026rsquo;t have Windows as a control node. Although you can manage a Windows machine with a Linux/Mac. This means that if you are on Windows, please use WSL or switch to Linux or Mac, whichever is convenient.\nAnother side note. This time I don\u0026rsquo;t have my santosh.pictures domain which I used in previous posts. Instead, I will use this santoshk.dev domain. A few things about this domain:\nsantoshk.dev is mapped to netlify, which hosts my Hugo JAMStack website. So we only have *.santoshk.dev to tinker with. Unlike some previous posts in this series, this domain is not parked on Route53, but on namecheap.com. You don\u0026rsquo;t practically need to register/transfer your domain on namecheap. I\u0026rsquo;ve tried to make this article registrar agnostic.\nInstall Jenkins and Nginx with our Playbook # If you are following with me, you need to clone my ansible config \u0026gt; checkout to v.0.1.0 \u0026gt; install nginx and jenkins with our playbook.\nThis is the comamnd I used to install Jenkins and Nginx on my EC2 machine:\nansible-playbook -i inventory -u ubuntu playbooks/jenkins.yml ansible-playbook -i inventory -u ubuntu playbooks/nginx.yml If you don\u0026rsquo;t find yourself familiar with above command, you should definately checkout my previous post.\nOnce you run the nginx playbook which we have written by ourself, you should receive an IP address in the console output of the playbook. If you go to that address, you should see the Unlock Jenkins page.\nThe real need here on the Jenkins page is that we have to have secure connection between the browser and the Jenkins server on AWS. For that we need a domain.\nTask 1: Put a DNS entry for subdomain # I will be mapping ci.santoshk.dev to point to the IP address we got after running our nginx playbook. This is a manual process and will depend on where you have your domain parked.\nBasically what I have done is I have created an A Record, with host being the ci and Value being the IP of the nginx host.\nAdding A Record Task 2: Separate nginx config for each subdomain # Look at nginx.conf of v0.1.0 in our role, specifically the server block:\nIf we had planned to host the Jenkins on root domain then this config would work. But now that we are doing it on subdomain basis, we need to extract this to another file.\nTask 2.1: Separate out core nginx and jenkins config # We are going to remove line 43-45 from nginx.conf, and then put it in a separate file called jenkins.conf. Here are the content of this file.\nserver { server_name ci.santoshk.dev; location / { proxy_pass http://localhost:8080/; } } A little bit of explanation now. As you can see, we are overriding server_name directive in this file to tell nginx that following reverse proxy entry is for ci.santoshk.dev subdomain. Every reverse proxy entry will here be mapped to ci.santoshk.dev + path, e.g. ci.santoshk.dev + / = ci.santoshk.dev/. Which would go to localhost:8080.\nI hope I\u0026rsquo;m clear here, if not, please leave a comment and I\u0026rsquo;ll do my best to explain this.\nPlease also note that above nginx.conf is destined to be stored at /etc/nginx/nginx.conf. And the jenkins.conf is destined to /etc/nginx/conf.d/ci.santoshk.dev.conf on the nginx host.\nWith Jenkins and Nginx running. And nginx restarted with above separation, let\u0026rsquo;s check\u0026hellip;\n1 2 3 4 5 6 7 8 9 ubuntu@ip-10-2-1-10:/etc/nginx$ curl http://ci.santoshk.dev \u0026lt;html\u0026gt;\u0026lt;head\u0026gt;\u0026lt;meta http-equiv=\u0026#39;refresh\u0026#39; content=\u0026#39;1;url=/login?from=%2F\u0026#39;/\u0026gt;\u0026lt;script\u0026gt;window.location.replace(\u0026#39;/login?from=%2F\u0026#39;);\u0026lt;/script\u0026gt;\u0026lt;/head\u0026gt;\u0026lt;body style=\u0026#39;background-color:white; color:white;\u0026#39;\u0026gt; Authentication required \u0026lt;!-- --\u0026gt; \u0026lt;/body\u0026gt;\u0026lt;/html\u0026gt; Yeah, looks like we are going somewhere with this. Looks like this page is redirecting us to /login?from=%2F, which is the login page of Jenkins.\nBut same will not be true if you try to access the site http://ci.santoshk.dev on browser.\nConnection Error This happens because your browser is redirecting you from http://ci.santoshk.dev -\u0026gt; https://ci.santoshk.dev (note the s after http). In 2022, this is the default behaviour in all modern browsers.\nWe\u0026rsquo;ll circle back to this problem soon, but let\u0026rsquo;s first do the next thing.\nTask 2.2: Make nginx config part of jenkins role # I haven\u0026rsquo;t actually made any change in my ansible repo. Actually, the file I created above, it should go in jenkins role. For now, we are going to put it in roles/jenkins/files sub-directory.\n1 2 3 4 5 6 7 8 9 10 $ tree roles/jenkins/ roles/jenkins/ ├── files │ └── jenkins.conf └── tasks ├── debian.yml ├── main.yml └── redhat.yml 2 directories, 4 files We are also going to copy this config while jenkins is being install. For that, I\u0026rsquo;ve added a new taskin roles/jenkins/tasks/main.yml:\n1 2 3 4 5 6 7 8 9 10 - include: redhat.yml when: ansible_os_family == \u0026#34;RedHat\u0026#34; +- name: ensure /etc/nginx/conf.d/ directory exists + file: path=/etc/nginx/conf.d state=directory recurse=yes +- name: copy jenkins nginx config + copy: src=jenkins.conf dest=/etc/nginx/conf.d/ci.santoshk.dev.conf - name: run systemctl daemon-reload ansible.builtin.systemd: daemon_reload: yes If you do this, and run both the roles on a fresh instance, you should be able to run above mentioned curl and command and expect same result.\nTake the action: If you are following this tutorial on your own, and really want knowledge to retain, then verify that playbook is working and that you are able to curl from nginx host with same output.\nTask 3: Use variables and templates to refactor existing roles # Even at this stage we already have too many hard-coded keywords that we now need to use yet another concept in Ansible. This will make our work easy by allowing us to use different values for same variables. Yes, you guessed it right. We are going to use variables.\nThe documentation page for Using Variables is the first thing you should be consulting from. If you find it intimidating, I\u0026rsquo;m here to cover your back.\nEven at this point there are multiple places we can imrove at. One big refactor would be to remove the hardcoding of root domain.\nMy actualy intend to make my ansible config public is to make it accessible by the world, and in that case the santoshk.dev does not makes much sense. This value should be dynamic and something which should be passed from the command line while playing the book\nHardcoded root domain That way it will be easier for someone to use my roles. Let\u0026rsquo;s take that initiative:\nReplace all the occurance of santoshk.dev to {{ fqdn }}. This can be simply done by running a project wide Find and Replace. Task 3.1: Convert files to templates # If you replaced all the occurances of domain name with {{ fqdn }}, you\u0026rsquo;d have 2 .conf files, each in nginx and jenkins roles individually. These conf file in files sub-directory of roles are invalid now. This is because files can\u0026rsquo;t have variables. Every file in files subdir are specifically for the purpose of moving from controller to controlled node without any dynamicity.\n1 2 3 4 Changes not staged for commit: modified: roles/jenkins/files/jenkins.conf modified: roles/jenkins/tasks/main.yml modified: roles/nginx/files/nginx.conf We have to move them from files subdir to templates subdir. Also, let them have a .j2 extension which indicates a Jinja2 template file.\n1 2 3 Changes to be committed: renamed: roles/jenkins/files/jenkins.conf -\u0026gt; roles/jenkins/templates/jenkins.conf.j2 renamed: roles/nginx/files/nginx.conf -\u0026gt; roles/nginx/templates/nginx.conf.j2 You\u0026rsquo;d also want to change copy module in both the roles to template module.\n1 2 - copy: src=jenkins.conf dest=/etc/nginx/conf.d/ci.santoshk.dev.conf + template: src=jenkins.conf.j2 dest=/etc/nginx/conf.d/ci.{{ fqdn }}.conf Don\u0026rsquo;t forget the .j2.\n1 2 - copy: src=nginx.conf dest=/etc/nginx/nginx.conf mode=preserve + template: src=nginx.conf.j2 dest=/etc/nginx/nginx.conf mode=preserve Before I commit above changes, I need to store a default value for fqdn somewhere in the file.\nTask 3.2: Set defualt value to variables # Next, you need to define this fqdn variable somewhere in our ansible config repo. Where this variables goes is a good question to ask. The documentation says there could be 16 places where the declaration can go.\nOut of those 16, Let\u0026rsquo;s go through the most common ones to start with.\nInside role\u0026rsquo;s default subdir. This subdir is dedicated to static variables. Meaning that, if you have to construct a URL like https://archive.apache.org/dist/tomcat/tomcat-version/, then https://archive.apache.org/dist/tomcat/tomcat- part would go in default, as they are likely not to change.\nInside role\u0026rsquo;s vars subdir. Continuing with above example, the version part would go in this subdir as they will keep increasing with time.\nInside host_vars. This is for overriding any variable in default or vars which is specific to any machine.\nInside group_vars. This overrides everything listed above. group_vars is by convention used to override a specific group of machines. This grouping varies organisation to organisation.\nThe narrowest reasonable scope we can put this variable is in the jenkins role scope (vars and default). This could have worked, but if we do so, we\u0026rsquo;d have a duplicate. This is because nginx role also have the same domain name listed. And when we write our certbot role, we\u0026rsquo;d have the same duplicate. We need to think of something higher scope.\nFor this time, I\u0026rsquo;m going to go with group_vars approach this time. But later in this post, we\u0026rsquo;ll use both default and vars.\nFor this, let\u0026rsquo;s create a folder group_vars in root dir and have a file called all inside it.\n1 2 $ mkdir group_vars $ touch group_vars/all Open that file and write this:\n1 2 3 --- fqdn: santoshk.dev All of this together, and this works without error.\n1 2 PLAY RECAP ****************************************************************************************** 10.2.1.10 : ok=9 changed=3 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0 I ssh\u0026rsquo;d into that machine and verify that fqdn was replaced with the actual value.\nIf you want to override the value of fqdn from command line, you can pass -e \u0026quot;fqdn=example.com\u0026quot; to ansible or ansible-playbook command. Example below:\n1 ansible-playbook -i inventory -u ubuntu playbooks/nginx.yml -e \u0026#34;fqdn=example.com\u0026#34; Update till here can be found at https://github.com/santosh/ansible/tree/v0.3.0\nTask 4: Point Namecheap nameserver to CloudFlare for certbot # This situation is specific to me, as I\u0026rsquo;m using Namecheap to keep my domain, but I also want to do a DNS based validation to show ownership of the domain to Let\u0026rsquo;s Encrypt. Namecheap\u0026rsquo;s support for Let\u0026rsquo;s Encrypt is controversial. Although it\u0026rsquo;s right that you should be using paid certificates for business entity, but this does not apply for us who are learning to SSL. You might want to read this regarding support of certbot dns authentication on Namecheap.\nYou are good as long as you are using any of these DNS providers. As Namecheap is not one of them, I\u0026rsquo;ve decided to switch to one of those mentioned. If you have your domain not registered with any of those registrar in that list then I recommend you to switch to CloudFlare without spending any money.\nI\u0026rsquo;ve decided to point my domain\u0026rsquo;s nameserver to that of CloudFlare and manage DNS from there. I\u0026rsquo;ll quickly go through the procedure for switching this DNS settings from Namecheap to CloudFlare.\nSign up for and add site to CloudFlare Signing up with CloudFlare is totally free. Once you have signed up, look for a button saying \u0026ldquo;Add Site\u0026rdquo;.\nAdd a new site to CloudFlare Once you do so, choose the Free plan. This plan is enough for the stuff we are going to do.\nChoose the free plan Review DNS records Once you have selected the plan, CloudFlare will scan your existing domain for DNS entries and will offer you to create same entries with them to have the migration process seamless.\nReview DNS records to import Change you nameservers The only step now remaining is the changing the nameserver entry. Look up where you can change your name servers of you domain with your registrar.\nChange nameserver from Namecheap to CloudFlare I have updated by nameservers to point to these:\njohn.ns.cloudflare.com vera.ns.cloudflare.com So right now, I have domain with Namecheap, whose name servers are pointing towards CloudFlare.\nTask 5: Automate cert generation with Ansible # Getting back to Ansible after the subdomain entry already created. We need to use certbot as we did before in our previous post. In that post we used route53 dns plugin. But this time we are going to use cloudflare plugin.\nTask 5.1: Obtain CloudFlare API Token # certbot will have to talk to CloudFlare programatically to verify the ownership of the domain. For that reason, we have to have API Token from them to act on our behalf.\nTo do this\u0026hellip;\nHead over to API Tokens section of you CloudFlare profile and hop on the button saying \u0026ldquo;Create Token\u0026rdquo;. Scroll down and look for an option saying Create Custom Token choose \u0026ldquo;Get started\u0026rdquo;. Create a custom token to edit DNS zone. Make sure to include all zones. API Token to edit DNS Zone You may want replicate settings as indicated in above image.\n4. Continue to View Summary, then Create Token. You\u0026rsquo;ll get something like tqFnsPtJyFAKet0KENpeIpu8lt4j_eu6JlJlYhEM. Keep note of it.\nLook at certbot-dns-cloudflare if you still have any confusion.\nTask 5.2: Create a certbot role to to fetch cert # Now it\u0026rsquo;s time to start working on actual certificate generation. Inspired from Enabling HTTPS on domain(s) manually, I\u0026rsquo;m going to write a equivalent ansible role. We are not including this process in either of existing role beacuse of separation of concern. First of all, we need decoupling from Jenkins or Nginx role, so that we can reuse this role in absense of those role. We can always use all of them in combination.\nThe major difference here from the last post is that we are using CloudFlare this time. The IAM and Route53 is not relevant.\nIt\u0026rsquo;s better to first outline what we want to achive with this role:\nInstall certbot Install certbot cloudflare dns plugin Store dns_cloudflare_api_token in cloudflare.ini and run certbot After this role written, we\u0026rsquo;ll also need some updation in our existing roles.\nSo let\u0026rsquo;s quickly go through the files in certbot role.\nroles/certbot/tasks/main.yml # 1 2 3 4 5 6 7 8 9 - include: debian.yml when: ansible_os_family == \u0026#34;Debian\u0026#34; - name: create /etc/letsencrypt file: path=/etc/letsencrypt state=directory recurse=yes - name: copy cloudflare.ini template: src=cloudflare.ini.j2 dest=/etc/letsencrypt/cloudflare.ini - name: run certbot command: certbot certonly --dns-cloudflare --dns-cloudflare-credentials /etc/letsencrypt/cloudflare.ini --email \u0026#39;{{ letsencrypt_email }}\u0026#39; --domain \u0026#39;*.{{ fqdn }}\u0026#39; --agree-tos --non-interactive roles/certbot/tasks/debian.yml # 1 2 3 4 5 6 - name: update all packages on Debian apt: \u0026#34;upgrade=yes update_cache=yes cache_valid_time=86400\u0026#34; - name: install pip3 apt: name=python3-pip state=present - name: install certbot and certbot-dns-cloudflare command: pip3 install certbot \u0026#39;zope.interface\u0026gt;=5.3.0a1\u0026#39; certbot-dns-cloudflare roles/certbot/vars/main.yml # 1 2 --- letsencrypt_email: you@example.com roles/certbot/templates/cloudflare.ini.j2 # This API token is the same token we obtained previously.\n1 dns_cloudflare_api_token = {{ dns_cloudflare_api_token }} playbooks/nginx.yml # I have decided to run this role before nginx role, so this:\n1 2 3 4 5 hosts: - web roles: + - certbot - nginx With above files in place, I run this command to execute:\n1 ansible-playbook -i inventory -u ubuntu playbooks/nginx.yml -e \u0026#34;letsencrypt_email=\u0026lt;myemail@domain.com\u0026gt;\u0026#34; -e \u0026#34;dns_cloudflare_api_token=LtWvEXAMPLEvs8mZnqMs_syNvHdIMA2w9EcHPEhL\u0026#34; New thing here is the -e syntax. We have already defined dns_cloudflare_api_token in one of our templates above.\n1 2 PLAY RECAP ***************************************************************************************************** 10.2.1.10 : ok=14 changed=2 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0 Our certbot role will generate and keep the certs at /etc/letsencrypt/live/{{ fqdn }}/.\nTake the action: Run only the certbot role by disabling nginx role in the playbooks/nginx.yml. It should run without any error.\nTask 5.3: Configure HTTPS at nginx # Certbot role will generate the certs, but we also need to update nginx and jenkins role to use the certs. There are the changes I came up with:\nroles/nginx/templates/nginx.conf.j2 # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 @@ -17,6 +17,8 @@ http { access_log /var/log/nginx/access.log main; + server_tokens off; + ssl_protocols TLSv1.2 TLSv1.3; sendfile on; tcp_nopush on; tcp_nodelay on; @@ -40,6 +42,17 @@ http { # Load configuration files for the default server block. include /etc/nginx/default.d/*.conf; + listen 443 ssl; + + ssl_certificate /etc/letsencrypt/live/{{ fqdn }}/fullchain.pem; + ssl_certificate_key /etc/letsencrypt/live/{{ fqdn }}/privkey.pem; + + + # redirect non-https traffic to https + if ($scheme != \u0026#34;https\u0026#34;) { + return 301 https://$host$request_uri; + } + error_page 404 /404.html; location = /40x.html { } With server_tokens off, the response header sent by nginx won\u0026rsquo;t include anything which identifies this server as nginx server. Good for security purposes as the attacker won\u0026rsquo;t be able to know which version of nginx is server using and use exploits specific to that version.\nWith ssl_protocols TLSv1.2 TLSv1.3 we tell nginx to use TLSv1.2, and TLSv1.3. There are other older versions which I have omitted. It\u0026rsquo;s good to say with latest.\nAlong with port 80, we are listening on port 443. And for that to work, we need to configure ssl_certificate and ssl_certificate_key. Rest of the config is self-explanatory.\nroles/jenkins/templates/jenkins.conf.j2 # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 @@ -1,5 +1,11 @@ server { server_name ci.{{ fqdn }}; + + listen 443 ssl; + + ssl_certificate /etc/letsencrypt/live/{{ fqdn }}/fullchain.pem; + ssl_certificate_key /etc/letsencrypt/live/{{ fqdn }}/privkey.pem; + location / { proxy_pass http://localhost:8080/; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + } We need same listen and ssl_certificate* directive for jenkins server block.\nI ran the playbook and now ci.santoshk.dev does not refuses my connection.\nCorrect HTTPS setup of subdomain The code till here can be found at https://github.com/santosh/ansible/tree/v0.4.1.\nConclusion # In this post we learned about variables and templates in roles. We also saw what different location we can put our variables in. We also saw how we can inject those variables into templates. And then also learned how to override them from command line. Along with all these, we learned how we can fetch HTTPS certificate from Let\u0026rsquo;s Encrypt.\nUnfortunately this post is getting longer and it would be good to post next part of the series. In next post I\u0026rsquo;ll talk about setup cron for renewal, security hardening, and refactor roles.\nIf this post added value to your life, please consider sharing this with your network. And if you subscribe to my newsletter, you\u0026rsquo;ll get tips \u0026amp; tricks, tutorials \u0026amp; how-tos about software development. Please subscribe below.\n","date":"4 February 2022","externalUrl":null,"permalink":"/posts/2022/automate-https-certificates-with-ansible-roles/","section":"Posts","summary":"Recap # We have already seen how to enable HTTPS on a domain if you have a VPS and you are using Nginx. In one other post we have also talked about getting started with Ansible in which I wrote about basic concepts in Ansible such as inventory, module, playbook. We have also talked about Ansible Roles which are about directory structures.\n","title":"Automate HTTPS Certificates with Ansible Roles ft. Let's Encrypt \u0026 CloudFlare","type":"posts"},{"content":"","date":"4 February 2022","externalUrl":null,"permalink":"/tags/certbot/","section":"Tags","summary":"","title":"Certbot","type":"tags"},{"content":"","date":"4 February 2022","externalUrl":null,"permalink":"/tags/https/","section":"Tags","summary":"","title":"Https","type":"tags"},{"content":"","date":"22 January 2022","externalUrl":null,"permalink":"/tags/edtech/","section":"Tags","summary":"","title":"Edtech","type":"tags"},{"content":" Introduction # Ed-tech industry in India was valued at US$750 million in 2020, and is projected to be $1.04 billion by 2021. (source)\nI am documenting the process of edx installation on local system for evaluating. I\u0026rsquo;m not doing a comparision with any similar product here. Also, I\u0026rsquo;m writing this post from a develper\u0026rsquo;s perspective of course, because I\u0026rsquo;m a software developer. Don\u0026rsquo;t expect any business related expertise from here.\nWith this post, I\u0026rsquo;ll document my own learning. Mostly what technologies this project uses, and how the project is laid out and which tech is used by which component.\nWhat is Open edX? # Open edX is a open source project which powers edx.org and thousands of other online education sites.\nTalking about edx.org, edX is the online learning destination co-founded by Harvard and MIT. The Open edX platform provides the learner-centric, massively scalable learning technology behind it. Originally envisioned for MOOCs, Open edX platform has evolved into one of the leading learning solutions catering to Higher Ed, enterprise, and government organizations alike.\nYou can learn more about Open edX at their official website: https://openedx.org.\nOpen edX landing page Enough of marketing now, now let\u0026rsquo;s get straight back to installation part.\nInstallation # There are two recommended ways to install openedx. Both of them are Docker oriented.\nTutor: A community-supported Docker-based environment suited for both production and development. Devstack: A development environment based on Docker; useful if you want to modify Open edX code locally. As I\u0026rsquo;m just trying out this project. I\u0026rsquo;ll use the Tutor version.\nApart from installation method, you can either choose to install a specific version, or the latest one from GitHub. I\u0026rsquo;ve decided to use Lilac Release, which is one of the recent release. You can find all the releases on Open edX Platform Releases page\nRequirements # As per the official documentation, the system requirements are as follows:\nSupported OS: 64-bit Unix based system. Avoid WSL. Architecture: AMD64, but ARM64 may be supported in future. Required runtime: Docker v18.06+ with Docker Compose v1.22.0 Recommended hardware: 8GB RAM, 4CPU, 25GB disk space Note: I don\u0026rsquo;t know how does above hardware requirement goes with Kubernetes deployment. That will be a good candidate for my new post in this series.\npip installation # There are multiple ways to install Tutor. The first one listed on the documentation page shows the pip installation methed, that is the one we are going to use. Also, just for sake of awareness, I\u0026rsquo;m trying all of these on a Ubuntu 20.04 machine locally.\npip install tutor[full] Note: I am doing it in a virtual environment so my system namespace is not polluted.\nAt this moment you may want to play around with the tutor command.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 $ tutor --help Usage: tutor [OPTIONS] COMMAND [ARGS]... Tutor is the Docker-based Open edX distribution designed for peace of mind. Options: --version Show the version and exit. -r, --root PATH Root project directory (environment variable: TUTOR_ROOT) [default: /home/santosh/.local/share/tutor] -h, --help Show this message and exit. Commands: config Configure Open edX dev Run Open edX locally with development settings help Print this help images Manage docker images k8s Run Open edX on Kubernetes license Manage your Tutor licenses local Run Open edX locally with docker-compose plugins Manage Tutor plugins shell Interactive shell webui Web user interface xqueue Interact with the Xqueue server Configuration # In this section, we\u0026rsquo;ll do basic configuration to run our local instance of Open edX.\nThe next step I\u0026rsquo;m going to follow is:\n1 tutor config save --interactive This will ask some question.\n1 2 3 4 5 6 7 8 9 10 Are you configuring a production platform? Type \u0026#39;n\u0026#39; if you are just testing Tutor on your local computer [Y/n] n As you are not running this platform in production, we automatically set the following configuration values: LMS_HOST = local.overhang.io CMS_HOST = studio.local.overhang.io ENABLE_HTTPS = False Your platform name/title [My Open edX] Your public contact email address [contact@local.overhang.io] The default language code for the platform [en] Configuration saved to /home/santosh/.local/share/tutor/config.yml Environment generated in /home/santosh/.local/share/tutor/env The default location for the configuration is at $HOME/.local/share/tutor. You can also jump there by doing cd \u0026quot;$(tutor config printroot)\u0026quot;.\nconfig.yml is the main file here.\nYou can see there are multiple sub-directories inside env/ directory. These files here are generated from the main config.yml, and as a consequence, every time the values from config.yml are modified, the environment must be regenerated with tutor config save.\nAs we are going to test the local installation, I\u0026rsquo;d highly recommend going through the env/local directory. This directory has the docker-compose.yml files which are used to run the project locally.\nRunning # We are doing doing local install, so I\u0026rsquo;d recommend running tutor local once.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 $ tutor local Usage: tutor local [OPTIONS] COMMAND [ARGS]... Run Open edX locally with docker-compose Options: -h, --help Show this message and exit. Commands: bindmount Copy the contents of a container directory to a... createuser Create an Open edX user and interactively set their... dc Direct interface to docker-compose. exec Run a command in a running container importdemocourse Import the demo course init Initialise all applications logs View output from containers quickstart Configure and run Open edX from scratch reboot Reboot an existing platform restart Restart some components from a running platform. run Run a command in a new container settheme Assign a theme to the LMS and the CMS. start Run all or a selection of services. stop Stop a running platform upgrade Perform release-specific upgrade tasks Likewise, if you want to explore the k8s, or want to do development on edX, use the relevant tutor commands.\nBut for this post, I won\u0026rsquo;t go deep. Let\u0026rsquo;s start a local instance here:\n1 tutor local start The initial start may take some time as there are multiple images to be build. But at the end, you\u0026rsquo;d see something like this in your terminal:\n1 2 3 4 5 6 All services initialised The Open edX platform is now running in detached mode Your Open edX platform is ready and can be accessed at the following urls: http://local.overhang.io http://studio.local.overhang.io The first link is what a new student will see. The second link is something the course creator or site administrator will see.\nOpen edX studio landing page What\u0026rsquo;s next? # You can do multiple things with this installation now. Some of them include:\nLog in as administrator Import demo course Change the look and feel Tweak if you have programming experience Deploy to Kubernetes You can head over to https://docs.tutor.overhang.io/whatnext.html if you are interested in any of those.\nTech stack # Let\u0026rsquo;s talk about the main part here as a developer. The technologies used in this project are as follows:\nPython: There is heavy use of Python. Even tutor is written in Python. Docker: Docker of course is used, and there is also support for Kubernetes. Caddy \u0026amp; Nginx: Both are web servers and reverse proxies. Caddy is mostly used for load balancing and SSL termination. MongoDB \u0026amp; MySQL \u0026amp; Redis \u0026amp; Elasticsearch: Yes, all of them are used. devture/exim-relay: This is for handling mailing services. Open edX is also composed of different frontend which in openedx\u0026rsquo;s language is called a mfe or micro frontends.\nKey Takeaways # This project is good for someone like me who already has some experience in Python, MongoDB, Docker, Kubernetes, and Nginx. This can also be a good opportunity for me to learn more about Redis, Caddy and Elasticsearch.\nThis project is good for frontend developers too, as it uses React in it\u0026rsquo;s micro frontends.\nI intend to experiment with this project to hone my skills. And I might make a follow up post on openedx getting deep into the development and deployment side.\nPS: If you want to clean up the local setup environment, you can run docker rmi $(docker images -q). This will cleanup all the images which were build during the long wait.\n","date":"22 January 2022","externalUrl":null,"permalink":"/posts/2022/i-tried-openedx-and-its-awesome/","section":"Posts","summary":"Introduction # Ed-tech industry in India was valued at US$750 million in 2020, and is projected to be $1.04 billion by 2021. (source)\nI am documenting the process of edx installation on local system for evaluating. I’m not doing a comparision with any similar product here. Also, I’m writing this post from a develper’s perspective of course, because I’m a software developer. Don’t expect any business related expertise from here.\n","title":"I Tried Open edX and it's Awesome","type":"posts"},{"content":"","date":"22 January 2022","externalUrl":null,"permalink":"/tags/k8s/","section":"Tags","summary":"","title":"K8s","type":"tags"},{"content":"","date":"12 January 2022","externalUrl":null,"permalink":"/categories/architecture/","section":"Categories","summary":"","title":"Architecture","type":"categories"},{"content":"","date":"12 January 2022","externalUrl":null,"permalink":"/tags/design-pattern/","section":"Tags","summary":"","title":"Design-Pattern","type":"tags"},{"content":" Introduction # There are a few words, like unit testing, integration testing, mocking, dependency injection, object oriented proramming. These are all friends, and areall related to each other. In this post I\u0026rsquo;m going to talk how, and then it will start to make sense to you.\nThis post is divided into these subsections:\nPreamble The Testing part The OOPs part The dependency injection part A real life problem I\u0026rsquo;ll start with how I started with unit testing.\nWhile you read this post, take a moment to connect with me on LinkedIn.\nPreamble: My journey of testing # In my journey of TDD, at first there was unit test\u0026hellip;. I was not aware of test driven development from start. It took me some time to gain traction on unit testing. And even if I because aware to it, practicing it in Animation and VFX industry was really tough.\nProduction always wants things done as quickly as possible. It was hard to test stuff inside DCC packages like Maya, Nuke or Houdini etc. It is a direct jump to integration test or UI testing. I was not trained in testing like software engineers are. I don\u0026rsquo;t say that studios don\u0026rsquo;t write tested code. The company I currently work in tests. But for whatever business reasons, some development teams don\u0026rsquo;t invest time in testing. And I know, this could be true for any company which is not a tech company.\nWhen I was working with Go, I read a book called Learn Go with Tests. This book encouraged me to embrace Red-Green-Refactor mantra. That is how I came to know about Uncle Bob. Testing is also crucial from the perspective of deployment and CI/CD. The build/deployment should fail if any of the test is failing.\nAfter realizing all these, it was time for me to embrace testing. But I still find some people who say testing does not add any value to their workflow. That is just bullshit.\n\u0026hellip;then there there came databases. Testing an add function is easy. You test it with different values, and even with different type of data types. But have your tried testing an HTTP endpoint which writes row/document to a database? If you have, then you might know that if you test a route, whose handler writes rows to database would fail for second time endpoint is hit. See the link I metioned above to know what I\u0026rsquo;m talking about.\nThere are possibily two things that can be done in this case:\nUse a test database inside current dbms and delete the test table/collection every time test is run. This is more of an integration test. Because we are dealing with real database.\nAlthough integration tests have their own charm in software testing, I don\u0026rsquo;t like this method of testing specifically in this case. Why? Think of running your integration test in CI environment. You need a real database instance there, and I don\u0026rsquo;t want that. Also, a core rule for unit tests is that they should be autonomous.\nSecond option is to mock the call to database and stay in the realm of unit tests. This mocking is achieved by a technique called dependency injection. I have written a whole section dedicated to dependency injection below.\nFor now, let\u0026rsquo;s talk about both integration test and mocking a little bit before we dive deep into DI. Then we\u0026rsquo;ll circle back to this database example.\nThe Testing part: Interation Test and Mocking # In last section we had a situation, we have to test our application, and also do not have to forget that our application depends on a database.\nIntegration test in real life # You can continue unit testing with databases. But we\u0026rsquo;ll call database an external dependency here. And when a test is dependent on external resources, it is an integration test. And one of the core things about unit test is that it has to be self-contained i.e. not depend on external resources.\nWhen you depend on external resources, such as a database, it no longer remains a unit test, it becomes an integration test. Integration tests are different than unit tests. This is one stage above the unit test as seen in below diagram.\n1 2 3 4 5 6 7 /\\ / \\ UI / End-to-End /----\\ / \\ Integration/System /--------\\ / \\ Unit -------------- Example of a laptop manufacturer # Let us understand this scenario with an example of laptop production company. Like any other production company, its very unlikely that a single company will produce the entire laptop. Nope, that\u0026rsquo;s not what usually happens. Each component can come from a different company. The processor can come from company X whereas the hard drive come from company Y.\nWhen company X produces a processor, they will test it before it sells to the laptop manufacturer. We can consider the testing they do as unit test; that is because they test each single unit of processor. Same thing happens when hard drive manufacturer has to sell a unit to laptop manufacturer. They unit test each unit.\nHow how does this relates to our programming scenario? We\u0026rsquo;ll see next\u0026hellip;\nWhen the laptop company receives the parts from company X and Y, the laptop company also runs the same unit test from the specific companies. Thereafter the laptop company plugs the parts into rest of the laptop and runs a so called integration test to check how different units behave in combination to other parts.\nUnit Test vs Integration Test Visualization In a way you can say that integration tests are the big brother to unit tests; and is more like a next step to unit tests. They are used in a combination in software production as well as laptops, cars and whatnot.\nWe\u0026rsquo;ll not be talking about integration test here in this post, because I just wanted you to get acquainted with the word integration testing and don\u0026rsquo;t be confused between unit test and integration test.\nMocking in real life # Mocking is an art of replacing the part of the application you are testing with a dummy version of that part called a mock.\nMock objects are simulated objects that mimic the behavior of real objects in controlled ways, most often as part of a software testing initiative. A programmer typically creates a mock object to test the behavior of some other object, in much the same way that a car designer uses a crash test dummy to simulate the dynamic behavior of a human in vehicle impacts.\nHow do we do this in code? Let\u0026rsquo;s see an example in Python progamming language. Suppose the function below is part of a rather very large application.\nLet\u0026rsquo;s assume the function below is part of a very large application and depends on a third party API.\n1 2 3 4 5 6 def get_repo_by_username(username): \u0026#34;\u0026#34;\u0026#34;Get a repos for a specific username.\u0026#34;\u0026#34;\u0026#34; request_url = \u0026#34;https://api.github.com/users/{user}/repos?per_page=1\u0026amp;page=1\u0026#34;.format(user=username) response = requests.get(request_url) return response.json() In above code, the connection to api.github.com is an external dependency. Of course you can test this code, but you should keep these things in mind.\nYou will need access to api.github.com which might or might not be problem for you or your organisation. If test for this function is running on a CI/CD server like Jenkins, that machine also need to have access to api.github.com. For every test that connect to external server like in above case github.com, each test will have some connection overhead. This time will be added to total time which needs to run the test suite. Basically in Python, you can use @patch decorator from unittest.mock module. Then you can also make use of return_value or side_effect argument with the patched method. I\u0026rsquo;m not going to replicate all of what is already written in Getting Started with Mocking in Python\nAll in all, you can run your unit tests without use of mocking, but it will slow down the test execution time and will be dependent on external resources.\nNote: If you don\u0026rsquo;t mock above function, don\u0026rsquo;t forget that it is by theory not a unit test, but an integration test as we discussed in last section. This integration test tests integration between the application and GitHub.\nThe OOPs part: A principle from the OOPs world # Until now, we know what is integration test, what is unit test, and what is difference between them. We also know how we can use a technique called mocking to mock a call to external dependency.\nHeard of SOLID principles? Yes? You know what D stands for? D stands for Dependency Inversion Principle. If you have not yet heard about SOLID principle, please consider reading this wonderful post by Samuel Oloruntoba. The code example in that post uses Java, but you can easily map it to your favourite language OOPs contruct.\nDependency Inversion Principle states that:\nEntities must depend on abstractions, not on concretions. It states that the high-level module must not depend on the low-level module, but they should depend on abstractions.\nYou know how does it looks like in a picture?\nDependency Inversion Principle in a nutshell What I mean to say by above diagram is, DIP states that our code should not directly depend on the EmailGateway class. Rather it should depend on some other class which is based on abstract class IEmailGateway.\nThis helps us easily replace the EmailGateway code without affect the actual EmailGateway.\nThe dependency injection part: So where does DI fits in? # According to Wikipedia, dependency injection is a technique in which an object receives other objects that it depends on, called dependencies.\nRemember what I said in mocking section? GitHub in that case was a denpendency to our application. Did we do dependency injection when we mocked the call to GitHub? Yes, we did. While that example was not a very great one to explain this concept, I\u0026rsquo;ll go back to the application example which interacted with database.\nBut first, visualize a Composite Pattern in your mind. Suppose your application depends on a database to function, you\u0026rsquo;d not want go ahead and write the database interaction code tightly coupled to the application. Instead the way to do is to write an interface first. This helps us create our components in the application as orthogonal as possible.\nThis interface will have methods like, create record, read record, edit record, delete record. Then you can go ahead and fulfill the contract of the interface.\nThis might seem a long process now, but here are the benifits:\nYou can fulfill the contract for multiple database systems if you change them in future. Suppose you are using Postgres right now. You only need to implement the interfaces for postgres. You write methods to do create, read, edit, delete in postgres style.\nLater on if you decided to use MongoDB, you can re-implement those interfaces separately, without editing the postgres one.\nThis also complies with Open-Close principle of SOLID. Remember that the intent behind dependency injection is to achieve separation of concerns of construction and use of objects. This can increase readability and code reuse.\nhttps://en.wikipedia.org/wiki/Dependency_injection\nA real life problem # This exampleis mostly derived from one of my previous series which is called TDD Auth with FastAPI.\nThe problem # My fictious product has a signup system where user can come up and sign up. And as far as possible, I wanted to test this authentication system with same passion as other part of my systems.\nThe problem is when a test is run, it registers a new user. But when the tests run for second time, it fails. After some investigation, I found out that that entry is not removed when that test is finished.\nI don\u0026rsquo;t want to the test to execute something which writes to production database. I can definately rollback changes done to production database with something called fixtures (or tearDown mechanism), but that\u0026rsquo;s not an option.\nCan\u0026rsquo;t afford to start a database server when testing. I wanted to unit test my API server which connected to mongo instance. I just can\u0026rsquo;t test it because I can\u0026rsquo;t afford to start a separate database server just for sake of testing.\nThe solution # The solution is of course, our Dependency Injection!\nThe only way left to deal with this problem was to use a database like SQLite for testing purpose.\nSeparate database session object for testing and production # The database management system I was using was PostgreSQL. While both Postgres and SQLite are SQL databases, the ORM I was using played an important role to solve this problem as most of the database abstraction by done by SQLAlchemy.\nThe only thing which I had to do was to create a mechanism which returned a connection to SQLite when test was running and connection to Postgres when postgres was running. I created this:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 class BaseDBInit: def __init__(self, db_uri) -\u0026gt; None: self.db_uri = db_uri self.create_engine() models.Base.metadata.create_all(bind=self.engine) @abstractmethod def create_engine(self): pass def get_session(self): session = sessionmaker(autocommit=False, autoflush=False, bind=self.engine) return session class DBInitTest(BaseDBInit): def create_engine(self): self.engine = create_engine( self.db_uri, connect_args={\u0026#34;check_same_thread\u0026#34;: False} ) class DBInit(BaseDBInit): def create_engine(self): self.engine = create_engine(self.db_uri) I know it\u0026rsquo;s mostly inheritence, I had to do this because Python does not have a interface keyword, but you got the gist.\nThe client of these classes can pass in a db_uri and call the method get_session. Here is an example from test code.\n1 2 3 4 5 6 7 8 def get_test_db(): session = None try: session = DBInitTest(\u0026#34;sqlite:///./test.db\u0026#34;).get_session() session = session() yield session finally: session.close() Inject database dependency on runtime # Whenever my test runs, I have used a dependency injection mechanism which comes pre-baked with FastAPI which makes this work really easy. So here is what I had to do in addition to the code I wrote above.\n1 app.dependency_overrides[get_db] = get_test_db What above code does is it overrides get_db. This get_db is actual production version of database connector. This function looks something like this:\n1 2 3 4 5 6 7 8 def get_db(): session = None try: session = DBInit(\u0026#34;postgresql+psycopg2://postgres:postgres@localhost/fastauth\u0026#34;).get_session() session = session() yield session finally: session.close() You see how can same BaseDBInit be used to create connectors for both production and testing? This is the power of object oriented programming. app.dependency_overrides[get_db] = get_test_db is how I inject database dependency into the application.\nThis along side with pytest fixture I can remove SQLite database after every test run. My fixture simply looks like this:\n1 2 3 @pytest.fixture def delete_database(): os.unlink(\u0026#34;./test.db\u0026#34;) Which indeed removes test.db file as we defined in the get_test_db. I call this fixture when I have to run test which writes to database.\nFor an example in Go, I\u0026rsquo;d direct you to https://appliedgo.net/di/.\nAnd this is how we come to end of this post. If you are a first time reader and you liked this post, please consider subscribing to this blog via email on the main page. Also feel free to leave feedback in comments. You can reach me @sntshk. See you later.\nIf you liked this post, please share it with your network. Subscribe to the newsletter below for similar news.\n","date":"12 January 2022","externalUrl":null,"permalink":"/posts/2022/this-is-how-oops-and-mocking-are-related-to-dependency-injection/","section":"Posts","summary":"Introduction # There are a few words, like unit testing, integration testing, mocking, dependency injection, object oriented proramming. These are all friends, and areall related to each other. In this post I’m going to talk how, and then it will start to make sense to you.\nThis post is divided into these subsections:\nPreamble The Testing part The OOPs part The dependency injection part A real life problem I’ll start with how I started with unit testing.\n","title":"This is how OOPs and Mocking are related to Dependency Injection","type":"posts"},{"content":" Introduction # When 2021 started, we were already in the middle of pandemic. Working from home was a new experience. Companies in multimedia industry don\u0026rsquo;t provide work from home to their employees due to their tight compliance. But this pandemic redefined that workflow and gave us this opportunity. That too for all of us.\nThe Retrospective Starfish Model by Visual Thinkery I had mixed experience with work from home. Yes, we don\u0026rsquo;t get to see our collegues face to face. We have to be in front of webcam for that. I really wished virtual reality was so common that companies has made it default for meeting. \u0026#x1f605;\nIn this global lockdown period, I spent time with family after 6 years of a marathon of being out from my home. I am able to see now what life really is. I have started to get answers to all my childhood questions. According to me, childhood is just a demo of life. Everything which is taught is temporary, and we have to re-learn some stuff in order to grow. We can\u0026rsquo;t live with same truth which were taught. They become irrelevant.\nWith that said, let\u0026rsquo;s move to the more technical part of the post.\nPersonal Experiences # AWS EC2 as another PC # In 2021, I bought my first laptop. And it came pre-installed with Windows. And you might already know, I\u0026rsquo;ve been using Linux from about 10 years now. I could have dual-booted the laptop with Linux, but I also didn\u0026rsquo;t want to tamper my laptop for the time span of the warranty at least.\nSo in a nutshell, I\u0026rsquo;m using Amazon Linux 2 on EC2, Windows 10 on my local computer. This setup gives me best of both worlds. Will switch to Windows 11 on my laptop when it becomes more stable.\nUsing VPS as a development environment also keeps me closer to AWS and their eco-systems. It\u0026rsquo;s easy to get my hands dirty with all the buzz of cloud native and serverless going around.\nThis setup works in cases where your system is not very high end, but the task at hand requires some high end system. By high end, I mean superior in terms of CPU, RAM and Network. And as the workstation is in cloud, you can access it from any location from the earth.\nStarted exploring information security # This started when I was compelled to upgrade my website to HTTPS. HTTPS is also a pre-requisites to HTTP/2. Now I know what happens between the browser and the server when you first visit that site.\nOWASP is one of the project I stepped upon in infosec regard. Specially it\u0026rsquo;s Top 10 Web Application Security Risks. You may be already aware of Cross Site Scripting and SQL injection from before. This list compiles similar vulnaribilities which we should avoid while baking our application. Every web developer must read this before they want to create something of large scale.\nI\u0026rsquo;ll be creating more content with respect to information security in this year for sure.\nManaging Infrastructure is DevOps # The word DevOps was not always there when I provisioned my host to install WordPress on it. It has evolved over time. If you ask me where is the line between a actual development and devops arena, I would say development is only till you push your code to source control. Everything after that is DevOps till deployment of the application and monitoring. Yes, auto-scaling is also covered in the devops.\nI can do followig stuff on my own because I didn\u0026rsquo;t know the line between DevOps and a Developer.\nRegistering domains and provisioning servers Enabling HTTPS on a domain Unit test and CI/CD Managing DNS Setting up Jenkins and building on Jenkins (post coming soon) I\u0026rsquo;m thankful that enabling HTTPS on my domain has exposed me to information security.\nAlong with the above mentioned tasks, this year I also stepped into Terraform and Ansbile. And while being closely connected to AWS. I also had some hands on experience with CloudFormation to provision AWS resources. CloudFormation is also included in SAM framework which is a serverless offering from AWS.\nBut for now I am better off with Ansible for bootstrapping servers. And I also don\u0026rsquo;t understand how is Terraform cloud agnostic when we have to write code which is provider dependent. I mean, won\u0026rsquo;t it needed to be re-written when one wants to move to provider A to provider B?\nI first did this, and then later realized that these falls under the umbrella of DevOps.\nPomodoro does not work for me # Talking about time management, I have tried Pomodoro technique before to stay focused on the given task. Although it\u0026rsquo;s a good start if you are looking for something in this range. We even have diffenrent gadgets and online application which follows this techniques.\nBut with my personal experience, I think it\u0026rsquo;s too tight and does not gives much flexibility. One might want to break the session in middle due to this inflexibility. I\u0026rsquo;m saying this because when I keep working for long.. I feel like taking a break in middle of a pomo session. But I also feel like I taking a break disturbs the flow. In lockdown period, when almost everybody is at their home, there is more distraction because of the homely environment. And this worsen the situation.\nPomodoro was started by a student. And still I guess it works best for students only, not for doing some day job.\nFlowtime works better in my case. Unlike Pomodoro, Flowtime affects your psycology. You are not forced to work in a bound time. You can take as many breaks you want. And of whatever length you want. But before taking a break, you have to specify how longer your break will be. If you don\u0026rsquo;t come back to resume your task, you\u0026rsquo;ll feel guilty. That\u0026rsquo;s how it works. It is also each with Flowtime to generate a report of how long you worked, how much time in total it took to complete the task.\nAlong with Flowtime, there are also some general lifestyle tip and tricks I follow to keep myself productive.\nApprox 10 years of GitHubbing \u0026amp; StackOverflowing # 10 years of GitHubbing I have almost been on GitHub for 10 years. It was 2018 when I got my first full-time job.\nDuring these years on GitHub, I was mostly active on Python and general web development related technologies.\nThere has been similar story for my StackOverflow.\n10 years of StackOverflowing In this 10 years, I\u0026rsquo;ve mostly asked questions than answering one. Once again, I was most active in Python tags spreading over all years.\nThe Year 2021 # Now as I do each year, this section and the next one I\u0026rsquo;ll talk about what I had planned for\nCourses # Software Development Processes and Methodologies This was perhaps the first course I did after getting my role as a Full Stack Developer. And as a someone who had not read about SDLC in college, content in this course helped me to understand what\u0026rsquo;s happening around me. I got to know what stakeholders are (although I knew who they are before, but I didn\u0026rsquo;t what there is a word for those people).\nI got to know about requirement vs specification, functional vs non-functional requirements. I got to know about different software development models, such as waterfall, spiral, agile et cetra.\nUltimate AWS Certified Developer Associate 2021 I have completed the Udemy course I was having from last year and now I\u0026rsquo;m reading whitepapers and using their services/SDKs.\nI have made a couple posts, which covers S3 events to invoke Lambda.\nOne new update regarding AWS certification is that my current employer funded for Developing on AWS classroom training for which I\u0026rsquo;m thankful for.\nDocker and Kubernetes: The Complete Guide I have gone through the Docker part already from this course. And in the last year I also worked on creating a continuous deployment pipeline.\nI can now successfully take a non-docerized application and dockerize it. I can also configure Jenkins to build and push images to Docker Registry.\nBooks # The Pragmatic Programmer I have finally finished this book by Andrew Hunt. This book is aimed at people who want to become moreeffective and more productive programmers. Perhaps you feelfrustrated that you don’t seem to be achieving your potential.Perhaps you look at colleagues who seem to be using tools tomake themselves more productive than you. Maybe yourcurrent job uses older technologies, and you want to knowhow newer ideas can be applied to what you do.\nI\u0026rsquo;m not in a mood to write a full review of the book here, but . And this is one book which I recommend to someone new in software development alongside the book Soft Skills.\nThe Year 2022 # Courses # Docker and Kubernetes: The Complete Guide I haven\u0026rsquo;t explored the Kubernetes last year, but this year I plan to explore it in the first quarter.\nData Structures in Real Life Projects This is my one step forward towards making data-structure less alien in life so that I can relate to it and think natively in data structure.\nBooks # Soft Skills: The Software Developer\u0026rsquo;s Life Manual ETA to finish this book was in 2021 itself, but there are some points discussed in this book which I want to implement in my real life. I think I should just complete that book in first pass and then come back to the chapters which needs focus. And this is what I\u0026rsquo;m going to do in this year.\nPython Testing with pytest I have been talking a lot about unit-testing and Test-Driven Development lately.\nI gained traction with TDD when I read Learn Go with Tests. It was a totally new experience for me to learn a new programming language. I was not only learning Go, but also TDD.\nWith this book, I want to be strong in testing in the strongest language I possess.\nMisc # Delve more into Data Structure and Algorithm I have already mentioned Data Structures in Real Life Projects above. Alongside that, I\u0026rsquo;m actively following Study Plans from Leetcode.\nLearn more about information security There are lot more blog posts to come year which will be related to information security as I explore more into this field. So brace yourself.\nFocus on System Design; HLD and LLD I have been getting calls from product based companies in past year, but this is one of the topics I\u0026rsquo;m not confident with. Another one being data structure and algorithms.\nFor lower level design, I\u0026rsquo;ve tried to read, interpret and write my own class diagrams, component diagram.\nteachyourselfcs.com There we some books I was reading on last year. I\u0026rsquo;ll keep following them at mild pace as I proceed with my day job. I focused more on Computer Networking part. I am following the book called Computer Networking: A Top-Down Approach.\nDo more Cloud Native development Kubernetes is my first step towards cloud native software development. Serverless is the next generation development paradigm. But according to me I think it does not play well with the traditional application servers like Flask or FastAPI. The entire Flask/FastAPI thing needs to be re-architectured into serverless paradigm.\nRelated Readings # https://santoshk.dev https://roadmap.sh/backend https://diyminddesign.com/everything-you-need-know-about-flowtime-technique-quick-guide/ ","date":"4 January 2022","externalUrl":null,"permalink":"/posts/2022/2021-in-review/","section":"Posts","summary":"Introduction # When 2021 started, we were already in the middle of pandemic. Working from home was a new experience. Companies in multimedia industry don’t provide work from home to their employees due to their tight compliance. But this pandemic redefined that workflow and gave us this opportunity. That too for all of us.\nThe Retrospective Starfish Model by Visual Thinkery I had mixed experience with work from home. Yes, we don’t get to see our collegues face to face. We have to be in front of webcam for that. I really wished virtual reality was so common that companies has made it default for meeting. 😅\n","title":"2021 in Review: Flowtime, Cloud Native, InfoSec, DevOps, Jenkins and more","type":"posts"},{"content":"","date":"4 January 2022","externalUrl":null,"permalink":"/tags/retrospect/","section":"Tags","summary":"","title":"Retrospect","type":"tags"},{"content":"","date":"4 January 2022","externalUrl":null,"permalink":"/series/year-in-review/","section":"Series","summary":"","title":"Year in Review","type":"series"},{"content":" Introduction # If you are following this series from start, you know that we can already create user from our API. But we have our tests failing right now, which is kind of smelly. In this post, we takle with the problem in our hand and discuss some methods we can do it with. One of them is dependency injection.\nDon\u0026rsquo;t panic, dependency injection is just a fancy word for something very common. Given that you already are familiar with Object Oriented Programming, you\u0026rsquo;d catch up in no time.\nSeries Index # We have already seen:\nProject Setup and FastAPI introduction Database Setup and User Registration Fix the failing test with mocking and dependency injection (you are here) Authentication Premier and Login Endpoint The solve the problem at hand, first we get to understand what the problem is\u0026hellip;\nWhile you read this post, take a moment to connect with me on LinkedIn.\nWhy is our unit test failing? # First of all, our test depends on a running database. Thus the test we are having right now is less of a unit test and more of an integration test.\nLet\u0026rsquo;s see at the output of our last test:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 $ pytest ================================================== test session starts ================================================== platform linux -- Python 3.7.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 rootdir: /efs/repos/fastauth plugins: anyio-3.3.4 collected 5 items tests/test_main.py . [ 20%] tests/test_users.py ...F [100%] ======================================================= FAILURES ======================================================== __________________________ TestUserRegistration.test_post_request_with_proper_body_returns_201 __________________________ self = \u0026lt;tests.test_users.TestUserRegistration object at 0x7fa6570ef8d0\u0026gt; def test_post_request_with_proper_body_returns_201(self): response = client.post( \u0026#34;/users/register\u0026#34;, json={\u0026#34;username\u0026#34;: \u0026#34;santosh\u0026#34;, \u0026#34;password\u0026#34;: \u0026#34;sntsh\u0026#34;, \u0026#34;fullname\u0026#34;: \u0026#34;Santosh Kumar\u0026#34;} ) \u0026gt; assert response.status_code == 201 E assert 409 == 201 E + where 409 = \u0026lt;Response [409]\u0026gt;.status_code tests/test_users.py:39: AssertionError ================================================ short test summary info ================================================ FAILED tests/test_users.py::TestUserRegistration::test_post_request_with_proper_body_returns_201 - assert 409 == 201 ============================================== 1 failed, 4 passed in 1.19s ============================================== Wooo\u0026hellip; looks like we are getting a 409 response instead of 201. We wrote that code with our hand ourselves. Remember this line?\n1 2 if db_user: raise HTTPException(status_code=409, detail=\u0026#34;Username already registered\u0026#34;) This is our culprit. And why we are getting this response? Yeah, because Username already registered.\nThis is happening because on the production database, the user already exists. Remember the word production I used here? If this application was public facing, we must not be testing this application against this one. Even if are doing integration testing, there should be a staging server we should be testing with.\nThere are quite a few ways we can test our user API code here. We\u0026rsquo;ll see both of them in brief. Here are both:\nMock the API calls Inject dependency on the go while execution Both have their own pros and cons and use cases.\nWhat we want from our tests? # We can handle the situation in our hands with different methods. But what we choose depends on what we want. At the current state of the application we are developing, we can create a temporary database in the postgres server for test and then delete it later on. That is totally achievable using pytest. But that is not something that I\u0026rsquo;m looking for.\nWhat I\u0026rsquo;m looking for is\u0026hellip;\nTo be able to run my test on CI/CD server. Which method we choose depends on what satisfies above goal. Both of the method I\u0026rsquo;ve described can be used to do so. Let\u0026rsquo;s see both of them, and then later will choose which one to go with.\nMock the API calls # This is one of the sneakiest way we can test our code in hand here. Let me go ahead and explain you what\u0026rsquo;s going on here.\nI\u0026rsquo;ll take an example of Calculator class for basic understanding. This concept will be easy to explain that way. Later on, we will adapt the same code.\ncalculator.py # 1 2 3 4 5 6 import time class Calculator: def add(self, x, y): time.sleep(5) return x + y Our Calculator.add simply returns the sum of x and y. But it does it after waiting for 10 seconds.\nNext is our test_calculator.py, which bypasses the above function by calling a mock instead of calling it directly.\ntest_calculator.py # 1 2 3 4 5 6 7 8 9 from unittest import TestCase from unittest.mock import patch class TestCalculator(TestCase): @patch(\u0026#39;calculator.Calculator.add\u0026#39;, return_value=9) def test_add(self, add): expected = add(2, 3) self.assertEqual(expected, 9) You don\u0026rsquo;t need to go too far to experiment with mocking and stuff. In above test file, we are adding 2 and 3, and we are expecting to return 9. Isn\u0026rsquo;t that what you were wondering about?\nBut this test will pass.\n1 2 3 4 5 6 $ python -m unittest . ---------------------------------------------------------------------- Ran 1 tests in 0.002s OK To explain the way part, I\u0026rsquo;ll have to go to ahead and explan the API of unitest.mock.patch decorator. Believe it or not, it has been there in Python 3 since 2012. And when I went to see the proposal of how it was added, I found out that mock library has been there even before it was added to standard library. It started in 2007. I was in class 6th back then.\nOkay, so I get a little bit off track there, but here it is..\nThe first argument to unitest.mock.patch is the target, meaning that the class or function for which we are creating stub for. Please note that whatever your input to this argument is should be importable otherwise in the current module. The second argument we have passed it the return_value, this tells the patch function to return a value of 9 whenever we call calculator.Calculator.add. Under the hood, @patch decorator uses mock.Mock class.\nUsing side_effect in @patch # We have seen how we can tell the patch function to return a hardcoded value when we ask it to run a function/class. But this is mostly boring. We want more dynamic behavior. What if we had another function which replicated our add function?\nThat\u0026rsquo;s what we\u0026rsquo;ll do in our next step. Let\u0026rsquo;s create a function without time.sleep and\nHere is a modified version of the test:\n1 2 3 4 5 6 7 8 9 10 11 12 13 from unittest import TestCase from unittest.mock import patch def mock_add(x, y): return x + y class TestCalculator(TestCase): @patch(\u0026#39;calculator.Calculator.add\u0026#39;, side_effect=mock_add) def test_add(self, add): expected = add(2, 3) self.assertEqual(expected, 5) This is more or less the same code, but we have a dynamic behavior now. We can pass our own values to the add function and mock_add function will be executed every time.\nMocking our TestClient.post # Let\u0026rsquo;s put the knowledge we have learned to use. We have our tests, and one of them are failing.\n1 2 3 4 5 6 def test_post_request_with_proper_body_returns_201(self): response = client.post( \u0026#34;/users/register\u0026#34;, json={\u0026#34;username\u0026#34;: \u0026#34;santosh\u0026#34;, \u0026#34;password\u0026#34;: \u0026#34;sntsh\u0026#34;, \u0026#34;fullname\u0026#34;: \u0026#34;Santosh Kumar\u0026#34;} ) assert response.status_code == 201 Here I have modified the test to call our mock version of TestClient.post. Here\u0026rsquo;s the diff:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 +from unittest.mock import patch + import pytest from fastapi.testclient import TestClient @@ -5,6 +7,17 @@ from main import app client = TestClient(app) +def mock_post(endpoint, json): + field_names = json.keys() + + good_mock_response = {\u0026#39;status_code\u0026#39;: 201} + bad_mock_response = {\u0026#39;status_code\u0026#39;: 500} + + if endpoint == \u0026#34;/users/register\u0026#34;: + if all(field in field_names for field in [\u0026#39;username\u0026#39;, \u0026#39;password\u0026#39;, \u0026#39;fullname\u0026#39;]): + return good_mock_response + + return bad_mock_response class TestUserRegistration: \u0026#34;\u0026#34;\u0026#34;TestUserRegistration tests /users/register\u0026#34;\u0026#34;\u0026#34; @@ -27,9 +40,12 @@ class TestUserRegistration: ) assert response.status_code == 422 - def test_post_request_with_proper_body_returns_201(self): - response = client.post( + @patch(\u0026#39;fastapi.testclient.TestClient.post\u0026#39;, side_effect=mock_post) + def test_post_request_with_proper_body_returns_201(self, post): + response = post( \u0026#34;/users/register\u0026#34;, json={\u0026#34;username\u0026#34;: \u0026#34;santosh\u0026#34;, \u0026#34;password\u0026#34;: \u0026#34;sntsh\u0026#34;, \u0026#34;fullname\u0026#34;: \u0026#34;Santosh Kumar\u0026#34;} ) - assert response.status_code == 201 + assert response[\u0026#39;status_code\u0026#39;] == 201 + I have written a similar logic to the original one in mock_post. I am testing for username, password and fullname in the json. If it is present, then returning status code 201. Then I\u0026rsquo;m looking for this status code in the test.\nNow the test passes no matter how many times I run it.\n1 2 3 4 5 6 7 8 $ pytest --no-header --no-summary ==================== test session starts ==================== collected 5 items tests/test_main.py . [ 20%] tests/test_users.py .... [100%] ===================== 5 passed in 1.47s ===================== Final words: My final words about mocking in our case would be that it is not quite elegant. Because at the end of the day we are test our APIs, and we can\u0026rsquo;t just mock calls to API. If we do so, what is the work for testing here? I just demonstrated this method for sake of knowledge. Please go ahead and revert all the changes to test_users.py till now.\nIf you are more interested in mocking, have a look at this post: https://semaphoreci.com/community/tutorials/mocks-and-monkeypatching-in-python\nBut we\u0026rsquo;ll next see more professional way to test our code here.\nOverriding dependencies aka Dependency Injection # I have written an entire post about dependency injection which is yet to be posted. In that post I talk how Object Oriented Programming and Dependency Injection are in the same boat.\nTo give you a visual cue of what dependency injection looks like, here I present you my hand crafted diagram.\nVisual cue of Dependency Injection In above diagram, FakeEmailGateway can be used to do testing related invocation without actually sending real emails.\nIf you a little bit of object oriented programming, you might know how it works. But in this post, I\u0026rsquo;ll not go deep into object oriented programming here. Please check out my other post titled OOPs and Mocking Meet at Dependency Injection.\nLook closely at our database dependency # Do you remember the signature of our register_user? If not, it looks something like this:\n1 def register_user(user: schemas.UserCreate, db: Session = Depends(get_db)): What\u0026rsquo;s so unique in this? Hint: Flask does not comes with this.\nYeah, you saw it correct. Second parameter to our route handler is db. And instead of directly calling get_db which talks to the production database, we have wrapped it with something called Depends. That\u0026rsquo;s an elegant way of handling our dependency. That\u0026rsquo;s the beauty of FastAPI.\nLet\u0026rsquo;s closely see what purpose this Depends serves in FastAPI.\nLooking at the definition of get_db, we can see that what this function provides it a db session.\n1 2 3 4 5 6 7 8 9 from fastauth.database import SessionLocal def get_db(): db = None try: db = SessionLocal() yield db finally: db.close() What if we can somehow modify this function to use a temporary SQLite database for testing? With SQLite database, we won\u0026rsquo;t need a dedicated database running. And we can easily create and delete database files as we need. And thinking about the CI/CD server where this test will be run. We can include a stage to cleanup dangling database files if created.\nOverriding our database dependency # Say hello to dependency_overrides. This is a dictionary whose key is a dependency e.g. the one we used above, and then the key is what we want to override this dependency with. So if your FastAPI instance is named app, the whole thing would look like this:\n1 app.dependency_overrides[get_db] = get_test_db Here the get_test_db is supposed to return a database session which uses a SQLite bind.\nSince we have two get_db function, and both having overlapping functionalities. I have went ahead and refactored the code a bit. My database.py looks like this right now.\nfastauth/database.py # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 +from abc import abstractmethod + from sqlalchemy import create_engine from sqlalchemy.ext.declarative import declarative_base from sqlalchemy.orm import sessionmaker -SQLALCHEMY_DATABASE_URL = \u0026#34;postgresql+psycopg2://postgres:postgres@localhost/fastauth\u0026#34; +from fastauth import models -engine = create_engine(SQLALCHEMY_DATABASE_URL) +Base = declarative_base() -SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine) -Base = declarative_base() +class BaseDBInit: + def __init__(self, db_uri) -\u0026gt; None: + self.db_uri = db_uri + self.engine = None + self.create_engine() + models.Base.metadata.create_all(bind=self.engine) + + @abstractmethod + def create_engine(self): + pass + + def get_session(self): + session = sessionmaker(autocommit=False, autoflush=False, bind=self.engine) + return session + + +class DBInitTest(BaseDBInit): + def create_engine(self): + self.engine = create_engine( + self.db_uri, connect_args={\u0026#34;check_same_thread\u0026#34;: False} + ) + + +class DBInit(BaseDBInit): + def create_engine(self): + self.engine = create_engine(self.db_uri) Instead of creating engine and session in a static manner, we have added a dynamic behavior to them. We have gone object oriented here and leveraging inheriting power of Python. Explanation below:\nGiven that there are two consumer of our session object, we have created a base class BaseDBInit to hold the code that is common to them. What is common to them? Let\u0026rsquo;s see __init__ dunder method for the answer. We are sticking db_uri to the instance. We are calling create_engine. And we are creating all the models in the database. Not matter what, this chunk of code will be run by both of the sub classes. We just said in above point that all the deriving classes will call create_engine method. But in BaseDBInit.create_engine there is no definition? Why? Because with abstractmethod, we are indicating that this function needs to be implemented in the deriving classes. The method with sessionmaker is pretty straighforward. We are doing it the old way, but we are doing it inside the method of BaseDBInit. We are using the engine which is attached to instance (self.engine). Next we are defining DBInitTest and DBInit which are derived from our BaseDBInit. In this class, we are defining create_engine which is specific to for testing (SQLite) and production (Postgres). Next, let\u0026rsquo;s now see how both DBInitTest and DBInit are being used in tests and production.\nmain.py # We have modified how we connect to database in the main.py file. Let\u0026rsquo;s see what:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 from fastapi import FastAPI, Depends, HTTPException from sqlalchemy.orm import Session -from fastauth import models, schemas, crud -from fastauth.database import engine, SessionLocal +from fastauth import schemas, crud +from fastauth.database import DBInit -models.Base.metadata.create_all(bind=engine) def get_db(): - db = None + session = None try: - db = SessionLocal() - yield db + session = DBInit(\u0026#34;postgresql+psycopg2://postgres:postgres@localhost/fastauth\u0026#34;).get_session() + session = session() + yield session finally: - db.close() + session.close() app = FastAPI() And let\u0026rsquo;s see the user test now.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 -from main import app +from main import app, get_db +from fastauth.database import DBInitTest + + +def get_test_db(): + session = None + try: + session = DBInitTest(\u0026#34;sqlite:///./test.db\u0026#34;).get_session() + session = session() + yield session + finally: + session.close() + + +app.dependency_overrides[get_db] = get_test_db + Other than get_test_db which we are using to get session from a SQLite database, we also need get_db function, to tell our FastAPI instance that \u0026lsquo;hey! we are overriding this particular function\u0026rsquo;. This thing is done on the last line above.\nWe also might need to refactor our testing code more as we add more and more test.\nIf you want to know more about testing with dependencies in FastAP, please have a look at these resources:\nhttps://fastapi.tiangolo.com/advanced/testing-dependencies/ https://fastapi.tiangolo.com/advanced/testing-database/ Why is test still red after injecting dependency? # Let\u0026rsquo;s look at how our tests do.\n1 2 3 4 5 6 7 8 $ pytest -q ..... [100%] 5 passed in 1.80s $ pytest -q [...] 1 failed, 4 passed in 1.67s It passes, then it fails. The same thing is happening again. Are we back to ground zero? Why we did all so much work if the result has to be all the same?\nI feel your pain. Let me explain why. You have to believe that we made a progress and not stuck at the same stage.\nOur test_post_request_with_proper_body_returns_201 is failing and that is correct behavior according to me. If a database has a row, which shouldn\u0026rsquo;t be there, it will fail. If we look at the Cleanup section of Anatomy of a test, it says that one test should not affect other test. The solution is to remove this row when this test is run.\nWe can use something called fixtures (or setUp/tearDown stuff if you are from xUnit background)\nNote: You might have one question in your mind right now. i.e. Why did we created so much classes and did so much refactoring when we could have done this with postgres itself. The answer to this is simple. First, we use a file based database which we can delete it any time. Secondly, as you might have guessed, it\u0026rsquo;s easy to deal with files on CD servers than with database servers.\nUsing fixtures to delete sqlite as needed # We all now know that deleting the sqlite database file before test_post_request_with_proper_body_returns_201 is the way to go. We are not going to set setup for test in this fixture. If you read first few paragraphs of Fixtures How-to, you\u0026rsquo;ll get a basic understanding of it.\nOur fixture is simple, it\u0026rsquo;s a function which will delete the database file. And because it\u0026rsquo;s a fixture, we\u0026rsquo;ll wrap it with pytest.fixture decorator.\n1 2 3 @pytest.fixture def delete_database(): os.unlink(\u0026#34;./test.db\u0026#34;) And if you want to use a fixture, you gotta use that fixture name (function name) as an argument to the test function.\n1 2 - def test_post_request_with_proper_body_returns_201(self): + def test_post_request_with_proper_body_returns_201(self, delete_database): That\u0026rsquo;s all you need to get going. Make sure all your imports are there. Let\u0026rsquo;s see how are tests are doing.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 $ pytest --no-header --no-summary ========================= test session starts ========================== collected 5 items tests/test_main.py . [ 20%] tests/test_users.py .... [100%] ========================== 5 passed in 1.74s =========================== $ pytest --no-header --no-summary ========================= test session starts ========================== collected 5 items tests/test_main.py . [ 20%] tests/test_users.py .... [100%] ========================== 5 passed in 1.73s =========================== $ pytest --no-header --no-summary ========================= test session starts ========================== collected 5 items tests/test_main.py . [ 20%] tests/test_users.py .... [100%] ========================== 5 passed in 1.73s =========================== No matter how many times I run my code, it always works as expected.\nConclusion # Which method is the best?\nIn this part of the series we mostly discussed two approches of dealing with our failing tests. One is the mocking, also known as fakes which is used to fake a certain chunk of code. Another one is fixtures, also known as set up and tear down. For our current scenario, I have chosen to go with fixtures method for now.\nThinking about the future, I plan to split my testing suite into two parts in future. One is unit test, which tests only the part of application which does not directly deals with database. Another suite will be for integration testing. I\u0026rsquo;ll use SQLite to test scenario of registering same user twice.\nIf you want to delve more into mocking example and practices in Python. I would suggest to give https://realpython.com/python-mock-library/ a read.\nIf you liked this post, please share it with your network. Subscribe to the newsletter below for similar news.\n","date":"11 December 2021","externalUrl":null,"permalink":"/posts/2021/tdd-approach-to-create-an-authentication-system-with-fastapi-part-3/","section":"Posts","summary":"Introduction # If you are following this series from start, you know that we can already create user from our API. But we have our tests failing right now, which is kind of smelly. In this post, we takle with the problem in our hand and discuss some methods we can do it with. One of them is dependency injection.\n","title":"TDD Approach to Create an Authentication System With FastAPI Part 3","type":"posts"},{"content":" Introduction # In the last post in the series we did the project setup and seen some FastAPI basics along with red-green-refactor mantra of TDD. We know that FastAPI comes with inbuilt integration of SwaggerUI. We also know that FastAPI makes use of non-blocking code to make who thing lightning fast. With that said, let\u0026rsquo;s jump into our second part of the series which is about database setup and user registration.\nSeries Index # Project Setup and FastAPI introduction Database Setup and User Registration (you are here) Fix the failing test with mocking and dependency injection Authentication Premier and Login Endpoint I love Postgres. I don\u0026rsquo;t have much strong opinion regarding Postgres, but I have seen this being used in production. I have worked with MySQL in past and Postgres is quite compatible with it. MySQL knowledge is transferrable to Postgres. On top of that, Postgres brings new things to the plate. One of the best benefits is the licensing. Postgres is community driven, while MySQL is owned by Oracle. Postgres has views. I\u0026rsquo;m also a big fan of Postgres\u0026rsquo; inbuilt data types. And I love the fact that Postgres is extensible.\nWhile you read this post, take a moment to connect with me on LinkedIn.\nSetting up PostgreSQL server # I\u0026rsquo;m not going to install PostgreSQL on my host system directly. Instead, I\u0026rsquo;ll install it on Docker layer. Make sure you have docker installed already.\nI\u0026rsquo;m going to write a docker-compose.yml file.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 version: \u0026#39;3\u0026#39; services: database: image: postgres:12 container_name: postgres_database environment: - POSTGRES_PASSWORD=postgres ports: - \u0026#34;5432:5432\u0026#34; volumes: - postgres_data:/var/lib/postgresql/data volumes: postgres_data: That\u0026rsquo;s enough for getting us started. We don\u0026rsquo;t need any fancy setup right now. I could have not written this compose file and instead passed these commands as flags. But I have made this compose file for better documentation.\nThings to note here is\u0026hellip;\nI\u0026rsquo;m using version 12 of postgres. Postgres password for the user is postgres. If you head over to the environment variable section in https://hub.docker.com/_/postgres/, you\u0026rsquo;ll see that the default user is also named postgres. If you read further in that section, you\u0026rsquo;d see that default value for POSTGRES_DB is also postgres. We have mapped port 5432 inside of docker to the host system\u0026rsquo;s 5432. In simple words, to our application, the postgres is on the host system where application is running. Then there is volumes stuff around. It is for data persistency, because containers don\u0026rsquo;t retain data when they are restarted. Run the database server # To run the container out of above compose file:\n$ docker compose up This command will output quite a lot of log messages. Wait until the last line of the log says database system is ready to accept connections.\nTest the database connection # This is the time we need to test our connection from our host system to docker container running postgres. We need something called database connector. There are different database connectors for different databases. For postgres itself there are around half a dozen connectors. But I love using psycopg (the actual package name is psycopg2).\nI\u0026rsquo;ll install a package called psycopg2-binary instead of psycopg2 as the later one requires some additional steps and development packages to build it from source, which can actually be avoided in our case.\nLet\u0026rsquo;s install this python package in our environment.\npip install psycopg2-binary Now let\u0026rsquo;s enter Python REPL to create some table and insert some records into them.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 $ python Python 3.7.10 (default, Jun 3 2021, 00:02:01) [GCC 7.3.1 20180712 (Red Hat 7.3.1-13)] on linux Type \u0026#34;help\u0026#34;, \u0026#34;copyright\u0026#34;, \u0026#34;credits\u0026#34; or \u0026#34;license\u0026#34; for more information. \u0026gt;\u0026gt;\u0026gt; import psycopg2 \u0026gt;\u0026gt;\u0026gt; conn = psycopg2.connect(\u0026#34;dbname=postgres user=postgres password=postgres host=localhost\u0026#34;) \u0026gt;\u0026gt;\u0026gt; cur = conn.cursor() \u0026gt;\u0026gt;\u0026gt; cur.execute(\u0026#34;CREATE TABLE test (id serial PRIMARY KEY, num integer, data varchar);\u0026#34;) \u0026gt;\u0026gt;\u0026gt; cur.execute(\u0026#34;INSERT INTO test (num, data) VALUES (%s, %s)\u0026#34;, (100, \u0026#34;abc\u0026#39;def\u0026#34;)) \u0026gt;\u0026gt;\u0026gt; cur.execute(\u0026#34;SELECT * FROM test;\u0026#34;) \u0026gt;\u0026gt;\u0026gt; cur.fetchone() (1, 100, \u0026#34;abc\u0026#39;def\u0026#34;) \u0026gt;\u0026gt;\u0026gt; conn.commit() \u0026gt;\u0026gt;\u0026gt; cur.close() \u0026gt;\u0026gt;\u0026gt; conn.close() We enter the REPL on line 1. Then let\u0026rsquo;s jump onto line line 5 where we import psycopg2. Pay attention to the connection string on line 6 to the psycopg2.connect. The default dbname and user is set to postgres, remember from last section? Right? host we can set to localhost as we have already bind inside container port to outside. One line 7 you can see that I\u0026rsquo;m creating a cursor object. You may have heard of database cursor? On line 8, I create a table named test. It has 3 fields: id, num and data. This table resides inside the postgres database. On line 9, I insert sample data on this table. And from line 10-12 we fetch the same data to make sure the data has been written. On line 13, we commit the transaction. Why? Because of atomicity. At last, we close the cursor and the connection to the database. If any of the steps have failed, please let me know.\nNote: For sake of simplicity, we are not going to setup any custom role or databases, but in production environment security matters.\nTest if data is in fact written # This time we are going to get into the container which running the postgres server and check by hand if the record we insterted with psycopg actually persist on the db instance.\nFor getting into container, we first need to know the name of the container. The name is the same as we specified in the container_name section of docker-compose.yml file. Check by doing is docker ps.\n1 2 3 $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 04d9f5dc11c1 postgres:12 \u0026#34;docker-entrypoint.s…\u0026#34; About an hour ago Up About an hour 0.0.0.0:5432-\u0026gt;5432/tcp, :::5432-\u0026gt;5432/tcp postgres_database We can see the name of the container in the NAMES section, which is postgres_database.\nWe do a docker exec to get a shell inside the container. Then we get into postgres CLI with psql command. -U postgres is to specify the role name.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 $ docker exec -it postgres_database bash root@04d9f5dc11c1:/# psql -U postgres psql (12.8 (Debian 12.8-1.pgdg110+1)) Type \u0026#34;help\u0026#34; for help. postgres=# \\c postgres You are now connected to database \u0026#34;postgres\u0026#34; as user \u0026#34;postgres\u0026#34;. postgres=# \\dt test List of relations Schema | Name | Type | Owner --------+------+-------+---------- public | test | table | postgres (1 row) postgres=# SELECT * FROM public.test; id | num | data ----+-----+--------- 1 | 100 | abc\u0026#39;def (1 row) postgres=# The first thing after I got into postgres shell is to connect to the database postgres with the command \\c postgres. postgres is the default database created for us when we start the postgres container for first time. You can override the default. Read Environment Variables section of https://hub.docker.com/_/postgres/. To describe the table (or say to show the schema of the table), I issued \\dt test. This is exactly the same as we specified with psycopg. Finally I did a SELECT * FROM public.test to show all the entries in the test table. And we get the same result as we got with Python REPL with psycopg. It is now confirmed that psycopg is doing it\u0026rsquo;s work. This is the last time we are touching psycopg in this tutorial. Later on, it will be handled by SQLAlchemy.\nSQLAlchemy layer between database and app layer # You know what I like about the SQLAlchemy the most? It helps prevent the vendor lock-in in long term. If I want to use another SQL based RDBMS tomorrow, I just need to swap the connection string and install some database connector package.\nWhy SQLAlchemy # Another good thing about SQLAlchemy is it\u0026rsquo;s Object Relational Mapper. Spend more time writing business logic and less time dealing with app to database interaction. Most common interactions are already handled by ORMs.\nBut it has downside too\u0026hellip; If you are a database ninja, you may find yourself crippled to the ORMs way of doing things. In that case you should write your own queries.\nUnless you are keen towards database management and want them to study inside out. Please use SQLAlchemy or something similar to deal with connection pooling and stuff.\nPrepare the database # I will go ahead and create a database and a table for the authentication system we are building.\nTo do so we\u0026rsquo;ll get into the container like we did before, and create a database called fastauth. Like so:\n1 CREATE DATABASE fastauth; That is all we need, and rest will be taken care by SQLAlchemy. But just for sake of information, I\u0026rsquo;ll put up the schema here.\n1 2 3 4 5 6 CREATE TABLE user_info( id serial PRIMARY KEY, username VARCHAR(50) NOT NULL, password VARCHAR(500) NOT NULL, fullname VARCHAR(50) NOT NULL ); Now we can go ahead and write python files which will be responsible for connecting with the database.\nAlthough I can go ahead and write all the logic in a single file, but I will create 4 new files just for sake of self-documentation.\ndatabase.py: for connecting postgres models.py: the jam of the sandwhich which talks to both side of the loaf, aka FastAPI and postgres schemas.py: to marshal/unmarshal data on/off from/to request/response crud.py: actions to manipulate data at postgres If I have to catogarize above files into database application code. I would say that database.py and models.py are more of database related files, schemas.py is for both, and crud.py is mostly the application code.\nWrite database related code # database.py: # 1 2 3 4 5 6 7 8 9 10 11 from sqlalchemy import create_engine from sqlalchemy.ext.declarative import declarative_base from sqlalchemy.orm import sessionmaker SQLALCHEMY_DATABASE_URL = \u0026#34;postgresql+psycopg2://postgres:postgres@localhost/fastauth\u0026#34; engine = create_engine(SQLALCHEMY_DATABASE_URL) SessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine) Base = declarative_base() So what are create_engine, declarative_base and sessionmaker? Let\u0026rsquo;s go through one by one:\ncreate_engine: create_engine creates a new Engine instance. Which basically let\u0026rsquo;s SQLAlchemy deal with most of the database stuff. As you can see, I have passed create_engine a database string to connect to our postgres instance. Similarly, connection string for various other db servers can be passed. MySQL? Oracle? SQLite? MSSQL? There are many other dialects that SQLAlchemy supports.\nsessionmaker: is a factory function which creates an insance of Session. Sessions can be used with Python\u0026rsquo;s context managers amoung various other features. Dig deeper with sessions at Using the Session. We have bind our engine with the session. In other words, session is using our engine.\ndeclarative_base: Pardon my database knowledge, but declarative_base has something to do with models we are going to create in below sections. It maps directly to the tables we have in the database system. Read more about object-relation mapping at Mapping Python Classes\nmodels.py: # 1 2 3 4 5 6 7 8 9 10 from sqlalchemy import Column, Integer, String from .database import Base class UserInfo(Base): __tablename__ = \u0026#34;user_info\u0026#34; id = Column(Integer, primary_key=True, index=True) username = Column(String, unique=True) password = Column(String) fullname = Column(String) Here we have used the declarative_base to create a UserInfo model. This represents a table in the database. We have different fields in the table such as id, username, password and fullname. The above code is self explanatory. The parameters to Column is the data type and some other column properties. Don\u0026rsquo;t be shy to lookup on internet if you don\u0026rsquo;t know about them.\nschemas.py: # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 from typing import List from pydantic import BaseModel class UserInfoBase(BaseModel): username: str fullname: str class UserCreate(UserInfoBase): password: str class UserInfo(UserInfoBase): id: int class Config: orm_mode = True Schemas created here will be used for validation of HTTP requests. And also for sending responses. It won\u0026rsquo;t make much sense now. But I\u0026rsquo;ll poke you again when dealing with response and request.\ncrud.py: # Our crud.py is mostly the application code which uses models and schemas.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 from sqlalchemy.orm import Session import bcrypt from . import models, schemas def get_user_by_username(db: Session, username: str): return db.query(models.UserInfo).filter(models.UserInfo.username == username).first() def create_user(db: Session, user: schemas.UserCreate): hashed_password = bcrypt.hashpw(user.password.encode(\u0026#39;utf-8\u0026#39;), bcrypt.gensalt()) db_user = models.UserInfo(username=user.username, password=hashed_password, fullname=user.fullname) db.add(db_user) db.commit() db.refresh(db_user) return db_user Reorganise files # Put all 4 files in a directory with same name as root directory.\nAlso create a empty file named __init__.py in the same directory. In my case, I\u0026rsquo;ve put them in fastauth.\nI have also moved the test_main.py to a dedicated tests/ directory. It also has the same __init__.py.\nHere is what my current directory structure looks like.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 $ tree . ├── docker-compose.yml ├── fastauth │ ├── crud.py │ ├── database.py │ ├── __init__.py │ ├── models.py │ └── schemas.py ├── main.py └── tests ├── __init__.py └── test_main.py 2 directories, 9 files $ pwd /home/ec2-user/workspace/fastauth Define requirement for registration endpoint # We have come a long way. Kudos to you.\nGetting back to our main.py which has the only endpoint which is /ping. We need to add another endpoint to /users to our service. But before that we need to write tests. And to write tests, we need to have the requirements done.\nThinks about what our endpoints will look like in a finished product. Here is what I can think of:\nThe consumer hits /users/register with GET method should get Method Not Allowed.\nThe API consumer hits /users/register with a POST method:\nWithout body = Should get error about required body parameters. With body: Should check in database if the user exists, if so: Throws error that user exists If user not found, returns a 201 response with passed username. Let\u0026rsquo;s do this much first, then we\u0026rsquo;ll continue with the login system.\nFunctional requirments turned into test # tests/test_users.py # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 import pytest from fastapi.testclient import TestClient from main import app client = TestClient(app) class TestUserRegistration: \u0026#34;\u0026#34;\u0026#34;TestUserRegistration tests /users/register\u0026#34;\u0026#34;\u0026#34; def test_get_request_returns_405(self): \u0026#34;\u0026#34;\u0026#34;registration endpoint does only expect a post request\u0026#34;\u0026#34;\u0026#34; response = client.get(\u0026#34;/users/register\u0026#34;) assert response.status_code == 405 def test_post_request_without_body_returns_422(self): \u0026#34;\u0026#34;\u0026#34;body should have username, password and fullname\u0026#34;\u0026#34;\u0026#34; response = client.post(\u0026#34;/users/register\u0026#34;) assert response.status_code == 422 def test_post_request_with_improper_body_returns_422(self): \u0026#34;\u0026#34;\u0026#34;all of username, password and fullname is required\u0026#34;\u0026#34;\u0026#34; response = client.post( \u0026#34;/users/register\u0026#34;, json={\u0026#34;username\u0026#34;: \u0026#34;santosh\u0026#34;} ) assert response.status_code == 422 def test_post_request_with_proper_body_returns_201(self): response = client.post( \u0026#34;/users/register\u0026#34;, json={\u0026#34;username\u0026#34;: \u0026#34;santosh\u0026#34;, \u0026#34;password\u0026#34;: \u0026#34;sntsh\u0026#34;, \u0026#34;fullname\u0026#34;: \u0026#34;Santosh Kumar\u0026#34;} ) assert response.status_code == 201 I have left one test here. The one which said Should check in database if the user exists, if so: Throws error that user exists. That is because I wanted to show you something. Let\u0026rsquo;s see what I\u0026rsquo;ve modified in main.py\nImplementation of the test # I am skipping the red-green-refactor thing here because I know what the requirements are and for the sake of this tutorial it would be too much detail. Anyway be have already seen what red-green-refactor feels like in our last post.\nLet\u0026rsquo;s fulfill our test cases in main.py.\nmain.py # I have added new bits together to use our database integration layer into our application. Instead of showing the entire file, I\u0026rsquo;ll just show the changes.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 diff --git a/main.py b/main.py index 47722cd..9c5eab9 100644 --- a/main.py +++ b/main.py @@ -1,7 +1,28 @@ -from fastapi import FastAPI +from fastapi import FastAPI, Depends, HTTPException +from sqlalchemy.orm import Session + +from fastauth import models, schemas, crud +from fastauth.database import engine, SessionLocal + +models.Base.metadata.create_all(bind=engine) + +def get_db(): + db = None + try: + db = SessionLocal() + yield db + finally: + db.close() app = FastAPI() @app.get(\u0026#34;/ping\u0026#34;) async def ping(): return {\u0026#39;msg\u0026#39;: \u0026#39;pong\u0026#39;} + +@app.post(\u0026#34;/users/register\u0026#34;, status_code=201, response_model=schemas.UserInfo) +def register_user(user: schemas.UserCreate, db: Session = Depends(get_db)): + db_user = crud.get_user_by_username(db, username=user.username) + if db_user: + raise HTTPException(status_code=409, detail=\u0026#34;Username already registered\u0026#34;) + return crud.create_user(db=db, user=user) Let\u0026rsquo;s just straight to line 29 and start from there. Not a single bit is extra here. Every token has some significance.\nOn line 29, we have the route /users/register for which we are going to define a handler on the next line. On success, it will respond with 201. Do you remember I told you that I\u0026rsquo;ll poke you when dealing with schemas? You can see them in use on line 29 and 30. When response_model is set to schemas.UserInfo, it means that on success, POST to /users/register responds with fields mentioned in schemas.UserInfo. Similarly, input to register_user() is user i.e. schemas.UserCreate. Meaning that /users/register tries to find every field from this schema, otherwise errors out. register_user also depends on db to function. On line 31-34 we simply see if the specified user already exists in database. If so, respond with a 400. Otherwise go ahead and insert a row in the user_info table with crud.create_user API we defined. Related reading: https://swagger.io/resources/articles/best-practices-in-api-design/\nTest /users/register # At first, all the test should pass if you are running on local and your database instance is running.\n1 2 3 4 5 6 7 8 9 10 11 $ pytest ================================ test session starts ================================== platform linux -- Python 3.7.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 rootdir: /efs/repos/fastauth plugins: anyio-3.3.4 collected 5 items tests/test_main.py . [ 20%] tests/test_users.py .... [100%] ================================= 5 passed in 1.71s =================================== The second time, test shouldn\u0026rsquo;t pass:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 $ pytest ================================================== test session starts ================================================== platform linux -- Python 3.7.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 rootdir: /efs/repos/fastauth plugins: anyio-3.3.4 collected 5 items tests/test_main.py . [ 20%] tests/test_users.py ...F [100%] ======================================================= FAILURES ======================================================== __________________________ TestUserRegistration.test_post_request_with_proper_body_returns_201 __________________________ self = \u0026lt;tests.test_users.TestUserRegistration object at 0x7fa6570ef8d0\u0026gt; def test_post_request_with_proper_body_returns_201(self): response = client.post( \u0026#34;/users/register\u0026#34;, json={\u0026#34;username\u0026#34;: \u0026#34;santosh\u0026#34;, \u0026#34;password\u0026#34;: \u0026#34;sntsh\u0026#34;, \u0026#34;fullname\u0026#34;: \u0026#34;Santosh Kumar\u0026#34;} ) \u0026gt; assert response.status_code == 201 E assert 409 == 201 E + where 409 = \u0026lt;Response [409]\u0026gt;.status_code tests/test_users.py:39: AssertionError ================================================ short test summary info ================================================ FAILED tests/test_users.py::TestUserRegistration::test_post_request_with_proper_body_returns_201 - assert 409 == 201 ============================================== 1 failed, 4 passed in 1.19s ============================================== There are a lot of things going on here. I\u0026rsquo;ll leave this as a suspense.\nConclusion # In last post we setup our environment. In this post we implemented our user registration system. But unfortunately our tests only pass for a single time. In next post I will discuss why this happens and what are the best practices we can apply here to fix the failing test. Until then, have a nice weekend.\nIf you liked this post, please share it with your network. Subscribe to the newsletter below for similar news.\n","date":"11 December 2021","externalUrl":null,"permalink":"/posts/2021/tdd-approach-to-create-an-authentication-system-with-fastapi-part-2/","section":"Posts","summary":"Introduction # In the last post in the series we did the project setup and seen some FastAPI basics along with red-green-refactor mantra of TDD. We know that FastAPI comes with inbuilt integration of SwaggerUI. We also know that FastAPI makes use of non-blocking code to make who thing lightning fast. With that said, let’s jump into our second part of the series which is about database setup and user registration.\n","title":"TDD Approach to Create an Authentication System With FastAPI Part 2","type":"posts"},{"content":"","date":"5 December 2021","externalUrl":null,"permalink":"/categories/cloud/","section":"Categories","summary":"","title":"Cloud","type":"categories"},{"content":" Introduction # This is second and final post in the series of creating a S3 thumbnail system with Lambda.\nYou must already be aware of the prerequisites from the last section. To follow this post, familarity with Python would benifit.\nThis is second part of the 2 section tutorial:\nTest the Trigger Make the Thumbnail (you are here) We are doing Make the Thumbnail.\nOverview # In this post, we\u0026rsquo;ll build upon the resources we provisioned in the last post of this series. If you have not yet covered that, please go and cover.\nWe have already provisioned a Lambda trigger function and the source bucket.\nAfter you have done that. You can head over to the next section.\nPrerequisites # Although we are not going to do any rocket science here, but prior familarity with following will help, but is not important.\nIAM AWS CLI Bash With that said, when we had no dependency in our Lambda function; as seen in our last post. Then we simply edited the code from the online editor. But as you know that Python standard library does not have anything to deal with images elegantly. In that case, we need to leverage libraries such as Pillow. When we work with third-party libraries, we have to package the dependency with the code itself; as a zip.\nCheck logs of previous invocations in CloudWatch # We have temp-thumbnail-trigger Lambda function which we worked with in last post. Let\u0026rsquo;s check their execution log.\nStep 1: # Note: Please use single quotes, as both single and double quotes have different meaning for bash.\n1 2 $ aws logs describe-log-groups --query \u0026#39;logGroups[*].logGroupName\u0026#39; | grep thumbnail \u0026#34;/aws/Lambda/temp-thumbnail-trigger\u0026#34;, There are logs for multiple AWS service in CloudWatch. We can see logs for all the services/arns. describe-log-groups subcommand lists all of such loggable service aka log groups. The output to describe-log-groups is very verbose (it\u0026rsquo;s a JSON), and I have narrowed it down to logGroups.logGroupName only.\nMake note of the output we received, we are gonna use that in our next command.\nStep 2: # Now we want to see something called log streams for this particular log group. Each stream is a result of an invocation here (correct me if I am wrong here).\nI have filtered it down the output to logStreams.logStreamName\n1 2 3 4 5 $ aws logs describe-log-streams --log-group-name \u0026#39;/aws/Lambda/temp-thumbnail-trigger\u0026#39; --query \u0026#39;logStreams[*].logStreamName\u0026#39; [ \u0026#34;2021/11/14/[$LATEST]f0c4ebb44233445ab27765977016f27f\u0026#34;, \u0026#34;2021/11/14/[$LATEST]f5d859c9009441c9a8f0cf60b33b47de\u0026#34; ] The above output shows that I have invoked this Lambda for 2 times. The second one is the latest one. I am going to use the second output in next step.\nStep 3: # We are now looking for log events in this particular log stream \u0026quot;2021/11/14/[$LATEST]f5d859c9009441c9a8f0cf60b33b47de\u0026quot;.\n1 2 3 4 5 6 7 8 $ aws logs get-log-events --log-group-name \u0026#39;/aws/Lambda/temp-thumbnail-trigger\u0026#39; --log-stream-name \u0026#39;2021/11/14/[$LATEST]f5d859c9009441c9a8f0cf60b33b47de\u0026#39; --query \u0026#39;events[*].message\u0026#39; [ \u0026#34;Loading function\\n\u0026#34;, \u0026#34;START RequestId: f168a35d-d4a1-4edc-909d-cd1f24612cd5 Version: $LATEST\\n\u0026#34;, \u0026#34;CONTENT TYPE: binary/octet-stream\\n\u0026#34;, \u0026#34;END RequestId: f168a35d-d4a1-4edc-909d-cd1f24612cd5\\n\u0026#34;, \u0026#34;REPORT RequestId: f168a35d-d4a1-4edc-909d-cd1f24612cd5\\tDuration: 226.76 ms\\tBilled Duration: 227 ms\\tMemory Size: 128 MB\\tMax Memory Used: 72 MB\\tInit Duration: 420.19 ms\\t\\n\u0026#34; ] As you can see, the Lambda has been invoked without raising any exceptions. Good sign.\nNow you know how to check CloudWatch logs with aws cli.\nCreate destination bucket # Choose whatever name you want. We are anyway going to reference this name in the Lambda code.\n1 2 3 4 $ aws s3api create-bucket --bucket dest-bucket-sntshk --region ap-south-1 --create-bucket-configuration LocationConstraint=ap-south-1 { \u0026#34;Location\u0026#34;: \u0026#34;http://dest-bucket-sntshk.s3.amazonaws.com/\u0026#34; } Update role policy # If you got an AccessDenied error in last post while testing the Lambda, that was because by default the Lambda has only permission to talk to CloudWatch (via role). After I attached a policy with GetObject permission to the role, Lambda was able to talk to AWS as well.\nThis time we need to attach one more policy to the role. This time we want to PutObject to our dest-bucket-sntshk (you will have a different name).\nFind the policy # If you followed the last post as it was, you should be able to run the following command and look for \u0026lt;funcname\u0026gt;-\u0026lt;role\u0026gt;-\u0026lt;sha\u0026gt;.\n1 $ aws iam list-roles --query \u0026#39;Roles[*].RoleName\u0026#39; I tried piping the output to grep, but it did work. But I was able to find temp-thumbnail-trigger-role-fc1p90sl.\nUpdate the policy # We\u0026rsquo;ll use the same method we used in last post to attach the new inline policy to put object in destination bucket.\nYou can update the existing role to add or update the policy from the web interface.\nMy temp-thumbnail-trigger-role-fc1p90sl has three policy now.\nThe default policy created for writing to CloudWatch.\ns3-get-object policy which has s3:GetObject access from arn:aws:s3:::source-bucket-sntshk/*.\n1 2 3 4 5 6 7 8 9 10 11 { \u0026#34;Version\u0026#34;: \u0026#34;2012-10-17\u0026#34;, \u0026#34;Statement\u0026#34;: [ { \u0026#34;Sid\u0026#34;: \u0026#34;VisualEditor0\u0026#34;, \u0026#34;Effect\u0026#34;: \u0026#34;Allow\u0026#34;, \u0026#34;Action\u0026#34;: \u0026#34;s3:PutObject\u0026#34;, \u0026#34;Resource\u0026#34;: \u0026#34;arn:aws:s3:::dest-bucket-sntshk/*\u0026#34; } ] } s3-put-object policy which has s3:PutObject access to arn:aws:s3:::dest-bucket-sntshk/*. 1 2 3 4 5 6 7 8 9 10 11 { \u0026#34;Version\u0026#34;: \u0026#34;2012-10-17\u0026#34;, \u0026#34;Statement\u0026#34;: [ { \u0026#34;Sid\u0026#34;: \u0026#34;VisualEditor0\u0026#34;, \u0026#34;Effect\u0026#34;: \u0026#34;Allow\u0026#34;, \u0026#34;Action\u0026#34;: \u0026#34;s3:PutObject\u0026#34;, \u0026#34;Resource\u0026#34;: \u0026#34;arn:aws:s3:::dest-bucket-sntshk/*\u0026#34; } ] } Prepare a deployment package # In this section we\u0026rsquo;ll the the actual code to do the transformation and then bundle the third-party package in the bundle.\nWrite actual code to generate thumbnail # This is the heart of the tutorial series. We are going to use Python to create our Lambda function. Let\u0026rsquo;s get started with the code first, then I\u0026rsquo;ll explain line by line.\nGo ahead and create a directory on your local machine and create a file called lambda_function.py (naming matters).\nlambda_function.py # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 import boto3 import uuid from urllib.parse import unquote_plus from PIL import Image import PIL.Image s3_client = boto3.client(\u0026#39;s3\u0026#39;) def resize_image(image_path, resized_path): with Image.open(image_path) as image: image.thumbnail(tuple(x / 2 for x in image.size)) image.save(resized_path) def lambda_handler(event, context): for record in event[\u0026#39;Records\u0026#39;]: bucket = record[\u0026#39;s3\u0026#39;][\u0026#39;bucket\u0026#39;][\u0026#39;name\u0026#39;] key = unquote_plus(record[\u0026#39;s3\u0026#39;][\u0026#39;object\u0026#39;][\u0026#39;key\u0026#39;]) tmpkey = key.replace(\u0026#39;/\u0026#39;, \u0026#39;\u0026#39;) download_path = \u0026#39;/tmp/{}{}\u0026#39;.format(uuid.uuid4(), tmpkey) upload_path = \u0026#39;/tmp/resized-{}\u0026#39;.format(tmpkey) s3_client.download_file(bucket, key, download_path) resize_image(download_path, upload_path) s3_client.upload_file(upload_path, \u0026#39;dest-bucket-sntshk\u0026#39;, key) Unlike what I usually do, I\u0026rsquo;ll start explaining from line 16:\nLine 16 defines a function called lambda_handler which is the entry point of the Lambda. And you see those event \u0026amp; context? event is the data that\u0026rsquo;s passed to the function upon execution. An event can come from a various number of places.. API Gateway, another Lambda and so on, each intermediate service in between can add or remove contents of the event. And for context, the main role of a context is to provide information about the current execution environment. You can read about context on AWS docs. One the just very next line we are running a for loop for all the records. Yes, the event which is coming to our Lambda will have one or more record. For each record, we are fetching the bucket name from the record dict, and the file name which is to be process. You can see we are using bracket notation to access the names from the dict. unquote_plus is to URL decoding. Line 18 to 20 deals with creating path variables for fetching and uploading images. On line 21 and 23 we do the actual fetching and uploading to the specific buckets. On line 22 we have a function call to resize_image, which is defined on line 9 through 12. resize_image simply reduces height and width of the image by a factor of 2. After resizing it saves to specified path. Package Pillow with the deployment bundle # As you know, we depend on an external package called Pillow which provides PIL in our source code. Lambda does not supports doing a pip install and rather offload this work on the function maintainer.\npip install --target ./package Pillow After above command, my directory structure looks something like this:\n1 2 3 4 5 6 7 8 9 $ tree -L 2 . ├── lambda_function.py └── package ├── PIL ├── Pillow-8.4.0.dist-info └── Pillow.libs 4 directories, 1 file The point to be noted here is we need to get into the package directory now. And then make zip of this directory in the parent directory.\n$ cd package $ zip -r ../my-deployment-package.zip . adding: Pillow.libs/ (stored 0%) adding: Pillow.libs/libwebpdemux-f117ddb4.so.2.0.8 (deflated 72%) adding: Pillow.libs/liblzma-d540a118.so.5.2.5 (deflated 66%) ... Now that we have created the zip with the dependencies, we will go ahead and include the function inside the zip.\nI\u0026rsquo;ll go back to the parent directory here:\n$ cd .. $ zip -g my-deployment-package.zip lambda_function.py adding: lambda_function.py (deflated 54%) Create a deployment to existing Lambda function # aws lambda update-function-code --function-name temp-thumbnail-trigger --zip-file fileb://my-deployment-package.zip Updating the function code would respond in something similar to this:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 { \u0026#34;FunctionName\u0026#34;: \u0026#34;temp-thumbnail-trigger\u0026#34;, \u0026#34;FunctionArn\u0026#34;: \u0026#34;arn:aws:lambda:ap-south-1:XXXXXXXXXXXX:function:temp-thumbnail-trigger\u0026#34;, \u0026#34;Runtime\u0026#34;: \u0026#34;python3.8\u0026#34;, \u0026#34;Role\u0026#34;: \u0026#34;arn:aws:iam::XXXXXXXXXXXX:role/service-role/temp-thumbnail-trigger-role-fc1p90sl\u0026#34;, \u0026#34;Handler\u0026#34;: \u0026#34;lambda_function.lambda_handler\u0026#34;, \u0026#34;CodeSize\u0026#34;: 3413118, \u0026#34;Description\u0026#34;: \u0026#34;An Amazon S3 trigger that retrieves metadata for the object that has been updated.\u0026#34;, \u0026#34;Timeout\u0026#34;: 3, \u0026#34;MemorySize\u0026#34;: 128, \u0026#34;LastModified\u0026#34;: \u0026#34;2021-11-25T10:12:36.000+0000\u0026#34;, \u0026#34;CodeSha256\u0026#34;: \u0026#34;XSW5FerVZKYdDvRG+XIXKU9Rlaj6xntRb2Ja3etanZU=\u0026#34;, \u0026#34;Version\u0026#34;: \u0026#34;$LATEST\u0026#34;, \u0026#34;TracingConfig\u0026#34;: { \u0026#34;Mode\u0026#34;: \u0026#34;PassThrough\u0026#34; }, \u0026#34;RevisionId\u0026#34;: \u0026#34;7946029f-b70d-43ef-8176-06818f8bfea1\u0026#34;, \u0026#34;State\u0026#34;: \u0026#34;Active\u0026#34;, \u0026#34;LastUpdateStatus\u0026#34;: \u0026#34;InProgress\u0026#34;, \u0026#34;LastUpdateStatusReason\u0026#34;: \u0026#34;The function is being created.\u0026#34;, \u0026#34;LastUpdateStatusReasonCode\u0026#34;: \u0026#34;Creating\u0026#34;, \u0026#34;Architectures\u0026#34;: [ \u0026#34;x86_64\u0026#34; ] } Test the function with new update # Testing is part of developers life. The test we are doing by manually invoking the test from the UI is know as ad-hoc testing. Here I present you a gif.\nSuccessful creation of thembnail from S3 trigger On the other hand, if you are facing a cannot import name '_imaging' from 'PIL' error when testing, next section is for you.\ncannot import name \u0026lsquo;_imaging\u0026rsquo; from \u0026lsquo;PIL\u0026rsquo; # There could be a case where you\u0026rsquo;d get a traceback like this:\n1 2 [ERROR] Runtime.ImportModuleError: Unable to import module \u0026#39;lambda_function\u0026#39;: cannot import name \u0026#39;_imaging\u0026#39; from \u0026#39;PIL\u0026#39; (/var/task/PIL/__init__.py) Traceback (most recent call last): There is incompatibility between the version of Pillow being used with that of Python. For me the fix was simple. I updated the Lambda runtime from Python 3.7 -\u0026gt; 3.8.\ncannot import name \u0026lsquo;_imaging\u0026rsquo; from \u0026lsquo;PIL\u0026rsquo; fix The fix could be different if you are following this tutorial anytime in future.\nHomework # As a homework, you can try to maintaining the hierarchy from the source bucket.\nConclusion # Today we saw how we can leverage S3 triggers to chain lambda function to it. But we are not limited to Lambdas, we can push to event to an SNS or SQS queue. And let some other consumer parse the event and work on it.\nAs already might have noticed, you are also not only limited to PUT events. You can leverage other types of CRUD event to trigger various other handlers. Possibilities are endless.\nIf you liked this post, please share it with your network. Subscribe to the newsletter below for similar news.\n","date":"5 December 2021","externalUrl":null,"permalink":"/posts/2021/create-thumbnail-worker-with-s3-and-lambda-make-the-thumbnail/","section":"Posts","summary":"Introduction # This is second and final post in the series of creating a S3 thumbnail system with Lambda.\nYou must already be aware of the prerequisites from the last section. To follow this post, familarity with Python would benifit.\nThis is second part of the 2 section tutorial:\nTest the Trigger Make the Thumbnail (you are here) We are doing Make the Thumbnail.\n","title":"Create Thumbnail Worker With S3 and Lambda: Make the Thumbnail","type":"posts"},{"content":"","date":"5 December 2021","externalUrl":null,"permalink":"/tags/s3/","section":"Tags","summary":"","title":"S3","type":"tags"},{"content":"","date":"5 December 2021","externalUrl":null,"permalink":"/series/s3-thumbnail-system/","section":"Series","summary":"","title":"S3 Thumbnail System","type":"series"},{"content":"","date":"27 November 2021","externalUrl":null,"permalink":"/tags/tls/","section":"Tags","summary":"","title":"Tls","type":"tags"},{"content":" Introduction # I have already posted about how we can automate installation of Jenkins \u0026amp; Nginx with Ansible. I have also done a post where I talk about how to enable HTTPS on a non-wildcard basis i.e. only for the root domain and not on subdomain.\nToday I\u0026rsquo;ll go through how go get and configure a HTTPS certificate from Let\u0026rsquo;s Encrypt for all the subdomain. I\u0026rsquo;ll automate all these using Ansible.\nFeel free to connect with me on LinkedIn.\nRecap # We have previously done:\nInstallation of nginx with ansible. Installation of Jenkins with ansible As a matter of fact, it\u0026rsquo;s very daunting nowadays to do Jenkins operations on a naked HTTP. I need some kind of privacy, and for that I\u0026rsquo;ll get a certificate from Let\u0026rsquo;s Encrypt to use it to enable HTTPS on my domain.\nWhat you\u0026rsquo;ll need # To be able to enable HTTPS on a domain, you need:\nA domain A VPS with sudo access. I use Amazon EC2. Some knowledge of nginx and reverse proxy. Some knowledge of ansible (optional). If you are new to Ansible, please refer to the Recap section.\nWhat we\u0026rsquo;ll cover # This post is chunked into 3 parts:\nWhat are Ansible roles? Enabling HTTPS on multiple sudomains with nginx and AWS Route53 Making an Ansible automation for the procedure You can also find the index on the right of the page if you are reading this post on a desktop.\nWhat are roles? # I didn\u0026rsquo;t cover roles in previous post pretty well. Now that I have more knowledge of it than before. I\u0026rsquo;ll expand on that post.\nRoles are nothing more than an organised directory structure. Everything directory has a significant, and it helps us deal with provisioning at scale.\nDirectory structure at the time of Hello World # Previously my ansible directory looked similar to this:\n1 2 3 4 5 6 7 8 9 10 $ tree . ├── ansible.cfg ├── inventory ├── jenkins.yml ├── nginx.conf ├── nginx.yml └── README.md 0 directories, 6 files That\u0026rsquo;s totally flat.\nMy nginx.yml looks like this:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 --- - name: install and start nginx hosts: web become: yes tasks: - name: install nginx command: amazon-linux-extras install nginx1 -y - name: copy config for jenkins routing copy: src: nginx.conf dest: \u0026#39;/etc/nginx/nginx.conf\u0026#39; mode: preserve notify: restart nginx - name: start nginx service: name: nginx state: started - name: get public IP uri: url: https://api.ipify.org?format=json method: Get changed_when: false register: public_ip - name: print public IP debug: msg: \u0026#34;{{ public_ip.json.ip }}\u0026#34; handlers: - name: restart nginx service: name: nginx Directory structure after learning about Roles # Roles are all about directory structures. See my directory tree listing below:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 ├── ansible.cfg ├── inventory ├── playbooks │ ├── jenkins.yml │ └── nginx.yml ├── README.md └── roles ├── jenkins │ └── tasks │ └── main.yml └── nginx ├── files │ └── nginx.conf ├── tasks │ └── main.yml └── handlers └── main.yml 8 directories, 9 files If I take the example of nginx here, the whole thing is divided into 4 files. I present the listing with file content.\nplaybooks/nginx.yml # 1 2 3 4 5 6 7 --- - name: install and start nginx become: yes hosts: - web roles: - nginx A couple things:\nThis looks familiar to our pre roles era nginx yaml file. Yes! we had hosts section listed at top. But now we have a section called roles. And you know what\u0026rsquo;s good thing about roles? You can have multiple of them in a single playbook.\nEach roles listed here is mapped to a directory inside the roles directory.\nYou can have your roles named anything else, you just have to override roles_path config inside ansible.cfg.\nroles/nginx/tasks/main.yml # The first the first file I\u0026rsquo;ll discuss. Ordering matters here as I\u0026rsquo;m teaching transitioning between pre roles era monolith yaml file into logical chunks.\nFirst let\u0026rsquo;s see the contents:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 - name: install nginx command: amazon-linux-extras install nginx1 -y - name: copy config copy: \u0026#34;src=nginx.conf dest=\u0026#39;/etc/nginx/nginx.conf\u0026#39; mode=preserve\u0026#34; notify: restart nginx - name: start nginx service: \u0026#34;name=nginx state=started\u0026#34; - name: get public IP uri: url: https://api.ipify.org?format=json method: Get changed_when: false register: public_ip - name: print public IP debug: msg: \u0026#34;{{ public_ip.json.ip }}\u0026#34; Significant amount of change here. Most common of all is that I have switched from a multiple lines of declaration to a single line declaration for a module. This is to preserve space.\nSome more changes below:\nPlease note on the location of this file in the hierarchy. It says tasks. It has a file called main.yaml. Everything in monolith nginx.yaml from the sections tasks is listed here. Nothing more. It has to be tasks block. main.(yml|yaml) is the default file which ansible interpreter looks for when scanning sub-roles directories. This also means that you can have more than one file inside any of sub-roles directory. tasks/main.yml can have conditionals. This hepls in the case where you want to write a cross-platform playbook. Now that apt and yum are two different package managers. You can actually run a command to check the platform and import platform specific tasks file from the directory. roles/nginx/handlers/main.yml # Nothing fancy here. We have handlers section from the monolith written here.\n1 2 3 4 5 6 --- - name: restart nginx service: name: nginx state: restarted enabled: yes roles/nginx/files/nginx.conf # We have nothing fancy here. This is the same file we created in the last post.\nAt the end, I would ask you to refer to Role directory structure, because files, tasks and handlers are not the only sub-directories which are allowed in the roles directory.\nWhen everything is setup, this is how the whole thing is invoked:\nansible-playbook -i inventory playbooks/nginx.yml In the same way, I am leaving this onto you to create jenkins role from the monolith playbook.\nWe\u0026rsquo;ll move to next next section now, which is about enabling HTTPS on nginx. But before you move to next section, run those playbooks on the host to have nginx and jenkins installed.\nEnabling HTTPS on domain(s), manually # Before I start this section, I have these already available to me.\nA spare domain called santosh.pictures, which resides on AWS Route53. A EC2 host running nginx which is publically facing to world on port 80, and jenkins which is reversed proxied by nginx; originally running on port 8080, but routed to /. A public hosted zone in which I have record for ci.santosh.pictures which points to above nginx instance. In fact, I can access the non-https version of website when I go to ci.santosh.pictures.\nNon-HTTPS version of ci.santosh.pictures What I want here is to have this Jenkins available on https://ci.santosh.pictures.\nStep 1: Install Certbot and Route53 authenticator # Trivia: Both ansible and certbot are written in Python.\nWhat is certbot? # certbot is a program written by EFF to obtain certs from Let\u0026rsquo;s Encrypt and (optionally) auto-enable HTTPS on your server.\ncertbot talk\u0026rsquo;s with Let\u0026rsquo;s Encrypt which is a certificate authority which issues X.509 certificates which in turn are used in Internet protocols such as TLS/SSL, which is the basis for HTTPS, the secure protocol for browsing the web.\nInstall certbot\nI have been using Amazon Linux 2 till now. After a lot of hit and trial I believe that it\u0026rsquo;s straightforward to install and configure certbot on Debian because of it\u0026rsquo;s relatively new packages.\nCurrently I\u0026rsquo;m using Ubuntu 20.04 so I\u0026rsquo;ll do this:\nsudo apt install certbot This installs certbot for Python 3. Nn the other hand Amazon Linux 2 installs for Python 2 which kinda messes things up.\nInstalling certbot would be enough if we were not doing wildcard certs. But that\u0026rsquo;s not the case here.\n1 2 3 4 5 6 7 8 9 10 11 12 13 ubuntu@ip-10-2-1-10:~$ certbot plugins - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - * standalone Description: Spin up a temporary webserver Interfaces: IAuthenticator, IPlugin Entry point: standalone = certbot.plugins.standalone:Authenticator * webroot Description: Place files in webroot directory Interfaces: IAuthenticator, IPlugin Entry point: webroot = certbot.plugins.webroot:Authenticator - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Install route53 authenticator\nInstalling route53 authenticator plugin is yet again simpler on Debian:\nsudo apt install python3-certbot-dns-route53 1 2 3 4 5 6 7 8 9 10 11 12 13 ubuntu@ip-10-2-1-10:~$ certbot plugins - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - * dns-route53 Description: Obtain certificates using a DNS TXT record (if you are using AWS Route53 for DNS). Interfaces: IAuthenticator, IPlugin Entry point: dns-route53 = certbot_dns_route53.dns_route53:Authenticator * standalone Description: Spin up a temporary webserver Interfaces: IAuthenticator, IPlugin ... Step 2: Configure Route53 authenticator # Congrats! When you installed route53 authenticator, and with that AWS SDK for Python is also installed (officially known as boto) as dependency. These are precursor to some AWS IAM stuff we\u0026rsquo;re going to commence.\nAs a best practice, we are going to create a IAM user and only give required permission to the user to perform this operation.\nCreate User\n1 2 3 4 5 6 7 8 9 10 $ aws iam create-user --user-name certbot-route53 { \u0026#34;User\u0026#34;: { \u0026#34;Path\u0026#34;: \u0026#34;/\u0026#34;, \u0026#34;UserName\u0026#34;: \u0026#34;certbot-route53\u0026#34;, \u0026#34;UserId\u0026#34;: \u0026#34;AIDARIMALWMXWEXAMPLES\u0026#34;, \u0026#34;Arn\u0026#34;: \u0026#34;arn:aws:iam::XXXXXXXXXXXX:user/certbot-route53\u0026#34;, \u0026#34;CreateDate\u0026#34;: \u0026#34;2021-11-26T00:14:20+00:00\u0026#34; } } You need this user to have the following permission:\nroute53:ListHostedZones route53:GetChange route53:ChangeResourceRecordSets We\u0026rsquo;ll create policy with these permissions and attach that policy to the user.\nCreate Policy\nCreate a file called policy.txt and have these lines of JSON written to it.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 { \u0026#34;Version\u0026#34;: \u0026#34;2012-10-17\u0026#34;, \u0026#34;Id\u0026#34;: \u0026#34;certbot-dns-route53 sample policy\u0026#34;, \u0026#34;Statement\u0026#34;: [ { \u0026#34;Effect\u0026#34;: \u0026#34;Allow\u0026#34;, \u0026#34;Action\u0026#34;: [ \u0026#34;route53:ListHostedZones\u0026#34;, \u0026#34;route53:GetChange\u0026#34; ], \u0026#34;Resource\u0026#34;: [ \u0026#34;*\u0026#34; ] }, { \u0026#34;Effect\u0026#34;: \u0026#34;Allow\u0026#34;, \u0026#34;Action\u0026#34;: [ \u0026#34;route53:ChangeResourceRecordSets\u0026#34; ], \u0026#34;Resource\u0026#34;: [ \u0026#34;arn:aws:route53:::hostedzone/YOURHOSTEDZONEID\u0026#34; ] } ] } Be sure to replace YOURHOSTEDZONEID to your actual hosted zone. You can find this on https://console.aws.amazon.com/route53/v2/hostedzones in the last column for your hosted zone.\nHere I\u0026rsquo;m creating a policy with the name route53-santosh.pictures:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 $ aws iam create-policy --policy-name route53-santosh.pictures --policy-document file://policy.txt { \u0026#34;Policy\u0026#34;: { \u0026#34;PolicyName\u0026#34;: \u0026#34;route53-santosh.pictures\u0026#34;, \u0026#34;PolicyId\u0026#34;: \u0026#34;ANPARIMALWEXAMPLE4DPU\u0026#34;, \u0026#34;Arn\u0026#34;: \u0026#34;arn:aws:iam::XXXXXXXXXXXX:policy/route53-santosh.pictures\u0026#34;, \u0026#34;Path\u0026#34;: \u0026#34;/\u0026#34;, \u0026#34;DefaultVersionId\u0026#34;: \u0026#34;v1\u0026#34;, \u0026#34;AttachmentCount\u0026#34;: 0, \u0026#34;PermissionsBoundaryUsageCount\u0026#34;: 0, \u0026#34;IsAttachable\u0026#34;: true, \u0026#34;CreateDate\u0026#34;: \u0026#34;2021-11-26T00:41:24+00:00\u0026#34;, \u0026#34;UpdateDate\u0026#34;: \u0026#34;2021-11-26T00:41:24+00:00\u0026#34; } } Please make note of ARN in the response JSON. We\u0026rsquo;ll need that in the next step.\nAttach IAM Policy to IAM User\nWe have the policy and the user created. Now the next step is to attach policy to the user so that user have permission to do the Route53-ish stuff.\n1 $ aws iam attach-user-policy --policy-arn arn:aws:iam::XXXXXXXXXXXX:policy/route53-santosh.pictures --user-name certbot-route53 This command does not responds with anything, but you can check that policy is attached by invoking list-attached-user-policies and looking up the PolicyName in the output:\n1 2 3 4 5 6 7 8 9 $ aws iam list-attached-user-policies --user-name certbot-route53 { \u0026#34;AttachedPolicies\u0026#34;: [ { \u0026#34;PolicyName\u0026#34;: \u0026#34;route53-santosh.pictures\u0026#34;, \u0026#34;PolicyArn\u0026#34;: \u0026#34;arn:aws:iam::XXXXXXXXXXXX:policy/route53-santosh.pictures\u0026#34; } ] } With this done, we can proceed to the next step which is about creating access key and putting it in appropriate place for AWS SDK to function.\nCreate access key\nWe need access key and access key secret to programmatically talk with AWS. We can create access key with AWS CLI like so:\n1 2 3 4 5 6 7 8 9 10 11 $ aws iam create-access-key --user-name certbot-route53 $ aws iam create-access-key --user-name certbot-route53 { \u0026#34;AccessKey\u0026#34;: { \u0026#34;UserName\u0026#34;: \u0026#34;certbot-route53\u0026#34;, \u0026#34;AccessKeyId\u0026#34;: \u0026#34;AKIARIMALWMEXAMPLE73\u0026#34;, \u0026#34;Status\u0026#34;: \u0026#34;Active\u0026#34;, \u0026#34;SecretAccessKey\u0026#34;: \u0026#34;VKD94MARJeztTFJlCWK0F/E6vTaEiPEXAMPLEKEY\u0026#34;, \u0026#34;CreateDate\u0026#34;: \u0026#34;2021-11-26T03:28:29+00:00\u0026#34; } } Note down AccessKeyId and SecretAccessKey.\nConfigure AWS SDK\nDid I tell you that AWS SDK for Python is downloaded as part of downloading Route53 authenticator plugin? For this SDK to work properly, we need to setup the AccessKeyId and SecretAccessKey retrieved from previous section.\nThere are quite a few ways we can configure the keys, but I like setting up ~/.aws/config. Here is my config file:\n1 2 3 [default] aws_access_key_id=AKIARIMALWMEXAMPLE73 aws_secret_access_key=VKD94MARJeztTFJlCWK0F/E6vTaEiPEXAMPLEKEY Note: In next section, we are going to run certbot as sudo, please put the config file in $HOME of root user.\nStep 3: Get certificate for your domain \u0026amp; subdomains from Let\u0026rsquo;s Encrypt # That was a long marathon for configuring the Route53 authenticator plugin. Next we get the certificate. Please note that I\u0026rsquo;m only generating certificate in this step and will configure nginx separately.\nGet the certificate\nSwitch to root user and make sure AWS credentials exists by running ls ~/.aws.\n1 2 3 ubuntu@ip-10-2-1-10:~$ sudo -i root@ip-10-2-1-10:~# ls ~/.aws config If the output of ls ~/.aws is ls: cannot access '/root/.aws': No such file or directory, please check the end of the last section.\nNow let\u0026rsquo;s proceed with certbot as we are already in interactive shell as root. Following the the one liner I\u0026rsquo;m gonna use.\n1 # certbot certonly --dns-route53 --email \u0026#39;you@example.com\u0026#39; --domain \u0026#39;santosh.pictures\u0026#39; --domain \u0026#39;*.santosh.pictures\u0026#39; --agree-tos --non-interactive Replace you@example.com with your actual email. This email is used to send notifications when expiration date of certs is close. The above command can also be reduced to following:\n1 # certbot certonly --dns-route53 -m \u0026#39;you@example.com\u0026#39; -d \u0026#39;santosh.pictures\u0026#39; -d \u0026#39;*.santosh.pictures\u0026#39; --agree-tos -n The output of above command looks something like this:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 Saving debug log to /var/log/letsencrypt/letsencrypt.log Credentials found in config file: ~/.aws/config Plugins selected: Authenticator dns-route53, Installer None Obtaining a new certificate Performing the following challenges: dns-01 challenge for santosh.pictures dns-01 challenge for santosh.pictures Waiting for verification... Cleaning up challenges IMPORTANT NOTES: - Congratulations! Your certificate and chain have been saved at: /etc/letsencrypt/live/santosh.pictures/fullchain.pem Your key file has been saved at: /etc/letsencrypt/live/santosh.pictures/privkey.pem Your cert will expire on 2022-02-24. To obtain a new or tweaked version of this certificate in the future, simply run certbot again. To non-interactively renew *all* of your certificates, run \u0026#34;certbot renew\u0026#34; - If you like Certbot, please consider supporting our work by: Donating to ISRG / Let\u0026#39;s Encrypt: https://letsencrypt.org/donate Donating to EFF: https://eff.org/donate-le Congrats! You have generated TLS certificates to be used with a web server (/etc/letsencrypt/live/santosh.pictures/fullchain.pem). Along with certs, we also have the private key (/etc/letsencrypt/live/santosh.pictures/privkey.pem).\nYou can find more configuration options to use certbot on it\u0026rsquo;s documentation page: https://eff-certbot.readthedocs.io/en/stable/using.html\nStep 4: Configure nginx with HTTPS # We need to tweak our nginx.conf a little bit. Before that, this is the version of nginx.conf with no https configured. This is also available at https://github.com/santosh/ansible/blob/v0.1.0/roles/nginx/files/nginx.conf:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 user nginx; worker_processes auto; error_log /var/log/nginx/error.log; pid /run/nginx.pid; # Load dynamic modules. See /usr/share/nginx/README.dynamic. include /usr/share/nginx/modules/*.conf; events { worker_connections 1024; } http { log_format main \u0026#39;$remote_addr - $remote_user [$time_local] \u0026#34;$request\u0026#34; \u0026#39; \u0026#39;$status $body_bytes_sent \u0026#34;$http_referer\u0026#34; \u0026#39; \u0026#39;\u0026#34;$http_user_agent\u0026#34; \u0026#34;$http_x_forwarded_for\u0026#34;\u0026#39;; access_log /var/log/nginx/access.log main; sendfile on; tcp_nopush on; tcp_nodelay on; keepalive_timeout 65; types_hash_max_size 2048; include /etc/nginx/mime.types; default_type application/octet-stream; # Load modular configuration files from the /etc/nginx/conf.d directory. # See http://nginx.org/en/docs/ngx_core_module.html#include # for more information. include /etc/nginx/conf.d/*.conf; server { listen 80 default_server; listen [::]:80 default_server; server_name _; root /usr/share/nginx/html; # Load configuration files for the default server block. include /etc/nginx/default.d/*.conf; location / { proxy_pass http://localhost:8080/; } error_page 404 /404.html; location = /40x.html { } error_page 500 502 503 504 /50x.html; location = /50x.html { } } } Now instead of the whole content + the https config. Here I present you the diff:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 diff --git a/roles/nginx/files/nginx.conf b/roles/nginx/files/nginx.conf index f1dd861..c63c37c 100644 --- a/roles/nginx/files/nginx.conf +++ b/roles/nginx/files/nginx.conf @@ -40,6 +40,17 @@ http { # Load configuration files for the default server block. include /etc/nginx/default.d/*.conf; + listen 443 ssl; + + ssl_certificate /etc/letsencrypt/live/santosh.pictures/fullchain.pem; + ssl_certificate_key /etc/letsencrypt/live/santosh.pictures/privkey.pem; + + # redirect non-https traffic to https + if ($scheme != \u0026#34;https\u0026#34;) { + return 301 https://$host$request_uri; + } + location / { proxy_pass http://localhost:8080/; } Explanation:\nLine 9 says along with listening on port 80, also listen on port 443. Line 11,12 specifies the path to certificate and private key we fetched in last section. Line 15-17 issue a 301 response to any client visiting http version and then redirect to https version. With these changes in place, and after reloading the nginx service. I can see the https enabled on my site.\nHTTPS version of ci.santosh.pictures Automating HTTP to HTTPS transition with Ansible # When I started writing this section, I started with a dilemma. I have a nginx role, and then a jenkins role, then I have a dream of enabling HTTPS. In which role does this automation of this https goes? Or do I create some other Role for this?\nAnd after giving a lot of thought I came to the conclusion that it is not the right time to write about this automation. And I\u0026rsquo;ll cover this automation when I learn more about Ansible. Topics like variables, ansible-vault are important to secure this repository I\u0026rsquo;m working with, as it is publicly exposed. Variables are important for dynamic behaviour which this repo needs.\nAnd just like I kept Ansible Roles for this post in the previous post, I\u0026rsquo;m keeping variables and ansible-vault for the next post where I\u0026rsquo;ll expand this topic.\nUpdate: The next post is now out: https://santoshk.dev/posts/2022/automate-https-certificates-with-ansible-roles/\nConclusion # I have reached milestone in learning Ansible. I started feel need for Ansible when I wanted to configure my own Jenkins server. When I configured my Jenkins server I also realized that without TLS it\u0026rsquo;s unsafe to do Jenkins operations. This also attracted me to learn more about cyber security in general.\nI can proceed with my Jenkins work now. Along with that I\u0026rsquo;ll keep exploring Ansible and cyber security.\nThere are a lot more to cover in nginx and jenkins configuration. I\u0026rsquo;ll for sure cover them in some other posts. Don\u0026rsquo;t forget to subscribe to the newsletter so that you are updated when new post comes out.\n","date":"27 November 2021","externalUrl":null,"permalink":"/posts/2021/wildcard-domain-certificate-using-aws-route53-and-lets-encrypt/","section":"Posts","summary":"Introduction # I have already posted about how we can automate installation of Jenkins \u0026 Nginx with Ansible. I have also done a post where I talk about how to enable HTTPS on a non-wildcard basis i.e. only for the root domain and not on subdomain.\nToday I’ll go through how go get and configure a HTTPS certificate from Let’s Encrypt for all the subdomain. I’ll automate all these using Ansible.\n","title":"Wildcard Domain Certificate Using Route53 and Let's Encrypt","type":"posts"},{"content":" Introduction # I have been tinkering with this new framework and have a little bit of experience with FastAPI as of now. What I\u0026rsquo;ve though of is to do a post about authentication and authorization. I\u0026rsquo;m sure going to do a registration and login system and nothing fancy this time. I am going to use JWT as authorization mechanism. This is mostly a walkthrough of new features in Python.\nI will not go in-depth in any of the topics, but will provide you with links which you can dig up to expand your knowledge.\nSeries Index # Project Setup and FastAPI introduction (you are here) Database Setup and User Registration Fix the failing test with mocking and dependency injection Authentication Premier and Login Endpoint Prerequisite # This post is fairly easy and even beginners can follow it. Following are some packages which I will be using in this post.\nPython as a language pytest as a test runner FastAPI as application layer PostgreSQL as database layer psycopg2, the database connector SQLAlchemy for ORM Docker for PostgreSQL Make sure Python and Docker are installed on your system. You should be able to do a docker run hello-world and python -V without error. Please avoid installing Python 2. Leave a comment if you are stuck, then I can do something.\nIf you are following this tutorial on Windows, I expect you to be working in WSL. On Mac, there should be no issue. For every other installation, follow along with me. Feel free to connect with me on LinkedIn or drop a \u0026ldquo;Hi\u0026rdquo; on Twitter.\nSetting up the project # Setup a virtual environment # Issue this command to your terminal.\n$ python3 -m venv .venv You should get back the prompt after a few seconds, depending on how fast your system is.\nYou should be able to see a .venv directory like shown below.\n1 2 3 4 5 $ ls -la total 12 drwxrwxr-x 3 ec2-user ec2-user 6144 Nov 11 13:12 ./ drwxrwxr-x 21 ec2-user ec2-user 6144 Nov 11 12:56 ../ drwxrwxr-x 5 ec2-user ec2-user 6144 Nov 11 13:12 .venv/ To get into the virtual environment, do:\n$ source .venv/bin/activate The prompt will be prepended with (.venv). In my case, my customized full prompt looks like this:\n(.venv) ec2-user at ip-10-2-1-250 in ~/workspace/fastauth $ As you can see, I have created a directory called fastauth, which is root of this project. Please also take note that from now on, whenever I run any command, I\u0026rsquo;m inside this virtual environment. So you should be too.\nHello World in FastAPI # Let\u0026rsquo;s take a TDD approach and write our test first. The requirements of our first endpoint are:\nRequest to /ping should respond with a HTTP 200 status. Request to /ping should respond with a custom JSON: {\u0026quot;msg\u0026quot;: \u0026quot;pong\u0026quot;}. Let\u0026rsquo;s write our first test:\n1 2 3 4 5 6 7 8 9 10 from fastapi.testclient import TestClient from main import app client = TestClient(app) def test_ping(): response = client.get(\u0026#34;/ping\u0026#34;) assert response.status_code == 200 assert response.json() == {\u0026#34;msg\u0026#34;: \u0026#34;pong\u0026#34;} Save above to a file called test_main.py.\nRun the test # To run the test, you need to have pytest installed.\npip install pytest Now run the test..\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 $ pytest =========================== test session starts =========================== platform linux -- Python 3.7.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 rootdir: /efs/repos/fastauth collected 0 items / 1 error ================================= ERRORS ================================== ______________________ ERROR collecting test_main.py ______________________ ImportError while importing test module \u0026#39;/efs/repos/fastauth/test_main.py\u0026#39;. Hint: make sure your test modules/packages have valid Python names. Traceback: /usr/lib64/python3.7/importlib/__init__.py:127: in import_module return _bootstrap._gcd_import(name[level:], package, level) test_main.py:1: in \u0026lt;module\u0026gt; from fastapi.testclient import TestClient E ModuleNotFoundError: No module named \u0026#39;fastapi\u0026#39; ========================= short test summary info ========================= ERROR test_main.py !!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!! ============================ 1 error in 0.17s ============================= The first error we would receive is this ModuleNotFoundError with a message No module named 'fastapi'. That\u0026rsquo;s because we really don\u0026rsquo;t have anything in our virtual environment other than pytest.\nWe are doing good with our TDD approach. We are following the red, green, refactor approach. In this approach we first write test to fail the test. Then we write the minimal code to pass the test (green resembles passed test). In third step we refactor redundant code.\nMoving forward, let\u0026rsquo;s get rid of this ImportError by installing fastapi in our virtual environment and run the test again.\n$ pip install fastapi [..output trimmed..] $ pytest This time I\u0026rsquo;m not going to show you the entire output, but the final line says:\nimport requests ModuleNotFoundError: No module named 'requests' That a red again.\nLet\u0026rsquo;s install the required package and try again.\n$ pip install requests [..output trimmed..] $ pytest The error this time is different:\nfrom main import app ModuleNotFoundError: No module named 'main' That\u0026rsquo;s because we haven\u0026rsquo;t created the main.py file yet. Let\u0026rsquo;s go ahead and create one and run the test again:\n$ touch main.py $ pytest from main import app ImportError: cannot import name 'app' from 'main' (.../fastauth/main.py) Minimal code to pass the test # We are yet again going to write minimal code to pass this test. Open up a file called main.py and write this:\n1 2 3 from fastapi import FastAPI app = FastAPI() Let\u0026rsquo;s run the test this time:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 $ pytest -q F [100%] ============================ FAILURES ============================= ____________________________ test_ping ____________________________ def test_ping(): response = client.get(\u0026#34;/ping\u0026#34;) \u0026gt; assert response.status_code == 200 E assert 404 == 200 E + where 404 = \u0026lt;Response [404]\u0026gt;.status_code test_main.py:9: AssertionError ===================== short test summary info ===================== FAILED test_main.py::test_ping - assert 404 == 200 1 failed in 0.78s Now this time we get some real error. We are getting a HTTP 404 error, that\u0026rsquo;s because that endpoint is not defined yet. That makes me really happy. I got some real error.\nI will go ahead and add route and route handler:\n1 2 3 4 5 6 7 from fastapi import FastAPI app = FastAPI() @app.get(\u0026#34;/ping\u0026#34;) async def ping(): return \u0026#39;Hello World!\u0026#39; Some explanation of the code now:\nWhat are can do with app is, we can wrap a handler function with it. The handler function above simply return the string 'Hello World'. Let\u0026rsquo;s look at the @app.get(\u0026quot;/ping\u0026quot;) part now. /ping is the route here. get is the GET operation on the endpoint. How about running the test again?\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 $ pytest -q F [100%] ============================================= FAILURES ============================================= ____________________________________________ test_ping _____________________________________________ def test_ping(): response = client.get(\u0026#34;/ping\u0026#34;) assert response.status_code == 200 \u0026gt; assert response.json() == {\u0026#34;msg\u0026#34;: \u0026#34;pong\u0026#34;} E AssertionError: assert \u0026#39;Hello World!\u0026#39; == {\u0026#39;msg\u0026#39;: \u0026#39;pong\u0026#39;} E + where \u0026#39;Hello World!\u0026#39; = \u0026lt;bound method Response.json of \u0026lt;Response [200]\u0026gt;\u0026gt;() E + where \u0026lt;bound method Response.json of \u0026lt;Response [200]\u0026gt;\u0026gt; = \u0026lt;Response [200]\u0026gt;.json test_main.py:10: AssertionError ===================================== short test summary info ====================================== FAILED test_main.py::test_ping - AssertionError: assert \u0026#39;Hello World!\u0026#39; == {\u0026#39;msg\u0026#39;: \u0026#39;pong\u0026#39;} Yeah! Seems like the first assertion passed. What it did not pass is the response message.\nLet\u0026rsquo;s correct that.\n5 6 7 @app.get(\u0026#34;/ping\u0026#34;) async def ping(): return \u0026#39;Hello World!\u0026#39; And the output:\n1 2 3 $ pytest -q . [100%] 1 passed in 0.53s Related reading: https://fastapi.tiangolo.com/tutorial/testing/\nFastAPI has support for OpenAPI, builtin. # One good thing among many about FastAPI is that, you don\u0026rsquo;t have to install Swagger separately. You can import and export the specs without much hassle. Let\u0026rsquo;s see what FastAPI has to offer. But before that, let\u0026rsquo;s start our development server.\nWe were, till now, not running the actual server. But were testing the routes. To run the actual server, we need a companion package of FastAPI called uvicorn.\npip install uvicorn and then run this command:\n$ uvicorn main:app --reload Let\u0026rsquo;s see what it does.\nuvicorn is the command to start the ASGI server interface. What we are passing to uvicorn is main:app which is the module:app (app here is the FastAPI instance). Basically what --reload does is, it reloads the new content from disk when file is updated on disk. Below is the mine output.\n1 2 3 4 5 6 INFO: Will watch for changes in these directories: [\u0026#39;/home/ec2-user/workspace/fastauth\u0026#39;] INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit) INFO: Started reloader process [20670] using statreload INFO: Started server process [20672] INFO: Waiting for application startup. INFO: Application startup complete. As you can see the output from the console, our server started on http://127.0.0.1:8000. So we should be able to see our ping response on http://127.0.0.1:8000/ping\nping response from the server What fun thing you could do with FastAPI is, nagivate to the SwaggerUI at http://127.0.0.1:8000/docs.\nSwagger UI at /docs And play around:\nPlay around with request and response You may want to download the OpenAPI spec from http://127.0.0.1:8000/openapi.json if you wish. This file will get populated as we follow this tutorial/series.\nIn next section in this series, I\u0026rsquo;ll show you how to setup database and proceed with user registration. To keep getting updates on my blog posts, please subscribe to my newsletter below.\n","date":"18 November 2021","externalUrl":null,"permalink":"/posts/2021/tdd-approach-to-creating-an-authentication-system-with-fastapi-part-1/","section":"Posts","summary":"Introduction # I have been tinkering with this new framework and have a little bit of experience with FastAPI as of now. What I’ve though of is to do a post about authentication and authorization. I’m sure going to do a registration and login system and nothing fancy this time. I am going to use JWT as authorization mechanism. This is mostly a walkthrough of new features in Python.\n","title":"A TDD Approach to Creating an Authentication System with FastAPI, Part 1","type":"posts"},{"content":"","date":"18 November 2021","externalUrl":null,"permalink":"/categories/auth/","section":"Categories","summary":"","title":"Auth","type":"categories"},{"content":" Introduction # In teenage days, I got misled by some movie that we should live life free style. Perhaps that was not the main message of the movie, but I started chilling in life. But as everybody knows, life has to change its direction\u0026hellip;\nImportance of following a Routine # But now in life I have started to encounter that routine is important. It helps keep things easy and in manner.\nWe don\u0026rsquo;t have to memorize certain things in life. Because we are already following a pattern. You are committed to a certain task which makes everything easy. If you don\u0026rsquo;t follow a routine, you are probably aimless in life. That\u0026rsquo;s the harsh truth. Please consider setting a long-term goal and work for it. And if you are aimlessly doing stuff, you probably won\u0026rsquo;t have a feeling of accomplishment. And if you don\u0026rsquo;t have a feeling of accomplishment, you\u0026rsquo;re no longer motivated to do things. Even if you somehow manage to do things, you\u0026rsquo;ll feel exhausted and mentally burned out very.\nPlan things and finish them # Plan things # You can consider writing down your pending works or things are suppose to do which will make your goal to https://trello.com.\nWhat I personally have done is, created lists in boards with names such as:\n2021 Q4 Monthly Goal Weekly Goal Done Trello board You can drag and drop your card/task (unit of work) among lists.\nWith that in place, what additional thing you can do is, restrict number of cards in those lists. E.g. the upper limit in my Weekly Goal is to 5 cards at most. This helps me focus on lesser things and keep things doing fast.\nOther features of Trello includes:\nSetting Checklist in the task Setting Deadline to the task Write comments Color code tasks and add labels Ability to collaborate with other Trello users (paid) Schedule things and execute them # Flowtime is a time management technique which I follow then I\u0026rsquo;m doing something serious. It is more forgiving than Pomodoro in term of time bounding.\ncolumns of flowtime Unlike Pomodoro, Flowtime affects your psycology. You are not forced to work in a bound time. You can take as many breaks you want. And of whatever length you want. But before taking a break, you have to specify how longer your break will be.\nStart only things which you can finish # This will ease you in being committed. And will gently push you to not pick any random stuff.\nIf anything new comes to your mind, give it a thought, wait till next morning. Does it still resides in your mind? If yes, then proceed commiting time to it, otherwise not.\nThis is important to do because sometimes in life many thoughts come to your mind. Not all of them are important. I have experienced this personally, I have started a bunch of courses on Coursera in excitement and have never completed any of them.\nHow do I maintain a routine # I belong to Google eco-system so I use Google Calander to create reminders for a certain time daily, and weekly for weekly tasks. You can use other services that\u0026rsquo;s up to you. Alternatives could be Microsoft Calander, Apple Calendar, or other third party services.\nIn Google Calandar you can drag and drop remainders here and there; that\u0026rsquo;s kinda fun. I get a reminder on my phone. And I can always use the Google Calander web UI to check upcoming tasks.\nGoogle Calander in action This obviously does not forces me to do the work, but rather act as a guideline. It okay to feel bad if you are not on a routine because that is what routine is for.\nSubscribe only to the Stuff you Consume # If you are not able to fit that newsletter you subscribed to in your routine, you probably don\u0026rsquo;t want that thing in your life.\nThis also applies to user groups or forums of which you are member of.\nUnsubscribe to unnecessary emails I keep my mail inbox as clean as possible. I instantly unsubscribe to newsletter I don\u0026rsquo;t want in life. They just create a distraction and waste time.\nConclusion # These are the things I\u0026rsquo;m practicing in my daily life. How do you feel about it? Do you do things differently? Don\u0026rsquo;t hesitate to comment or reach me on Twitter.\n","date":"18 November 2021","externalUrl":null,"permalink":"/posts/2021/general-lifestyle-tips-and-tricks-for-programmers/","section":"Posts","summary":"Introduction # In teenage days, I got misled by some movie that we should live life free style. Perhaps that was not the main message of the movie, but I started chilling in life. But as everybody knows, life has to change its direction…\nImportance of following a Routine # But now in life I have started to encounter that routine is important. It helps keep things easy and in manner.\n","title":"General Lifestyle Tips and Tricks for Programmers","type":"posts"},{"content":"","date":"18 November 2021","externalUrl":null,"permalink":"/tags/lifestyle/","section":"Tags","summary":"","title":"Lifestyle","type":"tags"},{"content":" Introduction # In this post, we are going to see how we can use existing AWS infrastructure to create thumbnail for uploaded images. In this venture, we are going to use S3 trigger to execure Lambda function to do whatever we want. But in this case, we\u0026rsquo;ll create thumbnail so that we can save on some bandwidth when only a smaller version of image is required on the frontend.\nSo let\u0026rsquo;s see the pre-requisites:\nSome familarity with S3 Some familarity with Lambda Some familarity with AWS CLI Although the tutorial can be followed through AWS Console, I\u0026rsquo;ve decided to explore AWS CLI as it can do things quicker in some cases.\nThis post is crudely divided into 2 sections:\nTest the trigger (you are here) Make the thumbnail While you read this post, take a moment to connect with me on LinkedIn.\nOverview # Talking about the overview of this section, in this section, we\u0026rsquo;ll check if trigger is getting invoked properly when a file is uploaded to a bucket. We\u0026rsquo;ll do the initial setup to achieve all these.\nLet\u0026rsquo;s move to the next subsection.\nCreate a source bucket # This is the only bucket we\u0026rsquo;ll create in Test the trigger section. Later on in Make the thumbnail section, we will create another bucket.\n1 2 3 4 $ aws s3api create-bucket --bucket source-bucket-sntshk --region ap-south-1 --create-bucket-configuration LocationConstraint=ap-south-1 { \u0026#34;Location\u0026#34;: \u0026#34;http://source-bucket-sntshk.s3.amazonaws.com/\u0026#34; } Let\u0026rsquo;s go to the above command\u0026rsquo;s flags one by one:\naws is the command line interface to the Amazon Web Services API. Each service* in the AWS toolbelt is available as a subcommand to this aws CLI. s3api is the command which is used to create and delete bucket among many others. create-bucket is the sub-command/action to s3api. Rest of the flags provide option to this create-bucket. --region ap-south-1 provides region to the bucket. Note: Please take note of this region. We\u0026rsquo;ll have to create Lambda function in the same region for this experiment to work. --create-bucket-configuration LocationConstraint=ap-south-1 is just another required flag in my case. You may not have to pass this flag if you are creating bucket in us-east-1 region. Once the command has been done executing, it will spit out some output. Ignore this for now.\nUpload a test object to bucket # After we are done with creating the bucket, let\u0026rsquo;s put a test object in the bucket before proceeding to the Lambda trigger part.\n1 2 3 4 5 $ aws s3api put-object --bucket source-bucket-sntshk --key TestImage.jpg --body TestImage.jpg { \u0026#34;ETag\u0026#34;: \u0026#34;\\\u0026#34;7ad9294573ba03dd91442e0c4aba4ad7\\\u0026#34;\u0026#34; } Explanation of the command:\nput-object is used to upload individual object to given bucket. --bucket source-bucket-sntshk provides the name of the bucket. --key TestImage.jpg is the prospective name of the file on the S3 side. --body TestImage.jpg is the name of the file on our local system. On successful upload. You will see an \u0026ldquo;ETag\u0026rdquo; output as shown above.\nCreate the Lambda function # While creating a Lambda function is fairly simple, it\u0026rsquo;s pre-requsites are not. That is because we need to give our function some role so that it is able to communicate to AWS services. This is the case where I refrain to use AWS CLI. Things are quicker with console in this case.\nStep 1: Go to Lambda console page and push the button called Create function.\nStep 2: Choose the option Use a blueprint. This is the reason we are not using CLI, as CLI does not gives a way to use blueprints. With this radio button checked, search for s3 and choose the python option (you can choose Node.js if you like, all the best). Now click Configure.\nStep 3: Type in a function name in the Function name section. And in Execution role section, choose Create a new role with basic Lambda permission. You may choose Use an existing role if you are following this tutorial for the second time.\nBasically what role allows our lambda function to do is to give it permission to do certain things on other AWS services. We can have different policies attached inside our roles. The role used in our context does these two things:\nOne is to check which new files has been uploaded to the S3 bucket. Another is for analytical purpose and to upload logs to CloudWatch. Basic information for Lambda Step 4: In next section, we see settings related to S3 triggers. Choose the bucket we created in earlier section. For event type, I am selecting All object create events. For Prefix and Suffix option, we have left them blank. But they are pretty self-explanatory and you can enable them if you like.\nS3 trigger options for lambda Before pushing the button Create function, you can review the sample code that is provided by the blueprint.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 import json import urllib.parse import boto3 print(\u0026#39;Loading function\u0026#39;) s3 = boto3.client(\u0026#39;s3\u0026#39;) def lambda_handler(event, context): #print(\u0026#34;Received event: \u0026#34; + json.dumps(event, indent=2)) # Get the object from the event and show its content type bucket = event[\u0026#39;Records\u0026#39;][0][\u0026#39;s3\u0026#39;][\u0026#39;bucket\u0026#39;][\u0026#39;name\u0026#39;] key = urllib.parse.unquote_plus(event[\u0026#39;Records\u0026#39;][0][\u0026#39;s3\u0026#39;][\u0026#39;object\u0026#39;][\u0026#39;key\u0026#39;], encoding=\u0026#39;utf-8\u0026#39;) try: response = s3.get_object(Bucket=bucket, Key=key) print(\u0026#34;CONTENT TYPE: \u0026#34; + response[\u0026#39;ContentType\u0026#39;]) return response[\u0026#39;ContentType\u0026#39;] except Exception as e: print(e) print(\u0026#39;Error getting object {} from bucket {}. Make sure they exist and your bucket is in the same region as this function.\u0026#39;.format(key, bucket)) raise e Step 5: Press the Create function button.\nTest the function manually # Time to test if our function is really working before we proceed to the next step. Follow the steps below:\nNote: The GUI might change in future. But I suppose there will always an option to \u0026ldquo;Test\u0026rdquo; as long as there is an option to view code in the console.\nNavigate to your Lambda function. Scroll down until you see tabs like \u0026ldquo;Code\u0026rdquo;, \u0026ldquo;Test\u0026rdquo;, \u0026ldquo;Monitor\u0026rdquo;, etc. You should be by default at \u0026ldquo;Code\u0026rdquo; tab. That\u0026rsquo;s the tab we need to be in. You should see an orange button named \u0026ldquo;Test\u0026rdquo; with a caret symbol (downward facing arrow head). On clicking it, you should see options called Configure test event, click that. You will be presented with a dialog. Select Create test event, then in Event template, select s3-put, and then give this test a name. Configure test event for Lambda Now in the event JSON, replace the occurance of example-bucket with the name of bucket with created in Create a source bucket section. Replace the occurance of test%2Fkey with the test object we uploaded in Upload a test object to bucket section. Then push the \u0026ldquo;Create\u0026rdquo; orange button. Here is a little gif to show step 2-4.\nConfigure test event When tested, you should see Status: Succeeded on the upper right of the execution results.\nTest the function by uploading something # Please note that if you are getting AccessDenied error in previous section. You should see next section.\nNow that our test have started to pass, we should now proceed to test if it invokes if new object is uploaded to the bucket. Let\u0026rsquo;s do that.\n1 2 3 4 $ aws s3api put-object --bucket source-bucket-sntshk --key TestImage.png --body .\\Screenshot_20201108_004543.png { \u0026#34;ETag\u0026#34;: \u0026#34;\\\u0026#34;a71acffffa4934a9a0a556365a54b2a7\\\u0026#34;\u0026#34; } This command adds new object to the bucket.\nYou can see function invocation in the \u0026ldquo;Monitor\u0026rdquo; tab of our Lambda function. If you don\u0026rsquo;t see it, try adjusting the time window. Something like last 15 minutes or so.\nYou can also verify if the function executed by going to the pressing View logs in CloudWatch button and checking out the latest log to see if it throws an error or executes successfully.\nExtra: What the AccessDenied # Question: Hi Santosh, I don\u0026rsquo;t see a success message, instead I see a AccessDenied message in the Execution results. Why?\nAnswer: Make sure the role which Lambda create has two policies attached to it. One for writing log to CloudWatch and another for reading from S3 bucket. As part of the exercise, I\u0026rsquo;m going to deliberately delete S3 policy and attach in the gif below.\nAttach role with S3 GetObject permission Conclusion # In this post, we just checked how we can execute a piece of code when something happens on S3. There are a plathora of services which can be used with Lambda.\nIn next post, we will see how we can create a thumbnail based on the activity we did in this post.\nIf you liked this post, please share it with your network. Subscribe to the newsletter below for similar news.\n","date":"14 November 2021","externalUrl":null,"permalink":"/posts/2021/create-thumbnail-worker-with-s3-and-lambda-test-the-trigger/","section":"Posts","summary":"Introduction # In this post, we are going to see how we can use existing AWS infrastructure to create thumbnail for uploaded images. In this venture, we are going to use S3 trigger to execure Lambda function to do whatever we want. But in this case, we’ll create thumbnail so that we can save on some bandwidth when only a smaller version of image is required on the frontend.\n","title":"Create Thumbnail Worker With S3 and Lambda: Test the Trigger","type":"posts"},{"content":"","date":"24 October 2021","externalUrl":null,"permalink":"/tags/automation/","section":"Tags","summary":"","title":"Automation","type":"tags"},{"content":" Introduction # I don\u0026rsquo;t claim to be a DevOps expert. I just know some basic ssh and scp, that\u0026rsquo;s all. I have done whatever server configuration till date by ssh\u0026rsquo;ing into instances and then configuring them by hand. But now-a-days I feel like spinning up and destroying a server has increased in my life style. And doing it manually is just waste of time when we already have solution like Ansible.\nWith that said, I\u0026rsquo;ll start this post by a gyan which one of my friend recently gave me. This gyan turned out to be a business management thing, but it can be applied in real life as well. If you really want something to succeed, ask yourself these questions.\nWhat Why How Where Who When These questions will solidify your thought process. If you can answer these questions, you\u0026rsquo;ll be knowing what you\u0026rsquo;re doing, what you end goal is, you\u0026rsquo;ll come up with a roadmap, and you\u0026rsquo;ll be more commited to it.\nWhat: Learn Ansible.\nWhy: To save time by automating my server and package installations. Some of things to automate? nginx, jenkins, openvpn.\nHow: Youtube, Ansible Docs, The Internet.\nWhere: AWS Cloud\nWho: Santosh Kumar\nWhen: Before 2022 starts, approx 60 days in hand at the time of writing this.\nConnect with me on LinkedIn. And now is the time to get our feet wet\u0026hellip;\nGetting Started # If you are reading this to learn Ansible with me. I must tell you that I expect some knowledge beforehand from you. Below are the prerequisites.\nWorking knowledge of SSH. If you are reading this post, you must already have that. If not, try spinning up a EC2 instance and connect to it first. Working knowledge of Linux. I\u0026rsquo;m doing Ansible with Amazon EC2. Ansible works with Windows too, but I\u0026rsquo;m leaving it to some of you guys to take the liberty of journaling your journey with Windows. Working knowledge of AWS infrastructure. Are bare minimum, you should be able to spin up instances and able to connect to them (and each other) via SSH. Better if you know what a VPC is, what a subnet is, what roles are, what policies are, what permissions are, what security group is etc. If these words are new to you, please consider skimming through AWS Certified Cloud Practitioner video.\nFor those who are ready, let\u0026rsquo;s start with some theory class first\u0026hellip;\nWhat is an inventory? # From dictionary:\ninventory\nnoun\nThe stock of an item on hand at a particular location or business. a detailed list of all the items on hand In context of Ansible, an inventory is a list of all the hosts. A host can be a server, or network device or whatever we intend to automate with Ansible.\nWe store IP addresses or hostnames in the inventory file, or group of hostnames/IPs. More on this in later sections. On top of that, we can also store variables, and sources which are static or dynamic in nature.\nThe default location for inventory in Linux environment is /etc/ansible/hosts.\nWhat is a Module? # Modules are executable bits of code. Modules are like tools. From the knowledge of Ansible I have right now, each module represents a command which can be executed on the remote host.\nExample modules:\nping command yum service Some of the builtin modules can be found at: https://docs.ansible.com/ansible/latest/collections/ansible/builtin/index.html\nWhat is a Playbook? # Playbooks are YAML syntaxed files which store tasks which needs to be performed. In context of a playbook, task is modules + some extra info (more on this when we experiment).\nThere is a special syntax to playbooks which we will see when doing an example. In other words, playbooks are ansible\u0026rsquo;s configuration, deployment and orchestration language.\nPlaybook invoke Ansible modules. Playbooks are instruction manual on how to use modules. Playbooks are run against the inventory, which are like raw material.\nWhat is a Role? # Roles are playbooks and related material in a named directory structures.\nI have wrote about What are roles in next iteration of this series. But if you are new to Ansible I don\u0026rsquo;t see any reason for you to worry about Ansible.\nHello World with Ansible # For sake of this tutorial, I have made this setup on AWS.\nI have 2 hosts in same subnet: My control node is 10.10.10.10. My one of the managed node is 10.10.10.11. For sake of this tutorial, I\u0026rsquo;m using the same private key to connect to both the instances, which indeed is not the best practice. But for now, let it be. I have installed ansible on my control node with sudo amazon-linux-extras install ansible2, but the method can be different among distros.\nAlthough I can, I\u0026rsquo;ll not touch the system config/inventory files and instead take a copy of them and modify them in my workspace.\nI do so like this:\n1 2 3 4 5 6 7 $ mkdir ~/workspace/ansible $ cd ~/workspace/ansible $ $ cat /etc/ansible/hosts \u0026gt; inventory $ cat /etc/ansible/ansible.cfg \u0026gt; ansible.cfg $ ls key.pem inventory ansible.cfg What\u0026rsquo;s going on above? I have create a directory in my $HOME named workspace/ansible and cd into it. Next I have taken a copy of /etc/ansible/hosts file which happens to have no host entry and write it to my workspace/ansible dir. I have done same thing with ansible.cfg file. Then I ls to show what I have in that directory.\nFor sake of simplicity, as you can see, I have kept the private key (key.pem, which I got while spinning up the instance) in same directory as all the files.\nSetting up the inventory # Let\u0026rsquo;s first go through the inventory file:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 # This is the default ansible \u0026#39;hosts\u0026#39; file. # # It should live in /etc/ansible/hosts # # - Comments begin with the \u0026#39;#\u0026#39; character # - Blank lines are ignored # - Groups of hosts are delimited by [header] elements # - You can enter hostnames or ip addresses # - A hostname/ip can be a member of multiple groups # Ex 1: Ungrouped hosts, specify before any group headers. ## green.example.com ## blue.example.com ## 192.168.100.1 ## 192.168.100.10 # Ex 2: A collection of hosts belonging to the \u0026#39;webservers\u0026#39; group ## [webservers] ## alpha.example.org ## beta.example.org ## 192.168.1.100 ## 192.168.1.110 [utility] 10.10.10.11 [nonexistent] 10.10.10.12 # If you have multiple hosts following a pattern you can specify # them like this: ## www[001:006].example.com # Ex 3: A collection of database servers in the \u0026#39;dbservers\u0026#39; group ## [dbservers] ## ## db01.intranet.mydomain.net ## db02.intranet.mydomain.net ## 10.25.1.56 ## 10.25.1.57 # Here\u0026#39;s another example of host ranges, this time there are no # leading 0s: ## db-[99:101]-node.example.com As seen above, the default hosts file comes with a full exhaustive list of examples. You may want to truncate it and start fresh. But for now, I have decided to keep it that way for future reference. Also on line 26-27, I have created a group called utility and put my single to be controlled host IP.\nI have added an extra group called nonexistent. This is for demo and we\u0026rsquo;ll use it in upcoming sections. Please note that I do not have any host with that address. It is just a dummy IP address I\u0026rsquo;m using.\nDoing \u0026ldquo;Hello, World!\u0026rdquo; in Ansible # We do \u0026ldquo;hello world\u0026rdquo; in ansible by using a module called ping. Let\u0026rsquo;s see following command.\n1 ansible all -m ping In above command we are saying that on all hosts, run a module (-m) called ping. Let\u0026rsquo;s run that command.\n1 2 $ ansible all -m ping [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match \u0026#39;all\u0026#39; At this point of time, nothing has happened. ansible executeble is looking for hosts in the system inventory file (the one from /etc/ansible/hosts), which is blank at the moment.\nUse local inventory file # To make it use my local inventory, I\u0026rsquo;ll pass it -i inventory. -i is short for --inventory-file. Let\u0026rsquo;s run that and see the output.\n1 2 3 4 5 6 7 8 9 10 11 12 $ ansible all -i inventory -m ping 10.10.10.11 | UNREACHABLE! =\u0026gt; { \u0026#34;changed\u0026#34;: false, \u0026#34;msg\u0026#34;: \u0026#34;Failed to connect to the host via ssh: Permission denied (publickey).\u0026#34;, \u0026#34;unreachable\u0026#34;: true } 10.10.10.12 | UNREACHABLE! =\u0026gt; { \u0026#34;changed\u0026#34;: false, \u0026#34;msg\u0026#34;: \u0026#34;Failed to connect to the host via ssh: ssh: connect to host 10.10.10.12 port 22: Connection timed out\u0026#34;, \u0026#34;unreachable\u0026#34;: true } Both hosts are UNREACHABLE! But if you look closely, both have different reasons. If you have an eye for detail, they are not the ansible errors, they are ssh errors.\nUsing the private keys # Let\u0026rsquo;s look into 10.10.10.11 for now. The error message says: Failed to connect to the host via ssh: Permission denied (publickey). Which means that ansible is able to make a connection to the host. But is not able to authenticate with current configuration.\nThere might be other ways to do this, but I\u0026rsquo;m gonna use the private key which I downloaded when launching the instance\u0026hellip;\nprivate key download dialog Let\u0026rsquo;s use that private key in conjunction with our command.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 $ ansible all --private-key key.pem -i inventory -m ping 10.10.10.11 | SUCCESS =\u0026gt; { \u0026#34;ansible_facts\u0026#34;: { \u0026#34;discovered_interpreter_python\u0026#34;: \u0026#34;/usr/bin/python\u0026#34; }, \u0026#34;changed\u0026#34;: false, \u0026#34;ping\u0026#34;: \u0026#34;pong\u0026#34; } 10.10.10.12 | UNREACHABLE! =\u0026gt; { \u0026#34;changed\u0026#34;: false, \u0026#34;msg\u0026#34;: \u0026#34;Failed to connect to the host via ssh: ssh: connect to host 10.10.10.12 port 22: Connection timed out\u0026#34;, \u0026#34;unreachable\u0026#34;: true } As you can see, one of our hosts has responded with our ping as pong. This is because we passed --private-key key.pem to ansible.\nYou can also override this setting in ansible.cfg file. You can use sed to change the setting file:\n1 2 sed -i \u0026#39;s@/path/to/file@~/workspace/ansible/key.pem@g\u0026#39; ansible.cfg sed -i \u0026#39;s/#private_key_file/private_key_file/g\u0026#39; ansible.cfg And then verify it.\n1 2 $ grep private_key ansible.cfg private_key_file = ~/workspace/ansible/key.pem Now I can run same command without --private-key key.pem.\n1 2 3 $ ansible all -i inventory -m ping 10.10.10.11 | SUCCESS =\u0026gt; { [...output trimmed...] Only execute modules on certain groups # Remember our inventory file had two groups? Named utility and nonexistent? And I told I\u0026rsquo;ll demo it in upcoming sections? This is the time.\nYou can limit the executions to a certain groups.\n1 2 3 4 ansible utility -i inventory -m ping 10.10.10.11 | SUCCESS =\u0026gt; { [...output trimmed...] The all can be replaced any other string which is a group in the inventory file. It was utility in this case.\nAlso, at this point, I want to point this out that one host can be in different groups and that\u0026rsquo;s completely normal.\nHello World with ansible-playbook # Note: I have removed the nonexistent group (i.e. 10.10.10.12) from my inventory file from now on. If you are following this guide on your own and you see unreachability error for 10.10.10.12, then please remove this host from the inventory file.\nWe saw how can we execute some commands on remote hosts with ansible command. Now we are gonna do the same work, but with the help of a Ansible Playbook. The command used to invoke an ansible playbook is ansible-playbook If you do Docker, you can relate ansible playbooks with docker compose and the file docker-compose.yml which stores all the configuration with the playbook we write here. Another common thing between them? YAML.\nI\u0026rsquo;m not doing the comparision on syntax basis, but the behavior basis. Because you can run the same thing with command line and the yaml file. The benefit with yaml file is that, things are documented and staged.\nA trivial Playbook # Let\u0026rsquo;s create a file in current directory and name it mytask.yml, and write following content to that file (which I have stolen from the Ansible docs):\n1 2 3 4 5 6 --- - name: My task hosts: all tasks: - name: Leaving a mark command: \u0026#34;touch /tmp/ansible_was_here\u0026#34; We\u0026rsquo;ll learn more about the syntax after we have done running this playbook. I have total 3 files in my ~/workspace/ansible now.\n1 2 $ ls mytask.yml inventory ansible.cfg Let\u0026rsquo;s execute our mytask playbook now. This is the command I will be executing:\n1 $ ansible-playbook -l utility -i inventory mytask.yml Did you note the little -l utility thingy here? The long version of -l is --limit, which limits the number of host we are executing this playbook on. --limit takes the host group as you can see. And then we have the same -i inventory we had before. Without -i inventory, ansible would look for hosts in /etc/ansible/hosts file, which at the moment is simply blank as we have already seen.\nansible-playbook takes in a YAML file. Get introduced to ansible-playbook. Don\u0026rsquo;t get overwhelmed by that example on that page.\nLet\u0026rsquo;s run the playbook now.\nOutput of the playbook # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 $ ansible-playbook -l utility -i inventory mytask.yml PLAY [My task] ****************************************************************************************************** TASK [Gathering Facts] ********************************************************************************************** ok: [10.10.10.11] TASK [Leaving a mark] *********************************************************************************************** [WARNING]: Consider using the file module with state=touch rather than running \u0026#39;touch\u0026#39;. If you need to use command because file is insufficient you can add \u0026#39;warn: false\u0026#39; to this command task or set \u0026#39;command_warnings=False\u0026#39; in ansible.cfg to get rid of this message. changed: [10.10.10.11] PLAY RECAP ********************************************************************************************************** 10.10.10.11 : ok=2 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 I can go ahead and check if playbook actually ran by going into my host and checking if /tmp/ansible_was_here file was actually created. I can do that by ssh itself by issuing following command.\n1 2 $ ssh 10.10.10.11 ls -l /tmp/ansible* -rw-rw-r-- 1 ec2-user ec2-user 0 Oct 24 15:53 /tmp/ansible_was_here Yes, it did. We successfully ran our first playbook.\nAnatomy of mytask.yml # Let\u0026rsquo;s look at our mytask.yml again.\n1 2 3 4 5 6 7 --- - name: My task hosts: all tasks: - name: Leaving a mark command: \u0026#34;touch /tmp/ansible_was_here\u0026#34; Please not that indentation matter when working with YAML files, just like indentation matters in Python\u0026rsquo;s syntax. With that said, I\u0026rsquo;ll go line by line:\n---. This is not a Ansible thing, but a YAML thing. A YAML file can optionally start with --- and end with .... People at redhat chose this for readability. Learn more about this. - name: My task. Before I even start typing this line. I would like to point out that, not only Python, but Ansible also heavily belives in readability and self documentation. There are some things without which ansible can work, but they are there for sake of self documentation. So that a new person in the team can get benefited. This line simply documents the name of the playbook. hosts: all. Do you remember utility and nonexistent from Setting up the inventory section? You can put thost groups here in the place of all. tasks:. This is a list of tasks. Each task at least should have one name and one module. - name:. This is the name of the task instead of the playbook we saw above. Please note that names are not important but highly encouraged. Just image above playbook output without any names. Won\u0026rsquo;t you be lost in the output? command:. command is one of the core modules of ansible. It is used to execute arbitrary command on remote hosts. Installing packages on hosts # I\u0026rsquo;m eager to install nginx on my system. That was the reason I started learning Ansible.\nI went ahead and came up with this playbook which uses 2 real life modules called yum and service. As I have CentOS based distro, thus yum. And I suppose service is agnostic to most distros.\nInstalling nginx # I put the following code in a file called nginx.yml in te root of our ansible workspace.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 --- - name: install and start nginx hosts: web tasks: - name: install nginx yum: name: - nginx state: latest - name: start nginx service: name: nginx state: started And I tried to run it. And this happened:\n1 2 3 4 5 6 7 8 9 10 11 12 13 $ ansible-playbook -i inventory nginx.yml PLAY [install and start nginx] ************************************************************************************** TASK [Gathering Facts] ********************************************************************************************** ok: [10.10.10.11] TASK [install nginx] ************************************************************************************************ fatal: [10.10.10.11]: FAILED! =\u0026gt; {\u0026#34;changed\u0026#34;: true, \u0026#34;changes\u0026#34;: {\u0026#34;installed\u0026#34;: [\u0026#34;nginx\u0026#34;], \u0026#34;updated\u0026#34;: []}, \u0026#34;msg\u0026#34;: \u0026#34;You need to be root to perform this command.\\n\u0026#34;, \u0026#34;rc\u0026#34;: 1, \u0026#34;results\u0026#34;: [\u0026#34;Loaded plugins: extras_suggestions, langpacks, priorities, update-motd\\n\u0026#34;]} PLAY RECAP ********************************************************************************************************** 10.10.10.11 : ok=1 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 \u0026ldquo;You need to be root to perform this command\u0026rdquo;. Fair enough. What I\u0026rsquo;m tring to do with nginx playbook is to install nginx, which obviously requires administrative privilege.\nAfter a bit of research, I found that I have to become root for privilege escalation. And the way be do it in playbook is by adding become: yes in the tasks which require them. In my case, both of them required sudo so I put them in the root indendation level, i.e. just below the hosts: web. Like so:\n1 2 3 4 5 6 --- - name: install and start nginx hosts: web become: yes tasks: And the playbook plays well:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 $ ansible-playbook -i inventory nginx.yml PLAY [install and start nginx] *********************************************************************************** TASK [Gathering Facts] ******************************************************************************************* ok: [10.10.10.11] TASK [install nginx] ********************************************************************************************* changed: [10.10.10.11] TASK [start nginx] *********************************************************************************************** changed: [10.10.10.11] PLAY RECAP ******************************************************************************************************* 10.10.10.11 : ok=3 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 This was the minimalist playbook I could have ever run. Next I\u0026rsquo;m going to do a little bit more and reverse proxy my Jenkins server using Nginx.\nInsider info: One of the reason I\u0026rsquo;m doing Ansible is to configure Jenkins server which is currently running on port 8080 without HTTPS, which is risky. First I want to configure nginx, then proceed with HTTPS Anywhere certificate, and then at the end I might end up doing Jenkins with ansible itself.\nDealing with config files: Copying the nginx.conf # You might be aware of recent post which I did about Enabling HTTPS on EC2 Hosted Website. I\u0026rsquo;m taking that work as a reference and proceed.\nHere is a modified version of nginx.conf I\u0026rsquo;m going to use. This code does not resembles the entire nginx.conf, but I want you to look at the highlighted text. You need to add that in the defualt configuration file. If in doubt, copy the entire gist code from here.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 http { server { server_name _; root /usr/share/nginx/html; location / { proxy_pass http://localhost:8080/; } error_page 404 /404.html; location = /40x.html { } error_page 500 502 503 504 /50x.html; location = /50x.html { } } } Please note that Jenkins is already running on port 8080 right now on my nginx machine and that is where I\u0026rsquo;m routing my incoming traffic to. If not Jenkins, you can run a python simple HTTP server with python3 -m http.server 8080 on 10.10.10.11 host for simplicity and testing purpose.\nI have to add a task for copying configuration to my playbook now:\n6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 tasks: - name: install nginx yum: name: - nginx state: latest - name: copy config for jenkins routing copy: src: nginx.conf dest: /etc/nginx/nginx.conf mode: preserve notify: restart nginx - name: start nginx service: name: nginx state: started handlers: - name: restart nginx service: name: nginx There are 2 new things going on here:\nThe copy module which is used to copy files around. You can read more about what it does and what alternative are available to it on copy documentation page. We also see the new handlers: block which works closely with notify: section. handlers are similar to tasks in some sense (thus in same indentation level). It is different in the sense that a handler is only run when it is notified by some tasks (correct me if I am wrong).\nYou must also know that handlers are only notified if the notifying task is actually done running. Means that, if the task has no changes to make, it won\u0026rsquo;t notify the handler. We will see this with help of an example now.\nLet\u0026rsquo;s run our playbook with our latest addition:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 $ ansible-playbook -l web -i inventory nginx.yml PLAY [install and start nginx] **************************************************************** TASK [Gathering Facts] ************************************************************************ ok: [10.10.10.11] TASK [install nginx] ************************************************************************** changed: [10.10.10.11] TASK [copy config for jenkins routing] ******************************************************** changed: [10.10.10.11] TASK [start nginx] **************************************************************************** changed: [10.10.10.11] RUNNING HANDLER [restart nginx] *************************************************************** ok: [10.10.10.11] PLAY RECAP ************************************************************************************ 10.10.10.11 : ok=5 changed=3 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 Handler only runs when a tasks chages the state # Without actually doing anything, try running the playbook again:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 $ ansible-playbook -l web -i inventory nginx.yml PLAY [install and start nginx] **************************************************************** TASK [Gathering Facts] ************************************************************************ ok: [10.10.10.11] TASK [install nginx] ************************************************************************** ok: [10.10.10.11] TASK [copy config for jenkins routing] ******************************************************** ok: [10.10.10.11] TASK [start nginx] **************************************************************************** ok: [10.10.10.11] PLAY RECAP ************************************************************************************ 10.10.10.11 : ok=4 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0 As you can see, the restart is not run this time. Go ahead and modify the config file and run the play as a homework.\nNext Steps # Sadly I\u0026rsquo;ll have to end this post here as it is getting too long. I have already done Jenkins installation playbook as of now if you want to take a look you can take it here and discover new modules:\nhttps://gist.github.com/santosh/78183e403e6a98d263130ba52aa0f593\nI\u0026rsquo;ve posted about configuring HTTPS on a wildcard basis in a separate post. This post is different from previous one in the sense that it configures HTTPS not only for the domain, but also it\u0026rsquo;s all of subdomains.\nConclusion # I also know I have not covered everything in Ansible and some of you might be sad on that. I did\u0026rsquo;t cover Roles, I didn\u0026rsquo;t cover templating. I didn\u0026rsquo;t cover those giant folder structure.\nMy journey with Ansible is not this short. Ansible is not only radically simple, but also you can extend it\u0026rsquo;s capability by a simple language called Python.\nPlease let me know how you feel about this post in the comment section. Feel free to share it with your network. And don\u0026rsquo;t forget to subscribe using the form below.\n","date":"24 October 2021","externalUrl":null,"permalink":"/posts/2021/getting-started-with-ansible-as-a-fullstack-developer/","section":"Posts","summary":"Introduction # I don’t claim to be a DevOps expert. I just know some basic ssh and scp, that’s all. I have done whatever server configuration till date by ssh’ing into instances and then configuring them by hand. But now-a-days I feel like spinning up and destroying a server has increased in my life style. And doing it manually is just waste of time when we already have solution like Ansible.\n","title":"Getting Started with Ansible as a Fullstack Developer","type":"posts"},{"content":"","date":"24 October 2021","externalUrl":null,"permalink":"/tags/reverseproxy/","section":"Tags","summary":"","title":"Reverseproxy","type":"tags"},{"content":" Introduction # There are a plenty of VPN providers out there. What\u0026rsquo;s common between all of them? You can\u0026rsquo;t trust them. They may have a no-log policy but as a company, there is always a tendency that they might share our browsing data to some third party. In this post you\u0026rsquo;ll learn how you can spin up your own VPN with Amazon AWS.\nThis post is not necessarily applicable to AWS, but any other IaaS. But I will use AWS.\nWhile you read this post, take a moment to connect with me on LinkedIn.\nNote: While this post is for educational purpose, I would like to bring it up that you must comply with AWS T\u0026amp;C (or any other VPC provider) and your local law. I will not be held reponsible for any blunder you make.\nPrerequisites # I am assuming that you are a cloud practitioner and you know your way around to ssh into your running Linux instance.\nThings you should know before starting.\nYour EC2 instance is billed hourly. The data coming and going out of the instance is charged per GB basis. t2.micro instance type is more than enough for this experiment.\nPlease check the pricing here and here and proceed accordingly. It won\u0026rsquo;t cost much if you are doing this for experimental purpose.\nInstall OpenVPN Server # OpenVPN is a virtual private network (VPN) system that implements techniques to create secure point-to-point or site-to-site connections in routed or bridged configurations and remote access facilities. It implements both client and server applications.\nWhat a typical VPN provider does is, when you are connected to the VPN provider, every request you make to internet, your source IP is seems to be coming from the VPN provider. The setup we are going to do will have show your EC2 instance\u0026rsquo;s public IP.\nInstallation Process # 1 2 sudo -i curl -O https://raw.githubusercontent.com/angristan/openvpn-install/master/openvpn-install.sh sudo -i is to change the current user to root.\nThe curl command downloads the installation script which is the easiest way to install OpenVPN on my Linux servers today.\n1 bash openvpn-install.sh At the given point, my typical installation looks like this:\n1 2 3 4 5 6 7 8 9 Welcome to the OpenVPN installer! The git repository is available at: https://github.com/angristan/openvpn-install I need to ask you a few questions before starting the setup. You can leave the default options and just press enter if you are ok with them. I need to know the IPv4 address of the network interface you want OpenVPN listening to. Unless your server is behind NAT, it should be your public IPv4 address. IP address: 10.2.1.201 Don\u0026rsquo;t creep out if you are doing this installation for the first time. The inputs are auto-filled which works for most users.\nNext question we get is about the public IP address:\n1 2 3 It seems this server is behind NAT. What is its public IPv4 address or hostname? We need it for the clients to connect to the server. Public IPv4 address or hostname: 13.233.130.22 This will also be auto-filled.\nNext up, we are asked if we want to enable IPv6 connectivity. I am not opting to install IPv6 for this time. I will do some tutorial for that at later time.\n1 2 3 4 5 Checking for IPv6 connectivity... Your host does not appear to have IPv6 connectivity. Do you want to enable IPv6 support (NAT)? [y/n]: n Choose a port where OpenVPN will run. I am opting the default port 1194. You may want to change it to something else.\n1 2 3 4 5 What port do you want OpenVPN to listen to? 1) Default: 1194 2) Custom 3) Random [49152-65535] Port choice [1-3]: 1 Next up is the protocol. I prefer UDP as most VPN providers do that.\n1 2 3 4 5 What protocol do you want OpenVPN to use? UDP is faster. Unless it is not available, you shouldn\u0026#39;t use TCP. 1) UDP 2) TCP Protocol [1-2]: 1 I am going with the default DNS resolver which installer provides. But you are always welcome to change this at your will.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 What DNS resolvers do you want to use with the VPN? 1) Current system resolvers (from /etc/resolv.conf) 2) Self-hosted DNS Resolver (Unbound) 3) Cloudflare (Anycast: worldwide) 4) Quad9 (Anycast: worldwide) 5) Quad9 uncensored (Anycast: worldwide) 6) FDN (France) 7) DNS.WATCH (Germany) 8) OpenDNS (Anycast: worldwide) 9) Google (Anycast: worldwide) 10) Yandex Basic (Russia) 11) AdGuard DNS (Anycast: worldwide) 12) NextDNS (Anycast: worldwide) 13) Custom DNS [1-12]: 11 1 2 3 4 5 6 7 8 9 10 11 12 13 Do you want to use compression? It is not recommended since the VORACLE attack make use of it. Enable compression? [y/n]: n Do you want to customize encryption settings? Unless you know what you\u0026#39;re doing, you should stick with the default parameters provided by the script. Note that whatever you choose, all the choices presented in the script are safe. (Unlike OpenVPN\u0026#39;s defaults) See https://github.com/angristan/openvpn-install#security-and-encryption to learn more. Customize encryption settings? [y/n]: n Okay, that was all I needed. We are ready to setup your OpenVPN server now. You will be able to generate a client at the end of the installation. Press any key to continue... I am not enabling any compression for this time.\nI am not customizing any encryption this time.\nThat\u0026rsquo;s all we need to do to continue installation.\nDepending on your distro type, the installer will use the package manager to download and install the package.\nCreate a Client Config File # As we already discussed in the start of the post, we are doing a single server and single client setup. For each client, we need to create a client configuration.\nThis configuration is automatically created as part of the OpenVPN installer. Go ahead and give a name to the configuration file.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 Tell me a name for the client. The name must consist of alphanumeric character. It may also include an underscore or a dash. Client name: first-client Do you want to protect the configuration file with a password? (e.g. encrypt the private key with a password) 1) Add a passwordless client 2) Use a password for the client Select an option [1-2]: 1 Note: using Easy-RSA configuration from: /etc/openvpn/easy-rsa/vars Using SSL: openssl OpenSSL 1.0.2k-fips 26 Jan 2017 Generating a 256 bit EC private key writing new private key to \u0026#39;/etc/openvpn/easy-rsa/pki/easy-rsa-9322.WOFVgi/tmp.kkk8CD\u0026#39; ----- Using configuration from /etc/openvpn/easy-rsa/pki/easy-rsa-9322.WOFVgi/tmp.CneLJl Check that the request matches the signature Signature ok The Subject\u0026#39;s Distinguished Name is as follows commonName :ASN.1 12:\u0026#39;first-client\u0026#39; Certificate is to be certified until Jan 7 02:07:48 2024 GMT (825 days) Write out database with 1 new entries Data Base Updated Client first-client added. The configuration file has been written to /home/ec2-user/first-client.ovpn. Download the .ovpn file and import it in your OpenVPN client. As you can see, the key file has been written to disk at /home/ec2-user/first-client.ovpn.\nDownload the key file to your own system # As you saw the last section, the config file is located at /home/ec2-user.\nThe next task is to download this ovpn file to my local system. I like to use scp, a cousin of ssh to do this.\n1 scp -i /path/to/key.pem ec2-user@xxx.xxx.xxx.xxx:/home/ec2-user/first-client.ovpn . I get the first-client.ovpn on my local machine. With this key in your hand, you can use this to connect a vast kind of devices, including Windows, MacOS, Linux, Android and iOS.\nInstall VPN client on local machine # There might be other implementation of VPN clients, but I use OpenVPN Connect on my machine. You can get it here:\nhttps://openvpn.net/vpn-client/ I have OpenVPN Connect installed on my machine and ready to take in the .ovpn config file we downloaded from the remote instance.\nOpenVPN Connect on my local machine Before we go ahead and have a private connection between you and your instance. We need to allow connection from internet to this instance.\nOpen port on the EC2 instance # At this point of time, if you are installing OpenVPN on the server, you might have connected via SSH. I assume your port 22 is open.\nSimilarly, we need to create a new hole in our firewall to allow OpenVPN connections on the choice of port we chose while installing. The default port is 1194.\nCreate a security group called openvpn and create an inbound rule with these settings:\nAllow incoming traffic from 1194 Go ahead and attach it to the running instance which is running the OpenVPN server.\nYou may as well wish to tighten the security here (by providing CIDR/IP ranges in Source field) so that not everyone on the internet is able to connect to us. On the other hand, if you wish to share this VPN with everyone and want to charge then leave it to 0.0.0.0/0.\nConnect your computer to the VPN server # You just need to click \u0026ldquo;Browse\u0026rdquo; in the panel shown below and browse to the .ovpn file.\nBrowse the config file on local computer After a connection has been made, you should see something like this:\nAn active VPN connection Conclusion # You can go ahead and configure this connection at the router level (assuming you have router capable of that) so that all the clients inside the network can be benefited.\nIn next post, I might come up with ways to block malware and tracker with our VPN connection.\nIf you are having trouble or have any suggestion, please leave a comment and we can discuss. If you liked this post, please share it with your network. Subscribe to the newsletter below for similar news.\n","date":"3 October 2021","externalUrl":null,"permalink":"/posts/2021/how-to-spin-up-your-own-vpn-on-ubuntu-20.04/","section":"Posts","summary":"Introduction # There are a plenty of VPN providers out there. What’s common between all of them? You can’t trust them. They may have a no-log policy but as a company, there is always a tendency that they might share our browsing data to some third party. In this post you’ll learn how you can spin up your own VPN with Amazon AWS.\n","title":"How to Spin Up Your Own VPN on Ubuntu 20.04","type":"posts"},{"content":" Introduction # Started with my Pentium 4 processor desktop with merely 256 MB of RAM. And today I\u0026rsquo;m port forwarding sitting on my laptop in a cloud enabled web. Today we\u0026rsquo;ll see how I have manage to make a tunnel between my laptop and EC2 instance that I own.\nIn this short post, I will show how we can forward port from a remote host to a local computer. We will forward a port from AWS EC2 to local computer with openssl.\nSo let\u0026rsquo;s cut the shit and proceed ahead.\nHow I do port forwarding with ssh? # This is how I open a tunnel from my local computer port 5901 to my EC2 instance in AWS with public ip of xxx.xxx.xxx.xxx. This command is run from the local computer.\n1 ssh -L 5901:localhost:5901 -i /path/to/key.pem ec2-user@xxx.xxx.xxx.xxx I find it typically easier to use our very own ssh command which we already have in Linux and if you are using Windows, you will get it with installation of Git Bash.\nSo let\u0026rsquo;s break down the command:\nssh - Our friendly ssh command. -L 5901:localhost:5901 - This tells the SSH to forward remote port 5901 to local port (thus localhost) 5901. -i /path/to/key.pem - This is to connect to remote host with a identity file. It is more secure than basic auth. ec2-user@xxx.xxx.xxx.xxx - This is my username and host IP. So basically you can say that there is a tunnel between xxx.xxx.xxx.xxx:5901 and localhost:5901.\nAnything running on xxx.xxx.xxx.xxx:5901 could also be viewed on your local computer. This too without opening any port on the remote host.\nAnd on top of that. This connection is end to end encrypted SSL.\nConclusion # There are multiple many ways to forward port. What method do you use when you have to forward a port?\nRelated Links: # https://superuser.com/q/284051/103134 https://en.wikipedia.org/wiki/Network_address_translation https://en.wikipedia.org/wiki/Port_forwarding ","date":"14 September 2021","externalUrl":null,"permalink":"/posts/2021/port-forwarding-and-how-to-do-it-with-ssh/","section":"Posts","summary":"Introduction # Started with my Pentium 4 processor desktop with merely 256 MB of RAM. And today I’m port forwarding sitting on my laptop in a cloud enabled web. Today we’ll see how I have manage to make a tunnel between my laptop and EC2 instance that I own.\nIn this short post, I will show how we can forward port from a remote host to a local computer. We will forward a port from AWS EC2 to local computer with openssl.\n","title":"Port Forwarding and How to Do It With SSH","type":"posts"},{"content":"","date":"2 September 2021","externalUrl":null,"permalink":"/tags/terraform/","section":"Tags","summary":"","title":"Terraform","type":"tags"},{"content":" Introduction # I have been using AWS with Terraform to provision my development instance on AWS cloud. When I wanted to configure dotfiles, change SSH port and configure some cron jobs, I hit a bottleneck. I literally was not able to do those tasks with user scripts because they had no effect. That was the time when creating custom image of the distro came into my mind.\nIn this post I\u0026rsquo;ll go through the step which I took to create new image which has some pre-configured software and config which was really hard to configure with Terraform itself.\nWhile you read this post, take a moment to connect with me on LinkedIn.\nEven if you are not a developer, you might find this post useful if you want certain packages pre-installed and pre-configured on the EC2 operating system. So without further ado, let\u0026rsquo;s get started.\ndev-on-ec2, my custom AMI Pre-requisites # AWS account. If you don\u0026rsquo;t have already, head over to https://aws.amazon.com/console/ and look for a button saying Create an AWS Account. You know how to create instances and connect to it via SSH. I won\u0026rsquo;t be going through how to create an instance and connect to it in this post. But I have an advice. While creating instance keep in mind the size of the root block device. Keep it to minimum. Later on if you want to create an instance based on your AMI, the minimum storage you can allocate at that point of time should be more than the one used while creating the instance for AMI creation.\nInstructions # I\u0026rsquo;ll use Amazon Linux 2 for the base image as I\u0026rsquo;m fan of Red Hat based distros. But if you know your way around your distro, there should be no problem.\nOutline of packages/configs we are going to configure in this AMI.\n• Install and configure basic packages\n• Install dotfiles\n• Change SSH port\n• Create cron job for routine tasks\nInstall and configure basic packages # SSH into your instance and run the following command one by one. But before even you do that, I want you to run export HISTFILE=/dev/null; history -d $(history 1). I\u0026rsquo;ll tell you what this command does at the [end of the post].\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 sudo yum update -y -q # Install initial tools sudo yum group install \u0026#39;Development Tools\u0026#39; -y -q sudo amazon-linux-extras install epel -y sudo yum install vim-X11 golang docker tmux tree python3 git-lfs htop -y # Configure initial tools sudo systemctl start docker sudo systemctl enable docker sudo groupadd docker sudo usermod -aG docker ec2-user sudo newgrp docker # Set upper limit to journalctl journalctl --vacuum-time=180d Let\u0026rsquo;s go through above commands one by one.\n1 sudo yum update -y -q The yum update -y -q command updates the system, and dose not ask for confirmation (-y), and does it silently (-q).\n1 sudo yum group install \u0026#39;Development Tools\u0026#39; -y -q This command install a group of packages tagged as Development Tools. Although I only use git from this group, having everything installed keeps things sorted. If you are tight on space, please consider only installing git with sudo yum install git.\n1 sudo amazon-linux-extras install epel -y Next we install epel which is a software repository which brings many other packages to yum\u0026rsquo;s doorstep.\n1 sudo yum install vim-X11 golang docker tmux tree python3 git-lfs htop -y These are the packages I use on regular basis.\nI use vim-X11 instead of vanilla vim for the mouse support. vim-X11 enables me to navigate windows with mouse. You probably should be aware of golang and docker if you read my blog on a regular basis. tmux enables me to use same ssh connection and multiplex multiple terminal window inside it. tree prints out directory tree in a hierarchical fashion. If you want to see it in action, go to your home folder and invoke it from there. 1 2 3 4 5 sudo systemctl start docker sudo systemctl enable docker sudo groupadd docker sudo usermod -aG docker ec2-user sudo newgrp docker With above commands, we start the docker service. We enable it on boot. Create a new group called docker. Add the ec2-user to the docker group. And then newgrp is used to change the current group ID during a login session.\n1 journalctl --vacuum-time=180d With this command, I intend to limit the log retention time to approx. 6 months.\nInstall dotfiles # I have a collection of dotfiles ranging in configuration of packages including\n1 2 3 4 5 # Install dotfiles. cd ~ git clone https://github.com/santosh/.dotfiles.git cd .dotfiles make install The above commands will clone and install my personal dotfiles.\nChange SSH port # I tend to change SSH port to something other than port 22.\n1 sed -i.bak \u0026#34;s/#Port 22/Port 12121/g\u0026#34; /etc/ssh/sshd_config Configure cron jobs # In this section, I\u0026rsquo;ll setup some tasks with crontab. Routine tasks include mostly the cleanup tasks to keep the EBS space free.\nThis is my entry in crontab:\n1 2 3 4 5 6 7 8 9 10 11 12 #!bin/bash # Monthly cleanup scheme for master_node of development environment. # 1. Clean docker cache docker system prune -f # 2. Clean yum cache sudo rm -rf /var/cache/yum # 3. Clean /tmp (files older than 15 days) find /tmp -ctime +15 -exec rm -rf {} + I tent to run this monthly, so I have created an entry in crontab like this:\n1 @monthly /path/to/shell/script.sh If you are not familiar with cron syntax, I find https://crontab.guru/ a great resource.\nWith this, my distro is ready to be transformed into an image or AMI which could later be used to boot up an cloud computer at AWS.\nHere is the complete bootstrap file.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 #!/bin/bash # Note: This script is intended to run in an interactive shell. # Use this script to create fresh bootstrapped image. { set +x; } 2\u0026gt;/dev/null sudo yum update -y -q sudo yum upgrade -y -q # Install initial tools sudo yum group install \u0026#39;Development Tools\u0026#39; -y -q sudo amazon-linux-extras install epel -y -q sudo yum install vim-X11 golang docker tmux tree python3 git-lfs htop -y -q echo Done installing packages. # Configure initial tools sudo systemctl start docker sudo systemctl enable docker if [ $(getent group docker) ]; then echo \u0026#34;docker group already exists; skipping.\u0026#34; else sudo groupadd docker fi sudo usermod -aG docker ec2-user sudo newgrp docker echo Done installing docker # Set upper limit to journalctl journalctl --vacuum-time=180d # Install dotfiles. cd ~ git clone https://github.com/santosh/.dotfiles.git cd .dotfiles make install echo Done configuring dotfiles. # Change SSH Port echo -n \u0026#34;ENTER a random port number: \u0026#34; read SSH_PORT if [[ ! $SSH_PORT =~ ^[0-9]+$ ]] ; then echo \u0026#34;SSH port number must be an positive integer.\u0026#34; exit fi sudo sed -i.bak \u0026#34;s/#Port 22/Port $SSH_PORT/g\u0026#34; /etc/ssh/sshd_config echo Done bootstrapping. A more updated version can always be found at https://github.com/santosh/.dotfiles/blob/master/bootstrap/amazon-linux.sh. You can run this script in a single go with following command.\nsh \u0026lt;(curl -s https://github.com/santosh/.dotfiles/raw/master/bootstrap/amazon-linux.sh) In next section, we\u0026rsquo;ll see how\nCreate a boot image/AMI # Make sure you\u0026rsquo;re at EC2 home in AWS Console.\nSelect the instance in which you ran commands mentioned in previous sections.\nDirections to create AMI from a running instance Give a name for the AMI and hit Create Image. Don\u0026rsquo;t forget to keep storage to minimum This image can be used to boot up other instances. dev-on-ec2, my custom AMI Do you remember? # Do you remember I told you to run export HISTFILE=/dev/null; history -d $(history 1)? This command disables history preservation for the current shell session. In case you missed it, you might see the entered command if you press up arrow in bash. I wanted you to have fresh bash history when you start a new instance.\nConclusion # Do you find something missing? Is something broken? Please let me know by leaving a comment and we\u0026rsquo;ll fix it together.\nIf you liked this post, please share it with your network. Subscribe to the newsletter below for similar news.\n","date":"2 September 2021","externalUrl":null,"permalink":"/posts/2021/when-user-data-scripts-are-not-enough-create-a-new-image/","section":"Posts","summary":"Introduction # I have been using AWS with Terraform to provision my development instance on AWS cloud. When I wanted to configure dotfiles, change SSH port and configure some cron jobs, I hit a bottleneck. I literally was not able to do those tasks with user scripts because they had no effect. That was the time when creating custom image of the distro came into my mind.\n","title":"When User Data Scripts Are Not Enough, Create a New Image","type":"posts"},{"content":"","date":"20 August 2021","externalUrl":null,"permalink":"/tags/microservices/","section":"Tags","summary":"","title":"Microservices","type":"tags"},{"content":"This is my rundown of AWS\u0026rsquo;s whitepaper about Implementing Microservices on AWS. Without further ado, let\u0026rsquo;s get started. In this article I talk more about micrsoservices and less about AWS. I also talk about paths already engraved around microservices.\nWhile you read this post, take a moment to connect with me on LinkedIn.\nWhat are microservices? # Microservices are an architectural and organizational approach to software development where software is composed of small independent services that communicate over well-defined APIs. These services are owned by small, self-contained teams.\nBenefits of microservices # So with microservices, the main benefits we get are:\nWe can divide developers into logical groups. Faster deployment cycles. From developer point of view, we get flexibility of choosing technologies we are familiar with. We can scale the application based on load horizontally which is cost effective) We get resiliency because if one machine goes down, there is always other machine to take over in shorter span of time, automatically. Contrast between a monolith and a microservice # We can contract microservices with monolith architecture. The table below shows differences between them.\nMonolith Microservices Application built as one unit, using different layers Applications built as a series of individual components Scaling any part of the app means scaling the entire app Each component can scale independently. As the applications change, their code bases become more complex Changes are simpler because they\u0026rsquo;re confined to individual components Teams have limited autonomy Teams have more autonomy to make changes as they need to As you can see in the monolith architecture image below, we have User Interface, Account Service, Cart Service, Shipping Service, Data Access Service layers.\nMonolith architecture In microservices pattern, we get to choose hardware based on our requirements. If a service which requires more IO throughput, it will get that. If a service requires memory, it will get it. If it needs more CPU it will get it. This all happens on per service basis. In monolith, you\u0026rsquo;d have to scale everything entirely.\nMicroservice architecture And if the Shipping Service team decided to change their data model, it will not affect other services in the application.\nContrast between Serverless and Microservice # Serverless is more of an architectural term in terms of infrastructure. You don\u0026rsquo;t need to provision, configure and scale hosts on your own.\nWhereas microservice design pattern can be implemented regardless of whether the machines are provisioned automatically or manually.\nAWS Shared Responsibility Model # While you develop your application in AWS Clould, you need to keep in mind that unlike other conventional hosting providers like PythonAnywhere / Heroku / DigitalOcean, you have more things to handle on your own. Things like networking, security groups, firewall etc, you need to configure on your own. Of course there are different levels of offering in AWS itself that provides heroku level of setup out of the box but you know what I mean.\nAWS\u0026rsquo;s Shared Responsibility Model Challenges faced while first moving to microservices # Common challenges encountered in the early stages of microservices adoption.\nEven small code changes take too long to release. In monolith, there is only one application. Easy to track stack traces. Manual deployments create lack of awareness of what\u0026rsquo;s running Low visibility reduces time to resolution What works in dev environments doesn\u0026rsquo;t always work in production. Up-front costs may be higher with microservices. Best practices for successful microservices adoption # These points are important. Many businesses are not able to adopt microservices because they don\u0026rsquo;t follow a plan.\nCI/CD # You need to bring CI/CD and DevOps practice in your application lifecycle. Automate your software delivery process using continuous integration and delivery (CI/CD) pipelines. Monitoring and Observability # Keep tract of all the components of your microservices architecture to understand if they\u0026rsquo;re performing correctly.\nUse metrics Measure application performance Logging Security # Implement the proper security measure across your microservices architecture.\nCode scanning/static analysis Runtime monitoring Incident response (automation and alerting) Everybody in the application lifecycle, from development, to building, to\ntesting, to deployment. Application/Service Management # Monitor and manage application tools to maintain a high quality of service.\nDeployment and application frameworks Application visualization and comprehension Configuration/secrets management/service discovery Distributed Systems Components # After looking at how AWS can solve challenges related to individual microservices, we now want to focus on cross-service challenges, such as service discovery, data consistency, asynchronous communication, and distributed monitoring and auditing.\nCommunication between services # When breaking down monolith to microservices, one pattern which can be used here is the Producer-Consumer pattern, also known as the publisher subscriber pattern.\nWith producer-consumer pattern we can break the system into two parts, i.e. Producers, and Consumers. Consider the example below.\nProducer/Consumer pattern with single queue As you can see, in sub/pub system there are two kinds of services involved. One is the publisher service, also known as the producer which is responsible for taking request from load balancer or other downstream service and then convert it into a task. This task goes to a queue which which accumulates all tasks.\nNext, there are certain number of workers which can subscribed to the queue. You\u0026rsquo;d pick a task from the queue, work on it, and then delete that task from the queue so that no task is done twice.\nOne benefit we get here is that we can scale (add or remote instances) workers and consumers separately. With monolith, we\u0026rsquo;d have to scale vertically.\nProducer/Consumer pattern with multiple queues Note that there are different kind of queues, routing, exchanges etc which will take another post to explain it in detail. But, working with technologies like RabbitMQ would expose you to such terminologies.\nService Discovery # Service discovery is essential part of microservices. Each host on the cluster needs to be known by every other host. If we go by the Wikipedia definition of service discovery, Service discovery is the automatic detection of devices and services offered by these devices on a computer network.\nThe distributed characteristics of microservices architectures not only make it harder for services to communicate, but also presents other challenges, such as checking the health of those systems and announcing when new applications become available. You also must decide how and where to store meta-store information, such as configuration data, that can be used by applications.\nIf we talk about software packages which can manage service discovery for us then we have server software like Traefik, Netflix Eureka, HashiCorp Consul.\nChatiness # By breaking monolithic applications into small microservices, the communication overhead increases because microservices have to talk to each other. In many implementations, REST over HTTP is used because it is a lightweight communication protocol but high message volumes can cause issues.\nCaching # Caches are a great way to reduce latency and chattiness of microservices architectures. Several caching layers are possible, depending on the actual use case and bottlenecks. Many microservice applications running on AWS use Amazon ElastiCache to reduce the volume of calls to other microservices by caching results locally. API Gateway provides a built-in caching layer to reduce the load on the backend servers. In addition, caching is also useful to reduce load from the data persistence layer. The challenge for any caching mechanism is to find the right balance between a good cache hit rate and the timeliness/consistency of data.\nFurther readings # The Twelve-Factor App Proxy Reverse Proxy Load Balancer https://en.wikipedia.org/wiki/Producer%E2%80%93consumer_problem https://refactoring.guru/design-patterns/facade If you liked this post, please share it on social media and don\u0026rsquo;t forget to subscribe below.\n","date":"20 August 2021","externalUrl":null,"permalink":"/posts/2021/microservices-and-aws/","section":"Posts","summary":"This is my rundown of AWS’s whitepaper about Implementing Microservices on AWS. Without further ado, let’s get started. In this article I talk more about micrsoservices and less about AWS. I also talk about paths already engraved around microservices.\nWhile you read this post, take a moment to connect with me on LinkedIn.\nWhat are microservices? # Microservices are an architectural and organizational approach to software development where software is composed of small independent services that communicate over well-defined APIs. These services are owned by small, self-contained teams.\n","title":"Microservices and AWS","type":"posts"},{"content":"","date":"27 April 2021","externalUrl":null,"permalink":"/tags/channel/","section":"Tags","summary":"","title":"Channel","type":"tags"},{"content":"","date":"27 April 2021","externalUrl":null,"permalink":"/tags/concurrency/","section":"Tags","summary":"","title":"Concurrency","type":"tags"},{"content":" Introduction # I\u0026rsquo;ve dabbled over many languages in the past. Let it be PHP, Java, Python, C/C++, JavaScript and now Go (some of them I only know the syntax). With the experience of all the languages, when I switched to Go, I had no pain understand slices, struct, interfaces, relatively less pain understanding pointers (I\u0026rsquo;m still learning). But in non of those languages, I did concurrent programming. Although in my first company I did some concurrent programming using Python threads, but the threads didn\u0026rsquo;t share any data between them.\nWhile you read this post, take a moment to connect with me on LinkedIn.\nTo understand how goroutines or/and threads work, you need to understand how process management works at the operating system level. If you have read about operating system, process management in particular then this concept would be very easy for you. With that said, we must understand that concurrency is not parallelism. Refer to the following pictures which I shamelessly copied from StackOverflow.\nconcurrency vs parallelism We\u0026rsquo;ll proceed gradually towards the more advance examples as we proceed.\nPattern 1: WaitGroup # Let us consider this example with basic goroutine.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 package main import ( \u0026#34;fmt\u0026#34; ) func hello() { fmt.Println(\u0026#34;Hello world goroutine\u0026#34;) } func main() { go hello() fmt.Println(\u0026#34;main function\u0026#34;) } Along with the main thread, a new go thread will be spawned for hello func. You might not see the output of hello as the program will exit before hello finishes.\n1 main function Concurrency is not about adding go keyword at the start of a function, rather it\u0026rsquo;s an art of orchestrating multiple go threads which in go world called goroutine.\nTo make this work, we need to add a WaitGroup to wait for all the goroutines to end processing.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 package main import ( \u0026#34;fmt\u0026#34; \u0026#34;sync\u0026#34; ) var wg sync.WaitGroup func hello() { fmt.Println(\u0026#34;Hello world goroutine\u0026#34;) wg.Done() } func main() { wg.Add(1) go hello() wg.Wait() fmt.Println(\u0026#34;main function\u0026#34;) } You create something called a WaitGroup. In most cases we deal with Add, Done and Wait methods of the WaitGroup. Out of which Add and Wait are called from the main goroutine, but Done is called from inside the goroutine\u0026hellip; when they are done executing.\nIf you are having trouble understanding the concept of WaitGroups, you can imagine a playground for children. There is a guard waiting at the main entry. The playground represents the main goroutine, and each kid who enters the playground represents a goroutine. If there is no waitgroup, the guard will close the main entry in the evening and go to home at the end of the day.\nBut what happens if everytime a kids enters, it does a WaitGroup.Add(1) and before entring the ground and does a WaitGroup.Done() when done their work at the ground? This way guard has track of how many goroutines are running inside the playground.\nYou can take the same analogy in the program. We do wg.Add(1) to represent 1 go routine. In this case it is hello. Then we say the program to Wait until the goroutine is Done.\n1 2 Hello world goroutine main function Moreover, the parameter to wg.Add could be the size of goroutine worker you want to reserve for the execution.\nPattern 2: Mutex # Mutex is short for mutual exclusion. To understand mutex, first let\u0026rsquo;s go through an example. This will make mutexes easier to understand.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 package main import ( \u0026#34;fmt\u0026#34; \u0026#34;sync\u0026#34; ) var x = 0 func increment(wg *sync.WaitGroup) { x = x + 1 wg.Done() } func main() { var w sync.WaitGroup for i := 0; i \u0026lt; 1000; i++ { w.Add(1) go increment(\u0026amp;w) } w.Wait() fmt.Println(\u0026#34;final value of x\u0026#34;, x) } You might be expecting the value of x to be 1000, but that barely happens however much time you try it. Although the number of goroutines launched is 1000 times, can you guess why the increment is less than 1000? Continue reading to find out.\n1 final value of x 961 Suppose we are in middle of the for loop and the value of x is 100. A goroutine is launched to increment x. At the same time another goroutine is launched at the same time for same reason. Before the first goroutine can finish the increment to 101, another one comes and increment 100 to 101. 2 goroutines are exausted to increment the number just by 1. The thing happening between these two goroutines is known as race condition.\nSo what can we do to overcome this situation? We are going to use Mutex from the sync package.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 package main import ( \u0026#34;fmt\u0026#34; \u0026#34;sync\u0026#34; ) var x = 0 func increment(wg *sync.WaitGroup, m *sync.Mutex) { m.Lock() // critical section starts x = x + 1 m.Unlock() // critical section ends wg.Done() } func main() { var w sync.WaitGroup var m sync.Mutex for i := 0; i \u0026lt; 1000; i++ { w.Add(1) go increment(\u0026amp;w, \u0026amp;m) } w.Wait() fmt.Println(\u0026#34;final value of x\u0026#34;, x) } m.Lock prevents any other goroutine to access the code between m.Unlock. Meaning that only one goroutine can process that part of code. If you run this code, you\u0026rsquo;d always get 1000 as answer.\n1 final value of x 1000 There is another concept in concurrent programming called deadlock which we are not going to discuss in this post.\nPattern 3: Channels # Till now, we are able to run routines concurrently to leverage resources, but still there is no communication between them.\nWe use channels in Go to achive this communication. A few things to note about channels:\nA channel is a FIFO queue. Once you put any data on a channel, the channel is blocked for write until data is read. The above statement is not true if the channel is buffered. Now there are two kinds of channels, buffered and unbuffered. If using an unbuffered channel, one need to pull the data from the other end of the queue before something new is put to the queue. Unbuffered channels are blocking (syncronous) by nature. Once you put something into it, some consumer has to pull it out from other end before writing something else into the channel.\nHere is one such example:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 package main import \u0026#34;fmt\u0026#34; func square(i int, c chan int) { sq:= i * i c \u0026lt;- sq } func main() { ints := []int{7, 2, 8, -9, 4, 0} squaresChan := make(chan int) for _, v := range ints { go square(v, squaresChan) squares := \u0026lt;-squaresChan fmt.Println(squares) } } Let\u0026rsquo;s walk through the code now. square is a function which takes an int i and a channel c on which it will return the square of the number. On line 11 we are creating a int array ints. On line 13 we are creating a communication channel which will be used by all the goroutines dealing with squares. On line 16 we execute the goroutine, on next line we accept response from the channel and setting into to the variable squares. On next line finally we print it. Also note that we are doing this as a loop, which you can see on line 15.\n1 2 3 4 5 6 49 4 64 81 16 0 I am not very pro at channels right now as I have not made use of it practically. But let\u0026rsquo;s see if I am able to leverage it in my personal projects.\nConclusion # As I have decided to stick with Go for a longer run, I gotta go learn the concurrency in Go inside out. I\u0026rsquo;ll try to write whatever I have learned in a meaningful way.\nIf you enjoyed this post, feel free to leave feedback and share it with like minded people.\n","date":"27 April 2021","externalUrl":null,"permalink":"/posts/2021/very-basics-of-concurrency-in-go/","section":"Posts","summary":"Introduction # I’ve dabbled over many languages in the past. Let it be PHP, Java, Python, C/C++, JavaScript and now Go (some of them I only know the syntax). With the experience of all the languages, when I switched to Go, I had no pain understand slices, struct, interfaces, relatively less pain understanding pointers (I’m still learning). But in non of those languages, I did concurrent programming. Although in my first company I did some concurrent programming using Python threads, but the threads didn’t share any data between them.\n","title":"Very Basics of Concurrency in Go","type":"posts"},{"content":"","date":"27 April 2021","externalUrl":null,"permalink":"/tags/waitgroup/","section":"Tags","summary":"","title":"Waitgroup","type":"tags"},{"content":"","date":"11 April 2021","externalUrl":null,"permalink":"/tags/git/","section":"Tags","summary":"","title":"Git","type":"tags"},{"content":" Introduction # Git Submodules are the concept related to modularity. One git repository can be added to another as a submodule and maintained separately. Instead of being tightly coupled, it is loosely coupled and is easy to maintain. Suppose you are working with softwareA which depends on libraryA, instead of copy-pasting the libraryA over and over again when a new version of the library is released what we can do is use submodule to make this process DRY and elegant.\nWhile you read this post, take a moment to connect with me on LinkedIn.\n1. Setting Up the Repo Structure # For this example I will we demonstrating this concept with my blogging engine Hugo and the submodule which is a theme for Hugo. I keep theme as a separate module because want to reduce overhead when updating to the new theme version, which will be deleting and adding a whole new version of theme repo.\nIf you are following this tutorial with me, you need to install Hugo on your system. Consult your OS\u0026rsquo;s package manager or if you using Windows, consult getting started guide at Hugo\u0026rsquo;s site.\nInstall Hugo # Easiest method for me to install hugo is to use go get command.\n$ go get -u github.com/gohugoio/hugo go: golang.org/x/net upgrade =\u0026gt; v0.0.0-20200904194848-62affa334b73 go: github.com/hashicorp/golang-lru upgrade =\u0026gt; v0.5.4 go: github.com/spf13/cobra upgrade =\u0026gt; v1.0.0 go: github.com/Azure/azure-pipeline-go upgrade =\u0026gt; v0.2.3 go: golang.org/x/sys upgrade =\u0026gt; v0.0.0-20200905004654-be1d3432aa8f go: github.com/google/wire upgrade =\u0026gt; v0.4.0 go: github.com/yuin/goldmark upgrade =\u0026gt; v1.2.1 go: github.com/pkg/errors upgrade =\u0026gt; v0.9.1 go: github.com/niklasfasching/go-org upgrade =\u0026gt; v1.3.2 go: github.com/bep/golibsass upgrade =\u0026gt; v0.7.0 [...output trimmed...] As I already have Hugo installed, this command will upgrade it to latest version. Command output will differ if you are installing a fresh Hugo.\nCreate a Hugo Site # Now I have the latest version of Hugo. Let\u0026rsquo;s start and create our main/parent git repository for my blog.\n$ hugo new site myblog Congratulations! Your new Hugo site is created in /home/sntshk/repos/myblog. Just a few more steps and you\u0026#39;re ready to go: 1. Download a theme into the same-named folder. Choose a theme from https://themes.gohugo.io/ or create your own with the \u0026#34;hugo new theme \u0026lt;THEMENAME\u0026gt;\u0026#34; command. 2. Perhaps you want to add some content. You can add single files with \u0026#34;hugo new \u0026lt;SECTIONNAME\u0026gt;/\u0026lt;FILENAME\u0026gt;.\u0026lt;FORMAT\u0026gt;\u0026#34;. 3. Start the built-in live server via \u0026#34;hugo server\u0026#34;. Visit https://gohugo.io/ for quickstart guide and full documentation. $ cd myblog $ git init Initialized empty Git repository in /home/sntshk/repos/myblog/.git/ $ git add . $ git status On branch master No commits yet Changes to be committed: (use \u0026#34;git rm --cached \u0026lt;file\u0026gt;...\u0026#34; to unstage) new file: archetypes/default.md new file: config.toml $ git commit -m \u0026#34;Initial commit\u0026#34; [master (root-commit) 0f1d8e5] Initial commit 2 files changed, 24 insertions(+) create mode 100644 archetypes/default.md create mode 100644 config.toml Above I have created a bare Hugo site, entered and initialized that directory as a git repo and did the initial commit. As you can see below, our site has no theme at this point in time.\n$ ls .git/ resources/ content/ layouts/ themes/ config.toml archetypes/ data/ static/ $ ls themes Add a theme # Now instead of picking up a theme and slamming into my themes/ directly (which will obviously work out of the box), I\u0026rsquo;ll add it as submolule.\nThe current theme I\u0026rsquo;m using is called hermit which is available at https://github.com/Track3/hermit. Let\u0026rsquo;s add this theme.\n$ git submodule add https://github.com/Track3/hermit.git themes/hermit Cloning into \u0026#39;/home/sntshk/repos/myblog/themes/hermit\u0026#39;... remote: Enumerating objects: 16, done. remote: Counting objects: 100% (16/16), done. remote: Compressing objects: 100% (11/11), done. remote: Total 787 (delta 1), reused 9 (delta 1), pack-reused 771 Receiving objects: 100% (787/787), 460.32 KiB | 615.00 KiB/s, done. Resolving deltas: 100% (344/344), done. Before we start the local server, we need to configure our config.toml file. For sake of simplicity, we\u0026rsquo;ll use sample theme hermit already comes with.\n$ cp themes/hermit/exampleSite/config.toml . At present the status of the repo is like so:\n$ git status On branch master Changes to be committed: (use \u0026#34;git restore --staged \u0026lt;file\u0026gt;...\u0026#34; to unstage) new file: .gitmodules new file: themes/hermit The .gitmodule file stores all the git modules added to the current repo. At current, it looks like this:\n1 2 3 [submodule \u0026#34;themes/hermit\u0026#34;] path = themes/hermit url = https://github.com/Track3/hermit.git It start with a section about the submodule.path is relative to the root, url tracks the URL for future updates. Also you\u0026rsquo;ll see themes/hermit is now listed as file and not a subdirectory.\nIt\u0026rsquo;s time to commit the changes.\n$ git commit -m \u0026#34;Add hermit theme\u0026#34; [master 651c044] Add hermit theme 2 files changed, 4 insertions(+) create mode 100644 .gitmodules create mode 160000 themes/hermi I will start the server and head over to http://127.0.0.1:1313 which is the default address for local prototyping in Hugo.\n$ hugo server- D Start building sites … | EN -------------------+----- Pages | 10 Paginator pages | 0 Non-page files | 0 Static files | 10 Processed images | 0 Aliases | 0 Sitemaps | 1 Cleaned | 0 Built in 56 ms... [...output trimmed...] Hermit Splash Screen 2. Working with Submodules # At this stage, you can keep adding new post (or making changes) to the main repository. Let\u0026rsquo;s make a few post to demonstrate this:\n$ hugo new posts/my-first-post.md /home/sntshk/repos/myblog/content/posts/my-first-post.md created $ hugo new posts/my-second-post.md /home/sntshk/repos/myblog/content/posts/my-second-post.md created $ echo Hello world \u0026gt;\u0026gt; content/posts/my-first-post.md $ echo Hello Mars \u0026gt;\u0026gt; content/posts/my-first-post.md My git log tree looks like this:\n$ git log * 61087cf - (HEAD -\u0026gt; master) Add a few posts (63 seconds ago) \u0026lt;Santosh Kumar\u0026gt; * b5a80a7 - Update config.toml for hermit (5 minutes ago) \u0026lt;Santosh Kumar\u0026gt; * 651c044 - Add hermit theme (7 minutes ago) \u0026lt;Santosh Kumar\u0026gt; * 0f1d8e5 - Initial commit (45 minutes ago) \u0026lt;Santosh Kumar\u0026gt; Submodules are tracked by the exact commit specified in the parent project, not a branch, a ref, or any other symbolic reference.\nWe have two repository now, the main repository which is maintained by you and hermit theme repository. At this stage the main repository can be pushed to remote, say at github.com/santosh/myblog.\nMostly you\u0026rsquo;ll be working with the main repo doing commits. Submodule resides in the repo and only get updated when you want to. If you add commits to a submodule, the parent project won\u0026rsquo;t know. You have to inform it manually.\nRemoving a Submodule # My website at present is in similar state as shown in the log above. Just that I have many blog post made and each commit represent a post made. Now I want to change my theme to something else. git doesn\u0026rsquo;t comes with a magic command to replace theme in my situation. We need to first delete the submodule and then add it back. Let\u0026rsquo;s get started. We\u0026rsquo;ll divide the process into 4 parts.\nDelete relevant files from the .gitmodules file. At present, my files .gitmodules looks like this:\n1 2 3 4 5 6 [submodule \u0026#34;themes/hermit\u0026#34;] path = themes/hermit url = https://github.com/Track3/hermit.git [submodule \u0026#34;themes/hugo-shortcodes\u0026#34;] path = themes/hugo-shortcodes url = https://github.com/santosh/hugo-shortcodes.git I\u0026rsquo;ll go ahead and delete the first 3 lines.\nDelete the relevant section from .git/config. While .gitmodules is a file which stores information about modules and should be commited to the repo. There is one other location where this data is stored i.e. .git/config file. This is all your local configuration of your repo.\nI removed these lines from the config file.\n1 2 3 [submodule \u0026#34;themes/hermit\u0026#34;] url = https://github.com/Track3/hermit.git active = true Run git rm --cached themes/hermit This command will remove the themes/hermit path from the git tracking. Be sure you don\u0026rsquo;t have a trailing slash in the module path when executing this command.\nDelete the now untracked submodule and commit the changes. This is half the way of my journey to replace my theme. I will delete my themes/hermit directory and commit the changes.\nAdding a Submodule # Installation process is same as described early in the tutorial. I\u0026rsquo;ll add Zzo theme now.\ngit submodule add https://github.com/zzossig/hugo-theme-zzo.git themes/zzo git submodule update --remote --merge The above submodule update command is run for all the submodule in the repo, which is kinda overkill in this situation, but it updates all the existing submodules to their latest.\nAfter some themes specific settings, my site is now ready to charm in the new theme.\nConclusion # Today we\u0026rsquo;ve got familiar with some git submodule stuff. If you like this post, you may want to subscribe to my newsletter.\nRelated Readings:\nhttps://www.atlassian.com/git/articles/core-concept-workflows-and-tips https://git-scm.com/book/en/v2/Git-Tools-Submodules https://www.vogella.com/tutorials/GitSubmodules/article.html ","date":"11 April 2021","externalUrl":null,"permalink":"/posts/2021/git-submodules-basics-and-how-to-replace-one-submodule-with-another/","section":"Posts","summary":"Introduction # Git Submodules are the concept related to modularity. One git repository can be added to another as a submodule and maintained separately. Instead of being tightly coupled, it is loosely coupled and is easy to maintain. Suppose you are working with softwareA which depends on libraryA, instead of copy-pasting the libraryA over and over again when a new version of the library is released what we can do is use submodule to make this process DRY and elegant.\n","title":"Git Submodules Basics and How to Replace One Submodule With Another","type":"posts"},{"content":"","date":"11 April 2021","externalUrl":null,"permalink":"/tags/hugo/","section":"Tags","summary":"","title":"Hugo","type":"tags"},{"content":"","date":"11 April 2021","externalUrl":null,"permalink":"/tags/modularity/","section":"Tags","summary":"","title":"Modularity","type":"tags"},{"content":" Introduction # In 2014, Google hinted that they are running tests taking into account whether sites use secure, encrypted connections as a signal in our search ranking algorithms. This means it will affect SEO in some way. Today in the age of HTTP2, https is default. Default in the sense that many server software will only allow HTTP2 when https is configured.\nThe motivation for this post came when I wanted to enable Google Analytics on my site which is under construction. At very least I will be able to get some analytics information.\nI will be using Amazon Linux 2 which seems to be made on top of CentOS, but certainly is not CentOS.\nWhile you read this post, take a moment to connect with me on LinkedIn.\nInstallation # In this section, I\u0026rsquo;ll guide you through the installation process of nginx. If you already have an instance of nginx running, you can skip to the configuration steps. Second thing to install is the certbot.\nI would enter following command in the terminal to install nginx as well as certbot:\nsudo yum install nginx certbot-nginx certbot is used for generating TLS certification for our website.\nFirst thing to do after installing nginx is to start and enable the service.\nsudo systemctl enable nginx sudo systemctl start nginx With this done, we can proceed to the configuration stage.\nConfiguration # First things first, let\u0026rsquo;s see if our side works without HTTPS.\nnginx # My /etc/nginx/nginx.conf is almost identical to the default instance. The only difference is in the server block where I have removed the instance of default_server.\n1 2 3 4 server { listen 80 default_server; listen [::]:80 default_server; server_name _; to\n1 2 3 4 server { listen 80; listen [::]:80; server_name _; This is for the main config. Now I choose to use separate configs for each of my domain. For that, conf.d subdirectory can be used to place configuration files.\nThis is my santosh.pictures.conf:\nserver { listen 80; listen [::]:80; root /var/www/html; server_name santosh.pictures; location / { proxy_pass http://localhost:4500; } } Thing to note here is that I have a Go application running on port 4500, and the domain I own is santosh.pictures. The above application has my gtag being hosted in an index.html, and the text Website under construction.\nAfter configuring this, you can reload the configuration.\nsudo systemctl reload nginx With this, I have just configured reverse proxy for my application. My site right now appear like this.\nsantosh.pictures without https Now I am going ahead to configure certbot.\ncertbot # Installing certbot-nginx also installs certbot. certbot-nginx makes our work by automating the renewal of certificate automatically.\nTo get started with cerbot, you have nothing much to do. This is a very automated process. You will be asked for some input. Here is how my prompt looked like when configuring the bot.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 ec2-user at ip-172-31-45-160 in /efs/repos $ sudo certbot Saving debug log to /var/log/letsencrypt/letsencrypt.log Plugins selected: Authenticator nginx, Installer nginx Enter email address (used for urgent renewal and security notices) (Enter \u0026#39;c\u0026#39; to cancel): sntshkmr60@gmail.com - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Please read the Terms of Service at https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf. You must agree in order to register with the ACME server. Do you agree? - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (Y)es/(N)o: yes - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Would you be willing, once your first certificate is successfully issued, to share your email address with the Electronic Frontier Foundation, a founding partner of the Let\u0026#39;s Encrypt project and the non-profit organization that develops Certbot? We\u0026#39;d like to send you email about our work encrypting the web, EFF news, campaigns, and ways to support digital freedom. - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - (Y)es/(N)o: n Account registered. Which names would you like to activate HTTPS for? - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 1: santosh.pictures - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Select the appropriate numbers separated by commas and/or spaces, or leave input blank to select all options shown (Enter \u0026#39;c\u0026#39; to cancel): 1 Requesting a certificate for santosh.pictures Performing the following challenges: http-01 challenge for santosh.pictures Waiting for verification... Cleaning up challenges Deploying Certificate to VirtualHost /etc/nginx/conf.d/santosh.pictures.conf Redirecting all traffic on port 80 to ssl in /etc/nginx/conf.d/santosh.pictures.conf - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Congratulations! You have successfully enabled https://santosh.pictures - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - IMPORTANT NOTES: - Congratulations! Your certificate and chain have been saved at: /etc/letsencrypt/live/santosh.pictures/fullchain.pem Your key file has been saved at: /etc/letsencrypt/live/santosh.pictures/privkey.pem Your certificate will expire on 2021-07-10. To obtain a new or tweaked version of this certificate in the future, simply run certbot again with the \u0026#34;certonly\u0026#34; option. To non-interactively renew *all* of your certificates, run \u0026#34;certbot renew\u0026#34; - If you like Certbot, please consider supporting our work by: Donating to ISRG / Let\u0026#39;s Encrypt: https://letsencrypt.org/donate Donating to EFF: https://eff.org/donate-le This is going to change my santosh.pictures.conf to something like this:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 server { root /var/www/html; server_name santosh.pictures; location / { proxy_pass http://localhost:4500; } listen [::]:443 ssl ipv6only=on; # managed by Certbot listen 443 ssl; # managed by Certbot ssl_certificate /etc/letsencrypt/live/santosh.pictures/fullchain.pem; # managed by Certbot ssl_certificate_key /etc/letsencrypt/live/santosh.pictures/privkey.pem; # managed by Certbot include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot } server { if ($host = santosh.pictures) { return 301 https://$host$request_uri; } # managed by Certbot listen 80; listen [::]:80; server_name santosh.pictures; return 404; # managed by Certbot } Once every change is done. You can reload the nginx service with systemctl reload nginx.\nAt this point, my address bar looks like this.\nsantosh.pictures with secure http Conclusion # HTTPS is the new HTTP nowadays already. For next time, I have thought to cover this nginx thing with either Terraform or Docker. Feel free to share it with your network if you found it useful.\nIf you are looking to enable HTTPS for all of your subdomains, consider reading Wildcard Domain Certificate Using Route53 and Let\u0026rsquo;s Encrypt. You can use other method of authentication instead of Route53 if you are using other provider than Route53.\nIf you liked this post, please consider sharing it with your network and please subscribe to the newsletter for new updates.\n","date":"11 April 2021","externalUrl":null,"permalink":"/posts/2021/how-i-enabled-https-on-my-ec2-hosted-website/","section":"Posts","summary":"Introduction # In 2014, Google hinted that they are running tests taking into account whether sites use secure, encrypted connections as a signal in our search ranking algorithms. This means it will affect SEO in some way. Today in the age of HTTP2, https is default. Default in the sense that many server software will only allow HTTP2 when https is configured.\n","title":"How I Enabled HTTPS on My EC2 Hosted Website","type":"posts"},{"content":"","date":"11 April 2021","externalUrl":null,"permalink":"/tags/secure/","section":"Tags","summary":"","title":"Secure","type":"tags"},{"content":" Introduction # This is a short post about inheritence in JavaScript.\nWhile you read this post, take a moment to connect with me on LinkedIn.\nBody # Not until ECMAScript 2015 had class syntax. Before that there was prototype inheritance\nSuppose there is a function:\n1 2 3 4 function FullName() { this.firstName = x; this.lastName = y; } Now when you want to return the fullname, you can either define a function inside the FullName function only, like this;\n1 2 3 4 5 6 7 function FullName() { this.firstName = x; this.lastName = y; this.fullName = function() { return this.firstName + \u0026#39; \u0026#39; + this.lastName; } } This is obviously correct, but there is an overhead that every time the FullName entity is created, it creates the fullName function which kind of wastage of space. What if there was the main place where there was a fullName function defined? And all runtime variables would be defined to new objects?\nThis is where the prototype comes in. Below is how we would define a fullName prototype to the FullName entity.\n1 2 3 FullName.prototype.fullName = function() { return this.firstName + \u0026#39; \u0026#39; + this.lastName; } Inheritence with protoypes # When you want to inherit from the FullName object. This is what you would do:\n1 2 3 function BigName() { FullName.call(this); } What this tells the interpreter is that runs the constructor of the FullName.\nWrong way for inheritance:\n1 FullName.prototype = BigName.prototype This will overwrite the FullName.\nCorrect way to do this:\n1 FullName.prototype = Object.create(BigName.prototype) Key takeaways: # Everything in JavaScript is an object. Just like Python.\nRelated readings: # https://developer.mozilla.org/en-US/docs/Web/JavaScript/Inheritance_and_the_prototype_chain https://javascript.info/prototype-inheritance ","date":"9 March 2021","externalUrl":null,"permalink":"/posts/2021/inheritence-in-javascript-with-prototypes/","section":"Posts","summary":"Introduction # This is a short post about inheritence in JavaScript.\nWhile you read this post, take a moment to connect with me on LinkedIn.\nBody # Not until ECMAScript 2015 had class syntax. Before that there was prototype inheritance\nSuppose there is a function:\n1 2 3 4 function FullName() { this.firstName = x; this.lastName = y; } Now when you want to return the fullname, you can either define a function inside the FullName function only, like this;\n","title":"Inheritence in JavaScript with Prototypes","type":"posts"},{"content":"","date":"9 March 2021","externalUrl":null,"permalink":"/tags/javascript/","section":"Tags","summary":"","title":"JavaScript","type":"tags"},{"content":" Introduction # Each year I post about what have I done in that particular year, inclining it more toward career. This is to document my process of learning over the year.\nI know I\u0026rsquo;m too late for a year in review post, but every time I wanted to post, there was something more high priority coming up. Today I have somehow managed to finish creating a featured image and make an end to the post.\nWithout further ado, let's continue. The Year 2020 # So what was new in the year 2020?\nCourses completed # Bits and Bytes of Computer Networking The reason for learned was rooted from a Docker support channel on Slack. I went there to ask for help for the issue I was debugging. I was told by a gentleman to have a basic understanding of Networking. After doing this course I feel confident about Docker now. I know this course doesn\u0026rsquo;t teaches you everything, but at least it gets you started. I plan to follow teachyourselfcs networking curriculum when I find it necessary.\nThe course starts with an introduction to OSI model, then goes into each layer of the model explaining the protocol and application used in that layer. How a packet propogates from one host to another. What metadata each packet holds. Then the course talks about Networking Services in which it talks about DNS, DHCP, NAT, VPN and proxy servers. It then talks about how we connect to internet with wired and wireless connections. What wired and wireless protocols are available and all. At last it talks about commands which can be used to debug networking issues on Linux and Windows.\nOperating Systems and You: Becoming a Power User This is a course by Google, delivered on Coursera just like the above course. Although I knew most of the stuff taught in Linux, this course gave me exposure to do the same stuff in Windows. This course prepares you for system administration job. Things which are covered are Users and Permissions management, Package and Software Management, Filesystems, and Process Management. I won\u0026rsquo;t they doing this course will get you a job as a Systems Engineer.. but yeah, if you want to get into it, this is a good start.\nGophercises Early in 2020 I decided to learn a new language and chose Go, because: a) It is good for backend. b) It is a good transition from Python. c) Many companies are already using it. Gophercises was one of the resource I was using. Other ones being Learn Go with Tests and rest of the internet. Go brings the easiness of Python and Robustness of C together.\nJon Calhoun believes in learning by doing. Gophercises comes with small exercises which deal with certain problem domain. It gives good exposure to language features that you will encounter later. Here is my code which I was working on: https://github.com/santosh/gophercises\nJavaScript30 This is comparable to Gophercises in Go. It teaches by making us doing. It cover all new ES6 and talks new methods of doing things in modern JavaScript. One nice thing about this series is that it encourages you to se Vanilla JavaScript. Contents are, but not limited to topics such as DOM, CSS, audio, video, harware access.\nFirst Year in Code This book is comparable to The Pragmatic Programmer: From Journeyman to Master, a book I\u0026rsquo;m continuing reading this year. It\u0026rsquo;s just that First Year in Code is available for free. I find some chapters in that book relatable to me.\nTechniques and Lesson learned # Exponential Backoff While learning about and designing APIs I dwelled into exponential Backoff. Exponential fallback is when you hit an API and for some reason the server wasn\u0026rsquo;t able to fulfill the request. Now the next time you hit the API the server will make you wait for double the time of last time. Say I hit an API, it failed, I will have to hit the API after 1 second, next time 2, then 4, then 8 and so.\nwaiting time increases with a factor of 2 Cache Servers and Eventual Consistency Back when I was a teenage, there already was concept like CDN. I use to download it using CDN only, there was no Node.js back then.\nNow at this stage, I know the bigger picture, and how is it related with other development ecosystem. With cache servers we can reduce the load from our application server. There are concepts like cache hit and cache miss. User might be reading an older data, so data invalidation is important. I have seen eventual consistency when working with Twitter\u0026rsquo;s API, where the number of likes are not consistent between API response and web UI.\nDistributed Computing In distributed computing architecture, the systems work by passing message to each other. These are the more practicle architecture where systems are located at more needy locations than being located on one place. Say your traffic is coming from Japan, then it\u0026rsquo;s better to keep your infrastructure there instead of US.\nThere is a theorem called CAP theorem which says that a distributed database system can only guarantee two out of these three characteristics: Consistency, Availability, and Partition Tolerance:\nA system is said to be consistent if all nodes see the same data at the same time. Availability in a distributed system ensures that the system remains operational 100% of the time. Every request gets a (non-error) response regardless of the individual state of a node. This condition states that the system does not fail, regardless of if messages are dropped or delayed between nodes in a system. Graceful Degradation Design your system in such a way that not a single failure can take down the entire system. In other words, it is an ability to maintain limited functionality even when a large portion of i has been destroyed or inoperative. You can read more about this topic on Wikipedia: https://en.wikipedia.org/wiki/Fault_tolerance\nDocumentations, not videos Back in time when I started ABCD of programming in school, I had not so fast internet connection, YouTube was not possible back then for learning. There were only blog posts and documentation. When I went to Delhi for Animation studies, that was the time internet was in abundance. From there I started a habit of rich internet resources like videos and all.\nNow after a long time ago, I feel like documentation are the primitive stuff and are not going anywhere. Now I am again going back to the documentation and specs level. Specs tell the things which are not covered by tutorials. I have read about AMQ protocol and HTTP protocols (both HTTP/1.1 and HTTP2). Inspiration for this came from a failed interview where I was not able to properly authentication mechanism to a CLI application. And there was no clear tutorial for that all across the internet.\nThe Year 2021 # Started and Ongoing # Following are the courses, books and topics for year 2021.\nAWS Certified Developer Started last year this is a course available on Udemy as Ultimate AWS Certified Developer Associate, still continuing on this in 2021. Shall be completing this by 2nd quarter of this year. At the time of writing this paragraph, I have completed 20/30 sections.\nThis is not the end, as I\u0026rsquo;m also planning to do a few mock test and apply for a AWS Certified Developer.\nteachyourselfcs.com Although I know a little bit about every topic in this collection, it\u0026rsquo;s better to follow them thoroughly so I can communicate with Engineering graduates better. This is the main content which I\u0026rsquo;ll be following for coming years. I have created a group on GitLab: https://gitlab.com/teachingmyselfcs. Although I\u0026rsquo;ll be following the main content recommended on the site listing, but suppose if there is any YouTube playlist I am following, it will go as a submodule to the course.\nBooks # The Pragmatic Programmer: From Journeyman to Master I have completed almost 50% of the book at the given moment. This book is full of references that will make you a good programmer. The way you write code will change (if not already). The author is very good at taking anologies from environment and teaching the concepts. I will write a blog post with the review of this book when it is complete.\nRich Dad Poor Dad I started reading this book when I was in college. I have learned about buying assets, not liability. More chapters to go.\nSoft Skills: The Software Developer\u0026rsquo;s Life Manual This book and the next book in this listing are the two books I would suggest any software developer who I meet in life. While The Pragmatic Programmer is focused toward software development only, Soft Skills covers the entire ecosystem related to software development. This book itself is written by John Sonmez who himself was a software developer. Among all of the topics, I like his idea about starting a blog and teaching people.\nThe thing is, when you teach some topic to other person. You first have to settle that concept into your life by keeping a reference of life event (of your choice) and the concept. It\u0026rsquo;s like one-to-one relationship or one-to-many relationship. The next step in teaching someone a concept is to understand what other person might have experienced in life which matches to your life. You could be having in-depth knowledge of a topic. But you can\u0026rsquo;t explain it to anyone, maybe you should work on interpersonal communications.\nComputer Systems: A Programmer\u0026rsquo;s Perpective This book about computer architecture is more inclined towards programmers. The point to note here is you may or may not have Computer Science background to read this book. In engineering colleges you read computer architecture books which are more inclined towards hardware production, this book strips that stuffs and only teaches you programming specific things. This book is part of teachyourselfcs curriculum.\nSoft Skills # Time Management # Affected by John Sonmez content, I started to become a finisher, meaning that I had to finish the work I have started or either choose not to start the work itself. I also learned that I need to have a proper routine, or a semi-flexible one. I also needed to track the progress of my work.\nSomewhere in the mid of the year, I started using Google Calender. Some of my routine work were migrated to Google Calender. From doing laundry to having a drink monthly, everything had a recurring reminder. And for non-recurring things as well. I learned how this systematic organisation can reduce stress.\nSomehow I ended up in a company where Personal Software Process was enforced. Well, I wasn\u0026rsquo;t able to retain the position in the company, but it was good experience knowing that to analyse the time, you first need to track it. Inspired from this time management thing I started using Trello. The action items that I need to do in 2021 are already laid out in my Kanban board.\nCommunication # This one new thing I learned in my life. It is not important that people must also be knowing or have experienced things that I have did. Represent to people what they can understand. Raise the level of curiosity in them.\nFew more communication tricks I learned was knowing what to communicate, this will help you stay on the topic and won\u0026rsquo;t make you appear boring to the other party. One more thing is to ask yourself, who is intended audience of the message.\nIf you are looking for a good resource to improve your communication skills, with a pinch of science is MasterClass by Neil deGrasse Tyson who teaches Scientific Thinking and Communication.\nConclusion # I would like to end this post with a shout out to some people which I\u0026rsquo;ve met recently in life. I would like to thank Mayur Amminbhavi for believing in me and approaching me with only 8 months of professional development experience and giving me such exposure to a multinational company. I followed my passion to become Fullstack Developer today. I aim to go in some tech giant in near future.\nI would also like to thank Prasanna Kumar for being such a great companion. Lokesh Vishwakarma, who was in the same team in two consecutive companies.\nIn the new team I\u0026rsquo;m working with Abhishek Gupta and Maan Singh who are very kind for a newcomer like me in fullstack development.\nWrapping this year on a positive note, I really hope to learn even more next year. Here’s hoping for a great 2021 for everyone!\n","date":"15 January 2021","externalUrl":null,"permalink":"/posts/2021/2020-in-review/","section":"Posts","summary":"Introduction # Each year I post about what have I done in that particular year, inclining it more toward career. This is to document my process of learning over the year.\nI know I’m too late for a year in review post, but every time I wanted to post, there was something more high priority coming up. Today I have somehow managed to finish creating a featured image and make an end to the post.\n","title":"2020 in Review: Soft Skills, Books, Cloud and SE Practices","type":"posts"},{"content":"","date":"19 December 2020","externalUrl":null,"permalink":"/tags/asynchronous/","section":"Tags","summary":"","title":"Asynchronous","type":"tags"},{"content":" Introduction # Promise is a language feature for asynchronous programming in JavaScript.\nAs JavaScript is single threaded, if you are doing something which waits for I/O, you create a Promise for that task. This task could be waiting for an API call from Twitter for example, or reading a file from a filesystem), and you let it run without getting in the way of normal execution flow.\nThe Promise at a later time either gets fulfilled (successful), also known as resolved in JS terminology or gets failed, also known as rejected in JS terminology.\nLet us understand it with an example. Below function returns a Promise.\n1 2 3 4 5 6 7 8 9 10 11 function doSomething(msg) { return new Promise(=\u0026gt; (resolve, reject) { // do a thing, possibly async, then… if (/* everything turned out fine */) { resolve(\u0026#34;It worked!\u0026#34;); } else { reject(\u0026#34;It failed\u0026#34;); } }) Now Promises have some methods attached to them which are useful when you want to run a block of code (callback function) when Promise either gets resolved or rejected. In code we would write something like this:\n1 2 3 4 doSomething(\u0026#34;1st Call\u0026#34;) .then(() =\u0026gt; doSomething(\u0026#34;2nd Call\u0026#34;)) .then(() =\u0026gt; doSomething(\u0026#34;3rd Call\u0026#34;)) .catch(err =\u0026gt; console.error(err.message)); Although you can have a catch at an intermediate position, one should keep it at last because it will show all the rejections at one same place, if there are more than one.\nIf you liked this post, please consider sharing it with your network and subscribe to the newsletter below.\n","date":"19 December 2020","externalUrl":null,"permalink":"/posts/2020/beginners-introduction-to-promises-in-javascript/","section":"Posts","summary":"Introduction # Promise is a language feature for asynchronous programming in JavaScript.\nAs JavaScript is single threaded, if you are doing something which waits for I/O, you create a Promise for that task. This task could be waiting for an API call from Twitter for example, or reading a file from a filesystem), and you let it run without getting in the way of normal execution flow.\n","title":"Beginner's Introduction to Promises in JavaScript","type":"posts"},{"content":"","date":"10 October 2020","externalUrl":null,"permalink":"/tags/benchmarking/","section":"Tags","summary":"","title":"Benchmarking","type":"tags"},{"content":"","date":"10 October 2020","externalUrl":null,"permalink":"/tags/data-structure/","section":"Tags","summary":"","title":"Data Structure","type":"tags"},{"content":" Introduction # Like most self-taught programmers, I spent a lot of my years dabbling with different technologies. But there comes a time in every programmer\u0026rsquo;s life when they have to learn data structures and algorithms to proceed in their careers. In this post, I will go through the basics of data structures, what purpose they serve, and what\u0026rsquo;s common between all of them.\nA data structure is a particular way of organizing real-life data in a computer so that it can be used effectively.\nJob blocked by Data Structures \u0026amp; Algorithm To understand data structures, we must first understand data. In very programming language we have some data types. In some programming languages, we have to explicitly express the data type by using keywords like int, float, bool, char etc. Even in scripting languages like Python and JavaScript we have those, but they are hidden from the programmer\u0026rsquo;s eyes. The interpreter handles them for us, and this behavior is counted in feature of the language.\nPrimitive Data Types and Memory # We must understand that for every data type the operating system allocated a specific amount of memory in RAM. Unlike our decimal number system, in which ones place can hold 10 distinct values (0 to 9), in the binary number system, ones place can only hold only 2 distinct values (0 and 1). One binary digit is called a bit (binary + digit). We store 8 bits together and call it a byte.\nHow binary is calculated from r/gif Because a bit can hold 2 values, 0 or 1, you can calculate the number of possible values by calculating 2^n where n is the number of bits. Given 8 bits per byte, a short integer which is allocated 2 bytes can store 2^16 (65,536) possible 0 and 1 combinations. If we split those between negative and positive values, the data range for a short is -32,768 to +32,767.\nYou can easily find how much a primitive data type takes in your favorite language, for Go, here is a document with such table: https://www.tutorialspoint.com/go/go_data_types.htm\nData Structure and Why We Study Them # Data structures are different from the primitive data types, but they are similar to them in the sense that it holds (structures) same data into certain predefined manners.\nWhen we study data structure and algorithm, we are talking (and concern) about space (RAM) and processing (CPU) efficiency. Both processing power and memory are finite. So we need to keep things as optimized so that it takes least amount of space and processing time. You won\u0026rsquo;t want your Lambda function to be running forever, would you? You would want it to take the least amount of time and take least amount of space, right?\nTypes of Data Structures # Our RAM is basically cells of memory as seen in below image.\nCells of memory in RAM Linear Data Structures # Array # An array is a fundamentally a list of similar values. Elements of an array are stored in contagious memory blocks. All the elements have to have same data types. Arrays have a fixed length. Array as we see and in RAM As you can see in the image, there is a section of RAM where all the data is stored in contagious memory block.\nSo Python list is not truly an array, but regarded as one. Dealing with similar value is always faster, that\u0026rsquo;s the reason you must have seen numpy using similar data type for defining new array.\nLinked List # Unlike arrays, a linked list is a data structure, in which the elements are not stored at contiguous memory locations. In a linked list data structure, we generally have a payload variable, one or two pointer variables that point to the next element in the list. Being not stored in contagious memory, it gives flexibility to allocated memory anywhere in RAM. Which is beneficial and can leverage available space smartly than arrays. Variations of Linked list are singly liked list, doubly linked list, circular linked list. Singly linked list only has one pointer to the next element but the doubly linked list has a pointer to the next as well as the previous element. Circular linked list are used when scheduling jobs for the processor, if one job is waiting for IO or network resource, the next jobs comes in. graphical representation of a singly linked list Now as you have seen in the case of arrays, each element of a linked list can be anywhere in the RAM.\nThere is a downside to too, you can\u0026rsquo;t access any item straight away in linked list as you would in an array.\nQueue # Queue is a data structure which can be related to a queue of students in a morning assembly. If entire queue has to go to the class, the first student in the queue has to move first; then follows the other students. Similarly, in queue, we can add data to the last of the queue and remove from the begining. Queues are also called FIFO (First In First Out) data structure. As seen from the perspective of element (data), the data first gets in, then the first data gets out. There have been software written just to manage task queues, which are highly used in distributed computing. RabbitMQ is one such example I have seen in wild. Stack # Does StackOverflow or stacktraces strike something in your mind? Yes, they are both related to same stack data structure.\nStack can be related to a stack of plates, to first add something to a stack you put the element at the top, when removing, you remove from the top. If you have to access the n-1 element, you have to first access the all the intermediate elements.\nNon-Linear Data Structures # Graph # Heard about Graph Theory from discrete mathematics? Yes, they are the same. If you haven\u0026rsquo;t, go watch some youtube videos. A graph consists of nodes and vertices. There is a plethora of algorithms related to this data structures which I will discuss in upcoming posts. Below is one of the first videos I watched on Graph Theory.\nTree # Tree is a kind of a graph. There are many variations of particular data structure, like there multiple kinds of trees, including binary tree, AVL tree, red-black tree etc. graphical representation of a treet This is by no means an exhaustive list of data structures, some data structures can be created by mixing them together, hash map is an example of one such data structures which is known to be very efficient. On top of that, some data structures can be implemented by using other data types e.g. stacks and queues can be implemented by using array, or a linked list.\nEvery data structure has their own significance, and there is best one.\nIn this post I\u0026rsquo;ll not dig into manipulating these data structures and this will be reserved for some other post in future in which I\u0026rsquo;ll cover algorithms.\nOperations of Data Structures # There are some generic operations we could do on a number of data structures. Let\u0026rsquo;s look at them one by one.\nAdd/Insert/Push/Enqueue\nWe can add a new item to an array of predefined length, we simply call them inserting. Push terminology is generally used when we add to the last of a data structure, like in stacks.\nThere can be multiple variations of the insert: add, addFirst, addLast, addByIndex\nVariations depend upon the nature of data structure. Like, you may insert at first of a linked list or array, but you may not in a stack.\nGet/Access\nVariations can include: getByIndex. You can\u0026rsquo;t get by index in a linked list. Search/Traverse\nSearching could be different depending on the algorithm we use. We can search an array in a linear fashion, but we can also use something called binary search where we eliminate half of the array by bisecting the array depending on the value we are searching for.\nPrerequisite for binary search is the data structures has to be sorted.\nRemove/Delete/Pop/Dequeue\nWhen we remove an element from last of an structure, we call it popping. In case of queues, it\u0026rsquo;s called dequeue.\nVariations may include: deleteFirst, deleteLast, deleteByIndex\nModify/Edit/Update\nThere is certain time taken by each of the operations, which I will discuss in a post where I discuss algorithms. Where to Learn and Practice Data Structures # I am by no means a guru of DSA, but I have recently come up across this resource by freeCoceCamp.\nFinally, this is not directly related to DSA, but having a Math background is always good and helps us in analytical thinking. For sake of that, if you don\u0026rsquo;t have a mathematical background, I would recommend solving problems from Project Euler. Going through the process of solving the problem will force you to look for a better optimized alternative which can only be done by researching on the topic.\nConclusion # If you found this post informative, please feel free to follow me on Twitter for related updates and send me a connection request on LinkedIn.\nIf you found something missing, or wrong, please let me know in comments so that everybody stays on the same page.\n","date":"10 October 2020","externalUrl":null,"permalink":"/posts/2020/data-structures-101/","section":"Posts","summary":"Introduction # Like most self-taught programmers, I spent a lot of my years dabbling with different technologies. But there comes a time in every programmer’s life when they have to learn data structures and algorithms to proceed in their careers. In this post, I will go through the basics of data structures, what purpose they serve, and what’s common between all of them.\n","title":"Data Structures 101","type":"posts"},{"content":"","date":"10 October 2020","externalUrl":null,"permalink":"/tags/performance/","section":"Tags","summary":"","title":"Performance","type":"tags"},{"content":" Introduction # The internet is already growing, no need to say that. With that is growing demands, increasing load on existing infrastructure. New, better protocols are defined in every direction, let it be OpenAPI or HTTP/3. It is not limited to HTTP. With time there has been seen a significant boom in messaging protocols like RabbitMQ and similar related services like Amazon SQS. With this post I have tried to collect as much information as I can about\nevolution of HTTP without getting too into any particular part.\nIn meantime, please consider connecting with me on LinkedIn\nIn today\u0026rsquo;s post I will dig and show what changes have been made over time with a little bit of context.\nHypertext Transfer Protocol (HTTP) is an application-layer protocol for transmitting hypermedia documents, such as HTML. It was designed for communication between web browsers and web servers, but it can also be used for other purposes. HTTP follows a classical client-server model, with a client opening a connection to make a request, then waiting until it receives a response.— Mozilla Developer Network Before getting started, for those who don\u0026rsquo;t know, when you send or receive an HTTP request, the packets go through these layers.\nOSI Model This video does a good job of explaining briefly how each layer works.\nAnd HTTP is just the application layer protocol, among tons other protocols in same layer. HTTP is implemented by different HTTP clients and servers.\nHTTP only presumes a reliable transport; any protocol that provides such guarantees can be used.\nWithout further ado, let\u0026rsquo;s get started.\nEarly Days # HTTP was developed by Tim Berners-Lee as you might have heard of. It was build between 1989-1991 and has no version number back then. It was later dubbed as HTTP/0.9.\nBack in the time. HTTP was..\na simple protocol to transfer HTML files from server to client there were no headers, no status codes. Just one TCP connection. The client sent a request, and server sent a response. It was that simple.\nHTTP/0.9 # HTTP/0.9 was developed by Tim Berners-Lee at European Organisation for Nuclear Research in 1989. The initial version of HTTP consist of 4 building blocks:\nHTTP HTML A web browser A web server Yes, it all started at the same time.\nThe first servers wer running out of CERN by early 1991.\nA simple request made by an HTTP client might look like this today:\nGET /index.html HTTP/1.1 But back then, HTTP/0.9 has no HTTP version. The requests used to look like this:\nGET /index.html HTML also had some very basic tags, all using all-caps. Some of the HTML tags of that time are TITLE tag, A (anchor) tag, P (paragraph) tag, H1..H6 tags, UL and OL tags, which are still being used till date. Some of them are either not being used or are replaced by something else, which includes IsIndex tags, PLAINTEXT tags, ADDRESS, HP1/HP2 tags et cetra. You can still find the documentation at CERN website.\nUnlike subsequent evolutions, there were no HTTP headers, meaning that only HTML files could be transmitted, but no other type of documents. There were no status or error codes: in case of a problem, a specific HTML file was send back with the description of the problem contained in it, for human consumption.\nHTTP/1.0 # The first official spec of HTTP came out in 1996 HTTP/1.0. Some of the notable advancement in HTTP version 1.0 include:\nStatus codes were started being used to let client know if request has been failed or successeded. Many more status codes were introduced in later version of HTTP. HTTP headers were introduced, both for the requests and the responses, allowing metadata to be transmitted and making the protocol extremely flexible and extensible. With the help of HTTP headers, this was the first time the ability to transmit other documents than plain HTML files has been added (thanks to the Content-Type header). HTTP method such as GET, HEAD, POST were introduced.\nAround this time only second iteration of HTML was introduced. If you feel deterministic, here is the official spec for this version of HTTP https://tools.ietf.org/html/rfc1945\nHTTP/1.1 # HTTP 1.1 was a proper standardization of HTTP and ran around for around 18 years until HTTP 2 came out.\nIn protocol version 1.1, we were able to make requests without waiting for other to complete. Additional cache control mechanisms were introduced. Servers were more versatile on delivering content on ground such as language and encoding. A Host header field was must in all HTTP/1.1 request messages. A 400 (Bad Request) status code may be sent to any HTTP/1.1 request message that lacks a Host header field. With the availability of Host headear it was possible to host multiple website on same IP adderss. A connection can be reused, saving the time to reopen it numerous times to display the resources embedded into the single original document retrieved. Pipelining has been added, allowing to send a second request before the answer for the first one is fully transmitted, lowering the latency of the communication. Content negotiation, including language, encoding, or type, has been introduced, and allows a client and a server to agree on the most adequate content to exchange. This means if language in your HTTP client is set to Spanish and a server has content in multiple languages, it would return the client the content in appropriate language. Caching was introduced. More on this at https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cache-Control In 2000, a new pattern for using HTTP was designed: representational state transfer (or REST), providing interoperability between computer systems on the internet. The term is intended to evoke an image of how a well-designed Web application behaves.\nTransport Layer Security was introduced around the same time. So was CORS.\nMore about HTTP/1.1 in https://tools.ietf.org/html/rfc2616. But later six-part specification obsoleting RFC 2616 which can be found at the bottom of https://en.wikipedia.org/wiki/Hypertext_Transfer_Protocol#History section.\nHTTP/2 # Officially standardized in May 2015, the new HTTP/2 protocol is faster than HTTP/1.1 and only works over HTTPS. HTTP was already being used by 8.7% of all websites.\nVersion 2 is focused on performance. With version 2, more performance can be yield with the same server. We still use HTTP requests. A single TCP connection is used to send as many request client or the server wants. This is called multiplexing (link to appropriate content.).\nSo if the client sends 3 requests, and the server responds with all 3. How does the client knows which response is for which request? This is where streams come in. Each http packet is tagged with a stream id.\nThe HTTP/2 protocol has several prime differences from the HTTP/1.1 version:\nIt is a binary protocol rather than a textual. It is a multiplexed protocol. Parallel requests can be handled over the same connection, removing the order and blocking constraints of the HTTP/1.x protocol. It compresses headers. As there are often similar amount a set of requests. This removes duplication and overhead of data transmitted. It allows a server to populate data in a client cache, in advance of being required, through a mechanism called server push. At the time of writing this, HTTP/3.0 has come out. But we\u0026rsquo;ll not talk about this as it is out of the scope of this post.\nhttps://developers.google.com/web/fundamentals/performance/http2\nThis site at the time of writing this post is being served to you via HTTP2.\nMore about HTTP/2 at https://httpwg.org/specs/rfc7540.html. For a gentle introduction, see https://www.ssl.com/article/an-introduction-to-http2/.\nHTTP 3.0 - QUIC # Until now, TCP was used as a transport medium for HTTP. In HTTP3 that transport medium is switched to UDP.\nIs still in development and being incorporated by various libraries in different languages.\nAs of now, the latest version of Chrome and Firefox already support HTTP3.\nConclusion # This is not a very exhaustive list or features I can list, but I have provided links wherever I can if you want to dig deeper.\nIf you liked this post, please share with your friends and subscribe below for new updates.\n","date":"19 September 2020","externalUrl":null,"permalink":"/posts/2020/a-brief-evolution-of-http/","section":"Posts","summary":"Introduction # The internet is already growing, no need to say that. With that is growing demands, increasing load on existing infrastructure. New, better protocols are defined in every direction, let it be OpenAPI or HTTP/3. It is not limited to HTTP. With time there has been seen a significant boom in messaging protocols like RabbitMQ and similar related services like Amazon SQS. With this post I have tried to collect as much information as I can about\nevolution of HTTP without getting too into any particular part.\n","title":"A Brief Evolution of HTTP","type":"posts"},{"content":"","date":"19 September 2020","externalUrl":null,"permalink":"/categories/backend/","section":"Categories","summary":"","title":"Backend","type":"categories"},{"content":" Introduction # Are you lost between Handle, HandleFunc, and HandlerFunc? If the answer is yes then this post is for you. To understand these pretty well we need to be familiar with interafces. Although interface is just a keyword, it\u0026rsquo;s confusing at first when you are switching from Python like language; which does not have a similar keyword.\nIn meantime, please consider connecting with me on LinkedIn\nWhat is an Interface? # In golang, we declare an interface like this:\n1 2 3 4 type geometry interface { area() float64 perim() float64 } In above code, the lines inside the curly brackets are the function signatures of two method namely area() and perim(). Whatever type implements those signature becomes a geometry.\nLet\u0026rsquo;s take an example of such an interface.\n1 2 3 4 5 6 7 8 9 10 type rect struct { width, height float64 } func (r rect) area() float64 { return r.width * r.height } func (r rect) perim() float64 { return 2*r.width + 2*r.height } On line 1-3 I have defined a new type which represents a rectangle. On line 5-10 I have implemented the geometry interface by defining the logic to calculate the area and perimeter of rectangle.\nThe beauty of interfaces it that there can be any type which can implement this interface. It also complies with Open-close principle. Below is an example of another shape implementing geometry interface.\n1 2 3 4 5 6 7 8 9 10 type circle struct { radius float64 } func (c circle) area() float64 { return math.Pi * c.radius * c.radius } func (c circle) perim() float64 { return 2 * math.Pi * c.radius } As you can see on line 5-10, the signature of the method is same, just the implementation differs. This way we can have as many shape as we want. There are many such examples in real-world for interfaces.\nTake an example of a car; a car manufacturer should first define an interface for an engine. This could be a spec which defines what part of the engine connects to what other part of the car. The interface could then be implemented by another engine and could be used in the same car because this new engine has the same interface like the first one.\nIf you are looking for an example in software world. You should always develop your application by first declaring an interface. Imagine an interface that interacts with database. One should be using an interface to connect to database which would have method signature like insert, delete etc. Now we can have any database engine no matter mongo or postgres as soon as we implement the insert and delete.\nRelated reading:\nhttps://gobyexample.com/interfaces https://medium.com/better-programming/a-real-world-example-of-go-interfaces-98e89b2ddb67 A Client and a Server # Let\u0026rsquo;s first establish the premise of the story. There is a server and there is a client. Although these both could be on same machine, but in real life there are different machines involved, and we will assume the same.\nThe client is someone requesting resources from the server. The request goes through all the network layers and reaches the HTTP server, which itself is an application layer protocol. The server\u0026rsquo;s job is to reply with a response. Somewhere between request and response happens goes the application logic.\nWhen request reaches to the server, it goes through something called a multiplexer. The job of the multiplexer is to match a route to specific handlers.\nWhat is a handler? Handler is a function which holds chunks of code which is suppose to run for every request made at that route. A handler function has to fulfill a certain signature, which you\u0026rsquo;ll see in next section.\nWhat is a route? Route is a path of the URL.\nURL structure What is http.Handler? # http.Handler - this is an interface. Anything that implements this interface is a handler. Implementing this interface means any function which has signature matching to ServeHTTP(ResponseWriter, *Request). This is what official documentation says. https://github.com/golang/go/blob/master/src/net/http/server.go#L62-L88\n1 2 3 type Handler interface { ServeHTTP(ResponseWriter, *Request) } The pointer to Request above is an struct which holds data from the client. ResponseWriter is an interface which allows us to respond to the request elegantly.\nSimple example:\n1 2 3 func ServeHTTP(w http.ResponseWriter, req *http.Request) { io.WriteString(w, \u0026#34;I am writing to the response\u0026#34;) } http.ResponseWriter has a method signature of Write([]byte) (int, error) which means any type that implements io.Writer can also write to the w; io.WriteString being one of them.\nEasy to understand, right? Now let\u0026rsquo;s add server multiplexer or router to the scene.\nThe Server Multiplexer # http.NewServeMux returns a ServeMux, which looks like this:\n1 2 3 4 5 6 type ServeMux struct { mu sync.RWMutex m map[string]muxEntry es []muxEntry // slice of entries sorted from longest to shortest. hosts bool // whether any patterns contain hostnames } ServerMux is the canonical abstraction of all routes and handlers.\nServerMux has 4 public methods, namely:\n1 2 3 4 func (mux *ServeMux) Handle(pattern string, handler http.Handler) func (mux *ServeMux) HandleFunc(pattern string, handler func(http.ResponseWriter, *http.Request)) func (mux *ServeMux) Handler(r *Request) (h Handler, pattern string) func (mux *ServeMux) ServeHTTP(w http.ResponseWriter, r *http.Request) I\u0026rsquo;ll go through them one by one.\nmux.Handle(pattern string, handler Handler) - This takes a URL pattern and a type which implements a Handler. What is a Handler once again? An interface with method signature of ServeHTTP(w http.ResponseWriter, r *http.Request). 1 2 3 4 5 6 7 8 9 10 11 12 13 14 type hotdog int func (m hotdog) ServeHTTP(w http.ResponseWriter, req *http.Request) { io.WriteString(w, \u0026#34;dog doggy doggggy\u0026#34;) } func main() { var d hotdog mux := http.NewServeMux() mux.Handle(\u0026#34;/dog/\u0026#34;, d) http.ListenAndServe(\u0026#34;:8080\u0026#34;, mux) } Above we can use d variable, which is a hotdog type which implements the Handler interface. The underlaying data type could be anything. In this case, it\u0026rsquo;s it. But it could have been a struct without any side-effect.\nhttp.HandlerFunc - HandlerFunc is a kind of a helper function that converts a standalone function (more on this next) to what mux.Handle takes. Let\u0026rsquo;s add into example above. 1 2 3 4 5 6 7 8 9 10 11 12 13 func madDog(w http.ResponseWriter, r *http.Request) { io.WriteString(w, \u0026#34;I am a mad dog.\u0026#34;) } func main() { var d hotdog mux := http.NewServeMux() mux.Handle(\u0026#34;/dog/\u0026#34;, d) mux.Handle(\u0026#34;/maddog/\u0026#34;, madDog) http.ListenAndServe(\u0026#34;:8080\u0026#34;, mux) } As you can see in the following screenshot, I can\u0026rsquo;t simply use the madDog function as argument to http.ServeMux.Handle. This is because mux.Handle is looking for a type which implements a ServerHTTP method.\nHandle wants a type with ServeHTTP implemented. To overcome this, we can wrap our function with http.HandlerFunc which makes the function mux.Handle compatible.\nhttp.HandleFunc - http.HandleFunc takes a standalone function instead of taking a type which implements Handler inteface. http.ListenAndServe takes a server address and any object which implements http.Handler to start a server. Normally we put a ServeMux, but it will also take any custom type which implements ServeHTTP(w http.ResponseWriter, r *http.Response).\nRelated reading:\nhttps://medium.com/@perennial.sky/understand-handle-handler-and-handlefunc-in-go-e2c3c9ecef03 https://rickyanto.com/understanding-go-standard-http-libraries-servemux-handler-handle-and-handlefunc/ FAQ # What is http.Handle vs mux.Handle? They both are same. When you use http.Handle, program will automatically create a default server multiplexer. But in most cases developers create a new mux. mux := http.NewServeMux().\nIn above example you once passed Handler and then a Mux to the ListenAndServe. How is that? ServeMux implements ServeHTTP(w http.ResponseWriter, r *http.Request), so it\u0026rsquo;s also a handler.\nConclusion # Today we have learned about quite a few about some of the most used functions and struct in Go standard library. If you have liked the post, please share it with your connection and subscribe below for new updates.\n","date":"31 August 2020","externalUrl":null,"permalink":"/posts/2020/difference-between-handler-handle-and-handlerfunc/","section":"Posts","summary":"Introduction # Are you lost between Handle, HandleFunc, and HandlerFunc? If the answer is yes then this post is for you. To understand these pretty well we need to be familiar with interafces. Although interface is just a keyword, it’s confusing at first when you are switching from Python like language; which does not have a similar keyword.\n","title":"Difference Between Handler, Handle and HandlerFunc","type":"posts"},{"content":"","date":"31 August 2020","externalUrl":null,"permalink":"/tags/interface/","section":"Tags","summary":"","title":"Interface","type":"tags"},{"content":"","date":"31 August 2020","externalUrl":null,"permalink":"/tags/multiplexer/","section":"Tags","summary":"","title":"Multiplexer","type":"tags"},{"content":"","date":"17 August 2020","externalUrl":null,"permalink":"/tags/architecture/","section":"Tags","summary":"","title":"Architecture","type":"tags"},{"content":"","date":"17 August 2020","externalUrl":null,"permalink":"/tags/delivery/","section":"Tags","summary":"","title":"Delivery","type":"tags"},{"content":" Introduction # The way we used to deliver resources to the client in a server-client setup is now drifting away from REST to a more modern delivery mechanism. Two of them are gRPC and GraphQL. While both these solve a different kind of problem, REST is going to stay for a while. It is the simplest to learn at least.\nYou might have seen job descriptions requiring skills like the ability to design RESTful API. What does it mean by RESTful API?\nThere are libraries made around this \u0026ldquo;RESTful\u0026rdquo;. One of them I have used in the past is Flask-RESTful. But let\u0026rsquo;s talk about what RESTful API really is for a good understanding.\nIn meantime, please consider connecting with me on LinkedIn\nSo RESTful API is a result of a software architectural style that defines a set of constraints to be used for creating Web services. There are 6 guiding principle which makes RESTful APIs. It isn\u0026rsquo;t that our servers and client won\u0026rsquo;t work if we don\u0026rsquo;t follow these principles; it will make sure we stay on the same page like others.\nThe guiding principles are:\n1. Client-server architecture # This constraint essentially means that client application and server application MUST be able to evolve separately without any dependency on each other. A client should know only resource URIs, and that’s all. Versioning can be used for breaking changes.\n2. Statelessness # All client-server interactions are stateless. There are no sessions are to be maintained. The server will not store anything about the latest HTTP request the client made. It will treat every request as new. No session, no history. Interactions work on the basis of the authentication mechanism such as OAuth, OpenID, JWT et cetra. It is the client\u0026rsquo;s responsibility to maintain user login stuff.\n3. Cacheability # Clients and intermediaries can cache responses. Responses must, implicitly or explicitly, define themselves as either cacheable or non-cacheable to prevent clients from providing stale or inappropriate data in response to further requests. Well-managed caching partially or eliminates some client-server interactions, further improving scalability and performance.\nWell-managed caching partially or completely eliminates some client-server interactions, further improving scalability and performance.\n4. Layered system # A client cannot ordinarily tell whether it is connected directly to the end server, or to an intermediary along the way. If a proxy or load balancer is placed between the client and server, it won\u0026rsquo;t affect their communications and there won\u0026rsquo;t be a need to update the client or server code.\n5. Code on demand # This one is optional. Servers can temporarily extend or customize the functionality of a client by transferring executable code: for example, compiled components such as Java applets, or client-side scripts such as JavaScript.\n6. Uniform interface # The uniform interface constraint is fundamental to the design of any RESTful system. It simplifies and decouples the architecture, which enables each part to evolve independently. The four constraints for this uniform interface are:\n6.1 Resource identification in requests # Individual resources are identified in requests, for example using URIs in RESTful Web services. The resources themselves are conceptually separate from the representations that are returned to the client. For example, the server could send data from its database as HTML, XML or as JSON—none of which are the server\u0026rsquo;s internal representation.\n6.2 Resource manipulation through representations # When a client holds a representation of a resource, including any metadata attached, it has enough information to modify or delete the resource\u0026rsquo;s state.\n6.3 Self-descriptive messages # Each message includes enough information to describe how to process the message. For example, which parser to invoke can be specified by a media type.[3]\n6.4 Hypermedia as the engine of application state (HATEOAS) # Having accessed an initial URI for the REST application—analogous to a human Web user accessing the home page of a website—a REST client should then be able to use server-provided links dynamically to discover all the available resources it needs. As access proceeds, the server responds with text that includes hyperlinks to other resources that are currently available. There is no need for the client to be hard-coded with information regarding the structure or dynamics of the application.\nAll the above constraints help you build a truly RESTful API, and you should follow them. Still, at times, you may find yourself violating one or two constraints. Do not worry; you are still making a RESTful API – but not “truly RESTful.”\nPlease subscribe to newsletter below if you liked the post.\nRelated Readings # https://security.stackexchange.com/q/83450/18179 https://en.wikipedia.org/wiki/Representational_state_transfer https://restfulapi.net/rest-architectural-constraints/ ","date":"17 August 2020","externalUrl":null,"permalink":"/posts/2020/what-does-a-true-restful-api-mean/","section":"Posts","summary":"Introduction # The way we used to deliver resources to the client in a server-client setup is now drifting away from REST to a more modern delivery mechanism. Two of them are gRPC and GraphQL. While both these solve a different kind of problem, REST is going to stay for a while. It is the simplest to learn at least.\n","title":"What Does a True RESTful API Means?","type":"posts"},{"content":"","date":"1 August 2020","externalUrl":null,"permalink":"/tags/algorithm/","section":"Tags","summary":"","title":"Algorithm","type":"tags"},{"content":" Introduction # From The Zen of Go:\nIf you think it’s slow, first prove it with a benchmark\nDon\u0026rsquo;t assume if things are slow. Benchmark it and see if they are really slow.\nOne thing to note here is benchmarking a program is different from profiling a program.\nBenchmarking is the way we check how fast our algorithm is for a given unit of the program. In benchmarking, we typically see how many iterations can a piece of code can run in a given time.\nProfiling is more of a complete picture. Profiling can help determine which methods are called and how long each method takes to complete. Profiling also tell us a lot of valuable things, like what percentage of CPU time was use where, where memory was allocated and things like that. We can say benchmarking is part of profiling.\nToday we\u0026rsquo;ll learn about benchmarking our function in Go. In meantime, please connect with me on LinkedIn\nHow to benchmark a function in Go # In Go, benchmarking is closely tied to the testing suite. This literally means you\u0026rsquo;ll write a benchmarking code in the same place you write your unit/integration tests.\nIf you are familiar with unit tests in Go already, you know you can test a function with below command:\ngo test -run=FuncName You can omit -run=FuncName part to run all the tests in the current level module. And you would benchmark a function the same function like this:\ngo test -bench=FuncName Let us understand this with the help of an example. Below is main.go with DoSomeWork function.\n1 2 3 4 5 // DoSomeWork is a example function which uses time.Sleep to // sleep for an arbitrary number of time. func DoSomeWork() { time.Sleep(time.Nanosecond * 500) } The above code is trivial self-documented, needs no explanation.\nNow, just like test functions follow a naming rule of TestXXX (where XXX is the function name), the benchmark also follows the same naming semantic. Here is BenchmarkDoSomeWork in main_test.go:\n1 2 3 4 5 func BenchmarkDoSomeWork(b *testing.B) { for i := 0; i \u0026lt; b.N; i++ { DoSomeWork() } } How to interpret benchmark output # When you run the benchmark suite using go test -bench=DoSome (or go test -bench=. for all benchmark in the current module).\nYou\u0026rsquo;ll see an output like:\nBenchmarkDoSomeWork 535690 2337 ns/op This means that the loop ran 4945279 times at a speed of 243 ns per loop.\nNow let us optimize our DoSomeWork by decreasing the time it\u0026rsquo;s sleeping for:\n1 2 3 func DoSomeWork() { time.Sleep(time.Nanosecond * 50) } The output again:\nBenchmarkDoSomeWork 1362243 890 ns/op As you can see, our program is taking less time per operation. Earlier it was 2337 nanosecond per operation and now it\u0026rsquo;s 890.\nYou might have noted that the number in the second column is inversely proportional to the number is the third column, which is obvious because when the function executes in less amount of time, it will be able to run more times in a certain time frame.\nConclusion # We have learned:\nHow to write benchmark tests for function. How to read the output of the test. If you require further assistance then please let me know in the comments section below! And don\u0026rsquo;t forget to subscribe to the newsletter below.\n","date":"1 August 2020","externalUrl":null,"permalink":"/posts/2020/benchmarking-in-golang-with-example/","section":"Posts","summary":"Introduction # From The Zen of Go:\nIf you think it’s slow, first prove it with a benchmark\nDon’t assume if things are slow. Benchmark it and see if they are really slow.\nOne thing to note here is benchmarking a program is different from profiling a program.\nBenchmarking is the way we check how fast our algorithm is for a given unit of the program. In benchmarking, we typically see how many iterations can a piece of code can run in a given time.\n","title":"Benchmarking in Go, with Example","type":"posts"},{"content":"","date":"1 August 2020","externalUrl":null,"permalink":"/tags/optimization/","section":"Tags","summary":"","title":"Optimization","type":"tags"},{"content":"The internet is filled with Go learning resources. And I am not going to repeat all the links here. I will list only the resources that I went through.\nI am a self-taught programmer from start. And have limited knowledge of computer science. Which of course is changing day by day. I had a background in Python before getting into Go. So my recommendations will also be aligned with my experience level.\nConnect with me on LinkedIn.\nLearn Go with Tests # The Learn Go with Tests not only teaches the language, but it also enforces you to follow TDD. Very motivating. The jobs I have applied in Backend also have TDD listed as one of the pluses. So TDD is a must if you want to grow as a software engineer. I would recommend using this resource alongside Go by Example.\nDifficulty Level: Beginner - Intermediate\nType: TDD, DIY\nGophercises # Once familiar with the syntax, gophercises will take you on a tour of the language. Exercises demonstrate what can be achieved with Go.\nI was stuck at the Blackjack lesson of the course. Reason 1 being my limited familiarity to Blackjack. And secondly, rapidly jump editing here and there without context. At some point I thought this series is not someone of my level.\nAlthough all the exercises I encountered will now were of easy to follow. I can say some were the one will make me think I should skip that exercise. Perhaps his paid courses maybe worth try. I haven\u0026rsquo;t tried though.\nI am still going through this material as of July 2020.\nDifficulty Level: Beginner - Intermediate\nType: DIY\nGolangBot # One of the best resources after when you want to get around with syntax.\nNaveen Ramanathan uses object-oriented analogies from Python to clear the concept of composing in Go. If you are coming from Python, you will understand concepts like interfaces, pointers, variadic functions in no time, that too with examples.\nThis series also covers concurrency related concepts with examples.\nDifficulty Level: Beginner\nType: Syntax, DIY\nGopherCon Videos # Of course, watch conference videos no matter what the language is. Just go to YouTube and search Gophercon.\nYou may narrow your search by looking up for a specific year, search term such as gophercon 2020 will work. Or maybe by country, how about gophercon india?\nThe videos range from beginners to syntax to workflow to news to what not.\nDifficulty Level: All\nType: All\nGophers Slack # I used to do a lot of IRC back in the time. But now SaaS such as Discord and Slack has gained a lot of popularity.\nIf there is only one resource to recommend. I would recommend this gopher slack workspace.\nOn gopher slack you can get quicks response which you can\u0026rsquo;t even get on StackOverflow. And you don\u0026rsquo;t even have to worry about your question getting closed.\nYou can find various channels ranging from beginners, to gophercises, to Hugo and what not. There is also a channel called #showandtell, where one can show off their project/blog and get feedback.\nDifficulty Level: All\nType: Chat\nGo by Example # While others might recommend A Tour of Go, I will recommend Go by Example. Both are equally good, I find the later one easy.\nI use them as a reference book.\nDifficulty Level: Beginner\nType: Reference\nBonus: StackOverflow Go tag info # If you are still low on resources, follow Go Tag info on StackOverflow. This is how I still find resources to any new technology I want to learn.\nDifficulty Level: All\nType: All\nEffective Go # This is not a tutorial per se, but Effective go will make you write idiomatic go. If you don\u0026rsquo;t comply with these rules, the gopher community will protest against you.\nThis is equivalent to the Zen of Python, but in an explanatory way.\nhttps://golang.org/help/ # There are many more resources out there on the web, many more being added by the time. These ones are the one with which I have spend most of the time.\nWhat what is your best resource? Did you choose something different? Please share it with the community.\nNote: Background image used is by Denise Yu.\n","date":"25 July 2020","externalUrl":null,"permalink":"/posts/2020/top-6-free-learning-resources-i-used-to-learn-go/","section":"Posts","summary":"The internet is filled with Go learning resources. And I am not going to repeat all the links here. I will list only the resources that I went through.\nI am a self-taught programmer from start. And have limited knowledge of computer science. Which of course is changing day by day. I had a background in Python before getting into Go. So my recommendations will also be aligned with my experience level.\n","title":"Top 6 Free Learning Resources I used to Learn Go","type":"posts"},{"content":" Introduction # In last post we learned how we can use functional options to pass parameter to our functions. In this post we\u0026rsquo;ll see an alternative way to do the same thing.\nI\u0026rsquo;ll take the same code from the last post to keep things simple.\nVariadic Functions for Options # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 type Coffee struct { CoffeeBeans bool Sugar bool SteamedMilk bool } func New(opts ...func(Coffee) Coffee) Coffee { coffee := Coffee{CoffeeBeans: true} for _, opt := range opts { coffee = opt(coffee) } return coffee } // WithSugar adds sugar to the coffee. func WithSugar(coffee Coffee) Coffee { coffee.Sugar = true return coffee } // WithSteamedMilk adds steamed milk to the coffee. func WithSteamedMilk(coffee Coffee) Coffee { coffee.SteamedMilk = true return coffee } // Prepare makes a cup of coffee with given items. // Ignore the implementation for now. func (c Coffee) Prepare() string { v := reflect.ValueOf(c) typeOfC := v.Type() var items []string for i := 0; i \u0026lt; v.NumField(); i++ { if v.Field(i).Interface() == true { items = append(items, typeOfC.Field(i).Name) } } return fmt.Sprintf(\u0026#34;Preparing coffee with %q:\u0026#34;, items) } func main() { s := New(WithSugar) fmt.Println(s.Prepare()) } Here on line 1-5 we define type Coffee (or class if you would like to say). This is the analogy our code is build upon because I like coffee whenever I can.\nOn line 7-15 we define our New function. func(Coffee) Coffee says New takes a function with Coffee object and spits out a Coffee Object. The ... part tells it can take many of these functions. Yes, there\u0026rsquo;s no limit. Then New function itself returns a Coffee object.\nIn the body of the function, we create a default instance of Coffee. Then run all the functions one by one. As all the function return same modified Coffee object, we can modify it an many time the number of function are passed. At last return that object.\nNow for every option we want to create, we can create a function with the same signature we saw above, i.e., func(Coffee) Coffee. I have created two such functions between. There can be any logic you want. just make sure to return the modified coffee object.\nOn line 46, we define a New coffee object with WithSugar parameter. The logic of the function will modify Coffee to turn Sugar field to be true.\nOn line 47, Prepare() prints out whatever fields are set to be true.\nNow let\u0026rsquo;s do the same thing with Options struct in next section.\nOptions Struct for Options # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 type Coffee struct { CoffeeBeans bool Sugar bool SteamedMilk bool } type Options struct { WithCoffeeBeans bool WithSugar bool WithSteamedMilk bool } func New(opts Options) Coffee { c := Coffee{ // well make coffeebeans necessary CoffeeBeans: true, Sugar: opts.WithSugar, SteamedMilk: opts.WithSteamedMilk, } return c } // Prepare makes a cup of coffee with given items. // Ignore the implementation for now. func (c Coffee) Prepare() string { v := reflect.ValueOf(c) typeOfC := v.Type() var items []string for i := 0; i \u0026lt; v.NumField(); i++ { if v.Field(i).Interface() == true { items = append(items, typeOfC.Field(i).Name) } } return fmt.Sprintf(\u0026#34;Preparing coffee with %q:\u0026#34;, items) } func main() { options := Options{WithSugar: true} s := New(options) fmt.Println(s.Prepare()) } Changes # 1. Variadic functions are gone # That seems like a save of real estate. Because every function takes a fare amount of space. Also there are less moving parts to work with.\nWithSugar and WithSteamedMilk are completely gone. Along with the signature of the New function, which is changed from being New(opts ...func(Coffee) Coffee) Coffee to New(opts Options) Coffee. Pretty concise, eh?\n2. A new Options struct is introduced # All the settings which could have been there in the form of functions are listed directly here. Functions were just a way to encapsulate the settings.\nYou must be wondering, can\u0026rsquo;t we modify all the fields in Coffee directly? Do we need a separate Options struct for this thing? That\u0026rsquo;s an extra step!\nNote that all the fields in Coffee struct could have been unexported. And if you have called this API from your driver code, you won\u0026rsquo;t be able to set it directly, as it is unexported. Consider this case:\ncoffee/coffee.go:\n1 2 3 4 5 6 7 package coffee type Coffee struct { coffeeBeans bool sugar bool steamedMilk bool } main.go:\n1 2 3 4 5 6 package main func main() { c := coffee.Coffee{coffeeBeans: true} fmt.Println() } You might get an error like this when run:\n# command-line-arguments ./main.go:10:21: unknown field \u0026#39;coffeeBeans\u0026#39; in struct literal of type coffee.Coffee That\u0026rsquo;s because coffeeBeans is an unexported field. The idea here I\u0026rsquo;m trying to put is, encapsulated inside, while only interface to modify it being the Option struct.\nThe struct method also gives freedom to set default values. As you can see on line 13-22, I have set CoffeeBeans to true, no matter what you pass to it.\n3. New function accepts an Option # Instead of taking functions, like before, we now create an option struct beforehand. And pass it New function.\nNew is also modified to generate a new Coffee object with all the options passed into it. Here we can also make some behaviors default. Like in the above example, even if the user passes CoffeeBeans value as false, our function will always create a Coffee object with CoffeeBeans.\nConclusion # I haven\u0026rsquo;t yet form any solid opinion with any of them. Both of them seem equally reasonable to use. I need to practice more and more in order to have a solid opinion on any of them.\nWhich one do you like any why? Let me know in the comments. And if you want to stay updated with new articles like this one, please subscribe to the newsletter below.\n","date":"18 July 2020","externalUrl":null,"permalink":"/posts/2020/optional-parameters-using-option-struct/","section":"Posts","summary":"Introduction # In last post we learned how we can use functional options to pass parameter to our functions. In this post we’ll see an alternative way to do the same thing.\nI’ll take the same code from the last post to keep things simple.\nVariadic Functions for Options # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 type Coffee struct { CoffeeBeans bool Sugar bool SteamedMilk bool } func New(opts ...func(Coffee) Coffee) Coffee { coffee := Coffee{CoffeeBeans: true} for _, opt := range opts { coffee = opt(coffee) } return coffee } // WithSugar adds sugar to the coffee. func WithSugar(coffee Coffee) Coffee { coffee.Sugar = true return coffee } // WithSteamedMilk adds steamed milk to the coffee. func WithSteamedMilk(coffee Coffee) Coffee { coffee.SteamedMilk = true return coffee } // Prepare makes a cup of coffee with given items. // Ignore the implementation for now. func (c Coffee) Prepare() string { v := reflect.ValueOf(c) typeOfC := v.Type() var items []string for i := 0; i \u003c v.NumField(); i++ { if v.Field(i).Interface() == true { items = append(items, typeOfC.Field(i).Name) } } return fmt.Sprintf(\"Preparing coffee with %q:\", items) } func main() { s := New(WithSugar) fmt.Println(s.Prepare()) } Here on line 1-5 we define type Coffee (or class if you would like to say). This is the analogy our code is build upon because I like coffee whenever I can.\n","title":"Optional Parameters Using Option Struct","type":"posts"},{"content":"","date":"18 July 2020","externalUrl":null,"permalink":"/tags/redability/","section":"Tags","summary":"","title":"Redability","type":"tags"},{"content":" Introduction # In Python you can assign default value to the functions, like so:\n1 2 def my_func(posarg, named_arg=True, another_named_arg=\u0026#34;Okay\u0026#34;) # logic goes here... But in Go you can\u0026rsquo;t do this:\n1 2 3 func myFunc(posarg, named_arg bool = true, another_named_arg string = \u0026#34;Okay\u0026#34;) { // logic goes here... } This will throw a syntax error. Try that?\nIn this post, we\u0026rsquo;ll see how can we overcome this issue the go way, with something called Function Options. Not sure if go is the only language that uses this syntax, if there is another language you are using, please let me know. Also, connect with me on LinkedIn\nAlso the end result will not look similar to the example above. But you will get the point and know how to proceed further.\nLet us go step by step.\nFunctions are First-Class Citizens # In a language, functions are called the first-class citizen when they are treated like any other object and can be passed around like any other value. For example, a function can be passed as an argument to a function, can be returned by other functions and can be assigned as a value to a variable.\nPython and JavaScript both are first-class function languages, so is Go.\nThis is more like an axiom on which we\u0026rsquo;ll build our proof on \u0026#x1f609;. We\u0026rsquo;ll bulid more upon this in next section.\nVariadic Function # We can\u0026rsquo;t have default parameters in our functions, but the language allows us to pass an unlimited number of parameters.\nIf you follow JavaScript, you must have seen the spread operator.\n1 2 3 const sum = (...nums) =\u0026gt; nums.reduce((a, b) =\u0026gt; a + b); console.log(sum(...[1, 2, 3, 4])) // 10 The same syntax is available in under the name variadic function. We can do similar thing in Go.\n1 2 3 4 5 6 7 8 9 10 11 func sum(nums ...int) int { res := 0 for _, n := range nums { res += n } return res } func main() fmt.Println(Sum(1, 2, 3, 4)) // 6 } Functional Options # We can combine these both behavior of variadic function and first-class functions to pass an unlimined number of parameters to our main function.\nFor a simple example, let\u0026rsquo;s create a new function which creates a cup of coffee. The idea here is to define the constructor function which will make the base coffee, make it accept function as argument (because function are first-class citizen in Go) which are variadic (meaning it accept any number of argument).\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 type Coffee struct { CoffeeBeans bool Sugar bool SteamedMilk bool } // New returns new instance of coffee func New(opts ...func(Coffee) Coffee) { coffee := Coffee{CoffeeBeans: true} for _, opt := range opts { coffee = opt(coffee) } return coffee } The idea here is to keep signature of the all function which are going to be argument to be same as defined in the main function.\n16 17 18 19 20 21 22 23 24 25 26 // WithSugar adds sugar to the coffee. func WithSugar(coffee Coffee) Coffee { coffee.Sugar = true return coffee } // WithSteamedMilk adds steamed milk to the coffee. func WithSteamedMilk(coffee Coffee) Coffee { coffee.SteamedMilk = true return coffee } In the above over-simplified example, the signature is func(Coffee) Coffee which means the all the functional option should take Coffee as a param and also should return one for option to work.\nYou can also modify the functional argument to accept arguments in its own. The key is to keep the signature same.\n1 2 3 4 5 6 7 func WithSugar(cubes int) func(coffee Coffee) Coffee { return func(coffee Coffee) Coffee { sugar := \u0026#34;sugar\u0026#34; + strconv.Itoa(cubes) coffee = append(coffee, sugar) return coffee } } The complete picture # Let\u0026rsquo;s finish our main package and test. I have added a Prepare method on Coffee to demonstrate how composition works in Golang. Here is the complete code:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 package main import ( \u0026#34;fmt\u0026#34; \u0026#34;reflect\u0026#34; ) type Coffee struct { CoffeeBeans bool Sugar bool SteamedMilk bool } func New(opts ...func(Coffee) Coffee) Coffee { coffee := Coffee{CoffeeBeans: true} for _, opt := range opts { coffee = opt(coffee) } return coffee } // WithSugar adds sugar to the coffee. func WithSugar(coffee Coffee) Coffee { coffee.Sugar = true return coffee } // WithSteamedMilk adds steamed milk to the coffee. func WithSteamedMilk(coffee Coffee) Coffee { coffee.SteamedMilk = true return coffee } // Prepare makes a cup of coffee with given items. // Ignore the implementation for now. func (c Coffee) Prepare() string { v := reflect.ValueOf(c) typeOfC := v.Type() var items []string for i := 0; i \u0026lt; v.NumField(); i++ { if v.Field(i).Interface() == true { items = append(items, typeOfC.Field(i).Name) } } return fmt.Sprintf(\u0026#34;Preparing coffee with %q:\u0026#34;, items) } func main() { s := New(WithSugar) fmt.Println(s.Prepare()) } In a later post, I\u0026rsquo;ve demonstrated an alternative approach to functional options using Options struct. Both patterns are seen all over the Go ecosystem.\nIf you want to stay updated with articles like this, please subscribe to newsletter below.\nRelated readings:\nhttps://dave.cheney.net/2014/10/17/functional-options-for-friendly-apis https://yourbasic.org/golang/variadic-function/ https://www.sohamkamani.com/golang/options-pattern/ ","date":"4 July 2020","externalUrl":null,"permalink":"/posts/2020/optional-parameters-using-functional-options-in-golang/","section":"Posts","summary":"Introduction # In Python you can assign default value to the functions, like so:\n1 2 def my_func(posarg, named_arg=True, another_named_arg=\"Okay\") # logic goes here... But in Go you can’t do this:\n1 2 3 func myFunc(posarg, named_arg bool = true, another_named_arg string = \"Okay\") { // logic goes here... } This will throw a syntax error. Try that?\nIn this post, we’ll see how can we overcome this issue the go way, with something called Function Options. Not sure if go is the only language that uses this syntax, if there is another language you are using, please let me know. Also, connect with me on LinkedIn\n","title":"Optional Parameters using Functional Options","type":"posts"},{"content":"","date":"20 June 2020","externalUrl":null,"permalink":"/tags/elasticsearch/","section":"Tags","summary":"","title":"Elasticsearch","type":"tags"},{"content":"","date":"20 June 2020","externalUrl":null,"permalink":"/tags/json/","section":"Tags","summary":"","title":"Json","type":"tags"},{"content":"","date":"20 June 2020","externalUrl":null,"permalink":"/tags/marshaling/","section":"Tags","summary":"","title":"Marshaling","type":"tags"},{"content":" Introduction # Send a POST request in golang is pretty daunting if you have a post body and you\u0026rsquo;re coming from a scripting language like JavaScript or Python. Here in Go, schema for JSON needs to be defined beforehand in order to marshal and unmarshal string back and forth to JSON.\nSimple POST # This marshaling/unmarshalling could be an oneliner if your request body is not nested, or is one level deep, or there is not even a body. Consider this example:\n1 2 3 4 5 func main() { requestBody, err := json.Marshal(map[string]string{\u0026#34;key\u0026#34;: \u0026#34;value\u0026#34;}) res, _ := http.Post(\u0026#34;https://httpbin.org/post\u0026#34;, \u0026#34;application/json\u0026#34;, bytes.NewBuffer(requestBody)) } POST with a Body # But imagine a request body like this, which is a sample body I am using to send a POST request to Elasticsearch:\n1 2 3 4 5 6 7 8 9 10 { \u0026#34;sort\u0026#34;: [ {\u0026#34;@timestamp\u0026#34;: {\u0026#34;order\u0026#34;: \u0026#34;desc\u0026#34;} }], \u0026#34;query\u0026#34;: { \u0026#34;match\u0026#34;: { \u0026#34;path.to.keyword\u0026#34;: { \u0026#34;query\u0026#34;: \u0026#34;the-term\u0026#34; } } } } If it was JavaScript, I could have simply sent post request like this:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 let endpoint = \u0026#34;https://httpbin.org/post\u0026#34; let query = { \u0026#34;sort\u0026#34;: [ {\u0026#34;@timestamp\u0026#34;: {\u0026#34;order\u0026#34;: \u0026#34;desc\u0026#34;} }], \u0026#34;query\u0026#34;: { \u0026#34;match\u0026#34;: { \u0026#34;path.to.keyword\u0026#34;: { \u0026#34;query\u0026#34;: \u0026#34;the-term\u0026#34; } } } } fetch(endpoint, { method: \u0026#34;POST\u0026#34;, body: JSON.stringify(query), headers: { \u0026#34;Content-Type\u0026#34;: \u0026#34;application/json\u0026#34;, }, }) But this is not dealing with scripting language!\nCreate Schema, aka struct # To do this in go way, we need to do this in this manner:\n36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 type Request struct { Query Query `json:\u0026#34;query\u0026#34;` Sort []Sort `json:\u0026#34;sort\u0026#34;` } type Sort struct { Timestamp Timestamp `json:\u0026#34;@timestamp\u0026#34;` } type Timestamp struct { Order string `json:\u0026#34;order\u0026#34;` } type Query struct { Match Match `json:\u0026#34;match\u0026#34;` } type Match struct { PathToKeyword PathToKeyword `json:\u0026#34;path.to.keyword\u0026#34;` } type PathToKeyword struct { Query string `json:\u0026#34;query\u0026#34;` } The Driver Code # Followed by the driver code to make a POST request, which would be:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 package main import ( \u0026#34;bytes\u0026#34; \u0026#34;encoding/json\u0026#34; \u0026#34;fmt\u0026#34; \u0026#34;io/ioutil\u0026#34; \u0026#34;net/http\u0026#34; ) func main() { requestBody, err := json.Marshal(Response{ Query: Query{ Match: Match{ PathToKeyword: PathToKeyword{ Query: \u0026#34;the-term\u0026#34;, }, }, }, Sort: []Sort{ { Timestamp: Timestamp{ Order: \u0026#34;desc\u0026#34;, }, }, }, }) res, _ := http.Post(\u0026#34;https://httpbin.org/post\u0026#34;, \u0026#34;application/json\u0026#34;, bytes.NewBuffer(requestBody)) defer res.Body.Close() body, _ := ioutil.ReadAll(res.Body) fmt.Println(string(body)) } Just for the sake of simplicity, I\u0026rsquo;ve decided not to handle errors.\nConclusion # You may find this nuance of go not very user friendly, but this was not one of the design golang of the language. And someone has well said\u0026hellip; (in Gopher Slack)\nGo is very much a non-magical language, you have to actually program :)— Peter Bourgon Related Links:\nhttps://medium.com/@masnun/making-http-requests-in-golang-dd123379efe7 https://mholt.github.io/json-to-go/ ","date":"20 June 2020","externalUrl":null,"permalink":"/posts/2020/sending-post-request-in-go-with-a-body/","section":"Posts","summary":"Introduction # Send a POST request in golang is pretty daunting if you have a post body and you’re coming from a scripting language like JavaScript or Python. Here in Go, schema for JSON needs to be defined beforehand in order to marshal and unmarshal string back and forth to JSON.\nSimple POST # This marshaling/unmarshalling could be an oneliner if your request body is not nested, or is one level deep, or there is not even a body. Consider this example:\n","title":"Sending POST Request in Go with a Body","type":"posts"},{"content":" Introduction # Have you ever ran into a situation where you wanted to version your application, but doing git tag and making a separate commit for updating version number was a two-step process? Isn\u0026rsquo;t that kinda frustrating? You can even miss to update the version number in your code, which happens to be with me a lot. In this post we\u0026rsquo;ll see how to convert this two-step process into one.\nTake a moment to connect with me on LinkedIn.\nIf you are an open-source enthusiast like me, you might have seen version number of software like these:\n1.0.2/v1.0.2 Which follows Semantic Versioning and is the most common and the recommended way. For example:\n$ bat --version bat 0.15.0 The above version number is often complemented with one of the following.\n20190202 (which obviously is a date of build)\n35fc2ad (which is a short version of sha1 hash of commit)\nThere might be others which I\u0026rsquo;m missing.\nWith help of a simple example, we\u0026rsquo;ll see how to combine these both steps into one. Once again let me remind what these two processes I am talking about:\nOne is some kind of version variable you maintain, so that it is visible when you do something like myapp --version. Second is the version number you maintain in your VCS (known as tags in git terminology) Let\u0026rsquo;s get with the example then?\nPrerequisites # You should have a fair knowledge in these two technologies.\nGo Git Bash (coreutils actually) Some C/C++ background info # As a compiled language, Go inherits a lot from C, and behaves the same way C/C++ does. I can\u0026rsquo;t say about other languages of the C family. But in C/C++ source code files are taken and compiled to object files. Then different object files are linked together to make the executable work.\nSome program without main functions (libraries) are developed separately and are linked on runtime. There are benefits for developing libraries separately, but that\u0026rsquo;s not the scope of this post.\nFor a software written in C/C++ I can see list of dynamically linked libraries as follows:\n$ ldd `which darktable` linux-vdso.so.1 (0x00007ffcc8fbc000) /usr/lib64/libstdc++.so.6 (0x00007f1a14ab0000) /lib64/libgcc_s.so.1 (0x00007f1a14a90000) libdarktable.so =\u0026gt; /usr/bin/../lib64/darktable/libdarktable.so (0x00007f1a146e8000) libc.so.6 =\u0026gt; /lib64/libc.so.6 (0x00007f1a144e8000) libm.so.6 =\u0026gt; /lib64/libm.so.6 (0x00007f1a143a0000) /lib64/ld-linux-x86-64.so.2 (0x00007f1a14ca8000) libpthread.so.0 =\u0026gt; /lib64/libpthread.so.0 (0x00007f1a14378000) libgomp.so.1 =\u0026gt; /lib64/libgomp.so.1 (0x00007f1a14330000) libglib-2.0.so.0 =\u0026gt; /lib64/libglib-2.0.so.0 (0x00007f1a14200000) libgtk-3.so.0 =\u0026gt; /lib64/libgtk-3.so.0 (0x00007f1a13a30000) libgdk-3.so.0 =\u0026gt; /lib64/libgdk-3.so.0 (0x00007f1a13928000) [...] In Go, there is one similar mechanism by which we can pass dynamic data to go build toolkit amid linking.\nSo without further ado, let\u0026rsquo;s see what it is.\nBuilding the Sample Application # Below is the minimal working example I will use to demonstrate the workings on linking.\nI have created a workspace named autover and written a main.go file in it.\n1 2 3 4 5 6 7 8 9 package main import \u0026#34;fmt\u0026#34; var Version = \u0026#34;development\u0026#34; func main() { fmt.Println(\u0026#34;version: \u0026#34;, Version) } Let\u0026rsquo;s compile and run?\n$ go build \u0026amp;\u0026amp; ./autover version: development ldflags # go build triggers go tool link in background preceded by flags and object files.\nFrom outside, we can pass-ldflags to the linker.\nThis is how typically -ldflag is passed to build process.\ngo build -ldflags=\u0026quot;-flag key=value\u0026quot; Now going forward, we can inject piece of text to a certain variable inside the code.\nFrom the link docs:\n-X importpath.name=value Set the value of the string variable in importpath named name to value. [...] importpath.name is the name of the variable and value is any text we want to assign to it.\nThis is how we\u0026rsquo;re gonna use this build flag:\ngo build -ldflags=\u0026#34;-X \u0026#39;main.Version=v0.1.1\u0026#39;\u0026#34; Remember the Version variable from the example? Yes, we can modify using -ldflags. We have to quote anything passed to -X though, just to make sure nothing breaks.\n$ go build -ldflags=\u0026#34;-X \u0026#39;main.Version=v0.1.1\u0026#39;\u0026#34; \u0026amp;\u0026amp; ./app version: v1.0.0 Do you see version number coming? Yes!!\nThe point to be noted is the variable name. It does not have to be main.Version. Is can be anything, and any module other than main.\ngit tag # But how we gonna mix git tag information to it? We can use shell command substitution to put output of another shell command as an input to other command. Like from above you might have noticed I did which darktable to ldd command. which darktable is substituted to /usr/bin/darktable on my system and this is what passed to ldd command.\nGuess what we\u0026rsquo;ll pass to -ldflags \u0026quot;-X 'main.Version'\u0026quot; instead v0.1.1? Yes, the output of git tag.\nThe whole build command will be\u0026hellip;\ngo build -ldflags=\u0026quot;-X 'main.Version=`git tag`'\u0026quot; \u0026amp;\u0026amp; ./autover Ninja technique to get latest tag # Now you might be afraid what if there are more than one tags in the repo? Would the assignment be messed up? The answer is yes. To get the first line of the output I tried this:\ngit tag | tail -n 1 This worked until I had a few tags.\n$ git tag 1.2.2 1.2.3 1.2.4 1.2.5 1.2.6 $ git tag | tail -n 1 1.2.6 But this will break on tags like 1.2.10 which actually comes to the first.\n$ git tag 1.2.10 1.2.2 1.2.3 1.2.4 1.2.5 1.2.6 1.2.7 1.2.8 1.2.9 As you can see, the latest tag is at the top. Very bad! Our method failed 😭. So much inconsistency.\nAfter a few research, I found a way to get the latest tag the git way.\ngit describe --tags Now you know what to do next.\nAs for the date and commit hash part, they can be achieved in the same manner.\nNote: If you are doing this in package other than main, you gotta use fully qualified package name.\nRelated reading:\nhttps://git-scm.com/docs/git-describe https://stackoverflow.com/a/1404862/939986 Now you are ready to incorporate this into your more complex codebase. But the basics will be same. If you find any error or inconsistency, please let me know in the comment. Or if you are a twitter person, I\u0026rsquo;m @sntshk.\nSubscribe below for more articles like this one.\n","date":"6 June 2020","externalUrl":null,"permalink":"/posts/2020/dynamically-insert-version-info-from-git-tag-to-your-app/","section":"Posts","summary":"Introduction # Have you ever ran into a situation where you wanted to version your application, but doing git tag and making a separate commit for updating version number was a two-step process? Isn’t that kinda frustrating? You can even miss to update the version number in your code, which happens to be with me a lot. In this post we’ll see how to convert this two-step process into one.\n","title":"Dynamically Insert Version Info From git tag to Your App","type":"posts"},{"content":"","date":"6 June 2020","externalUrl":null,"permalink":"/categories/workflow/","section":"Categories","summary":"","title":"Workflow","type":"categories"},{"content":"","date":"25 May 2020","externalUrl":null,"permalink":"/tags/frontend/","section":"Tags","summary":"","title":"Frontend","type":"tags"},{"content":"I will start this post with a quote from webassembly.org:\nWebAssembly (abbreviated Wasm) is a binary instruction format for a stack-based virtual machine. Wasm is designed as a portable target for compilation of high-level languages like C/C++/Rust, enabling deployment on the web for client and server applications.— webassembly.org That\u0026rsquo;s a lot of technical jargon. In very simple terms, Wasm provides a way to run code written in multiple languages on the web at near-native speed. Where multiple languages refers to all these languages.\nBasically what web assembly allows us to write code in any supported favorite language, compile it to binary (typically with the extension .wasm), and call it from JavaScript code in the browser.\nThis is going to break the stereotype of developing frontends in JavaScript, which has been the language for frontend. With the advent of frameworks like Vugu, and Vecty things seems to be changing in regards with Go. They are in fact in active development. How about DOM binding in Go? How about making games?\nWASM has been developed collaboratively by Mozilla, Microsoft, Google, Apple and W3C (I see W3C as similar to CNCF, but for WWW only). So, it isn\u0026rsquo;t going anywhere for sure.\nWeb Assembly in Go is still experimental API. And the API may change in future, which means it is backward incompatible.\nCurrent Usage # According to Wikipedia:\nWebAssembly became a World Wide Web Consortium recommendation on 5 December 2019 and, alongside HTML, CSS, and JavaScript, is the fourth language to run natively in browsers.— Wikipedia And regarding the support in browsers, this is what Wikipedia has to say:\nAs of May 2020, 91.44% of installed browsers (91.56% of desktop browsers and 92.94% of mobile browsers) support WebAssembly.— Wikipedia That\u0026rsquo;s too much of information for today. Let\u0026rsquo;s do some quick example.\nWasm and Go # As I am a Go fan, let\u0026rsquo;s do a Hello WASM! application with it. WebAssembly works with Go 1.11 and above. I am writing along this blog post with go verison 1.14.2.\nHello WASM # Powerpuff recepie for WASM There are 4 ingredients we need.\nwasm binary which we are going to compile wasm_exec.js which will hold the connector code between wasm binary and HTML file an HTML page to hook the wasm_exec.js file to a web server to host these three Out of all these, only the binary is going to be go specific. Rest are generic to all the supported languages. Even that you are familiar with JavaScript, you can find the documentation on MDN to write your own code to glue things together. Anyway, let\u0026rsquo;s deal with the binary:\n1 2 3 4 5 6 7 8 9 package main import ( \u0026#34;fmt\u0026#34; ) func main() { fmt.Println(\u0026#34;Hello WASM!\u0026#34;) } Normally to generate an executable, I would have done go build -o hello.wasm hello_wasm.go (frankly speaking, I just do go build on my Fedora). But to create a WASM binary we gotta set some environment variables. And those variables are GOOS=js GOARCH=wasm. So the full command would be something like this:\nGOOS=js GOARCH=wasm go build -o hello.wasm hello_wasm.go Next, copy wasm_exec.js to cwd from go installation directory:\ncp \u0026#34;$(go env GOROOT)/misc/wasm/wasm_exec.js\u0026#34; . If you are somehow not able to find the files there, like me, get it from https://github.com/golang/go/tree/master/misc/wasm. Just to make sure, this piece of code is from master branch, and you are expected to use it with latest version of golang. Otherwise go to specific branch and then download.\ncurl https://raw.githubusercontent.com/golang/go/master/misc/wasm/wasm_exec.js -O Use this HTML snippet:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 \u0026lt;!doctype html\u0026gt; \u0026lt;html\u0026gt; \u0026lt;head\u0026gt; \u0026lt;meta charset=\u0026#34;utf-8\u0026#34;\u0026gt; \u0026lt;title\u0026gt;Go wasm\u0026lt;/title\u0026gt; \u0026lt;/head\u0026gt; \u0026lt;body\u0026gt; \u0026lt;script src=\u0026#34;wasm_exec.js\u0026#34;\u0026gt;\u0026lt;/script\u0026gt; \u0026lt;script\u0026gt; const go = new Go(); let mod, inst; WebAssembly.instantiateStreaming(fetch(\u0026#34;hello.wasm\u0026#34;), go.importObject).then((result) =\u0026gt; { mod = result.module; inst = result.instance; }).catch((err) =\u0026gt; { console.error(err); }); \u0026lt;/script\u0026gt; \u0026lt;/body\u0026gt; \u0026lt;/html\u0026gt; WebAssembly.instantiateStreaming takes in our binary and import object. And returns WebAssembly.Module and a WebAssembly.Instance object, which in above example is assigned to mod and inst variable respectively.\nI\u0026rsquo;m going to ditch golang and start a web server with Python. 🤭 It\u0026rsquo;s simple.\n1 python -m http.server This will start serving files from the current directory. Head over to http://127.0.0.1:8000. Once the wasm_exec.html is loaded, you will see a button named Run.\nRun button\u0026rsquo;s output when everything is set correctly. Note: If you don\u0026rsquo;t see anything in the console, check if the correct file is linked in the JavaScript file. The default one is \u0026ldquo;test.wasm\u0026rdquo; in the HTML file, but we compiled hello.wasm.\nExporting a Go function to JavaScript # syscall/js is at the heart of all the operations between JavaScript and Go. And this is library agnostic.\nWhile going through the documentation you will find that there are only a few types exposed. We\u0026rsquo;ll will focus on Global Func and Value (and their methods) today.\njs.Global() can be used to get JavaScript global object. From which we can get the document and eventually everything in the DOM.\nThis is a JavaScript function which looks for an element with ID foo and sets the text to bar.\n1 2 3 4 5 function barFunc() { let para = document.createElement(\u0026#34;p\u0026#34;); para.innerHTML = \u0026#34;bar\u0026#34;; document.getElementsByTagName(\u0026#34;body\u0026#34;)[0].appendChild(para); } Equivalent code in Go would look like this:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 package main import \u0026#34;syscall/js\u0026#34; func main() { js.Global().Set(\u0026#34;barFunc\u0026#34;, js.FuncOf(barFunc)) // Prevent main from exiting select {} } func barFunc(this js.Value, inputs []js.Value) interface{} { document := js.Global().Get(\u0026#34;document\u0026#34;) p := document.Call(\u0026#34;createElement\u0026#34;, \u0026#34;p\u0026#34;) p.Set(\u0026#34;innerHTML\u0026#34;, inputs[0]) document.Get(\u0026#34;body\u0026#34;).Call(\u0026#34;appendChild\u0026#34;, p) return nil } Note that signature has to be like that. This will make barFunc available to global namespace.\nExposed Go function available in JavaScript Moreover, there is a whole lot of resources available regarding Go and WASM at https://github.com/golang/go/wiki/WebAssembly\nWe are not limited to DOM manipulation with WASM. We can use any existing browser API such as using network calls, LocalStorage, using WebGL etc. I would recommend this GopherCon video by Johan Brandhorst who is a contributor to golang codebase.\nSecurity Considerations # This was an eagle eye introduction to WebAssembly in Go. Let me end this post with some security concerns of WebAssembly.\nWebAssembly has been criticized for allowing greater ease of hiding the evidence for malware writers, scammers and phishing attackers; WebAssembly is only present on the user\u0026rsquo;s machine in its compiled form, which \u0026ldquo;makes malware detection difficult\u0026rdquo;.\nThe speed and concealability of WebAssembly have led to its use in hidden crypto mining on the website visitor\u0026rsquo;s device. Coinhive, a now defunct service facilitating cryptocurrency mining in website visitors\u0026rsquo; browsers, claims their \u0026ldquo;miner uses WebAssembly and runs with about 65% of the performance of a native Miner.\u0026rdquo;\nA June 2019 study from the Technische Universität Braunschweig, analyzed the usage of WebAssembly in the Alexa top 1 million websites and found the prevalent use was for malicious crypto mining, and that malware accounted for more than half of the WebAssembly-using websites studied.\nSubscribee below for more updates like this.\n","date":"25 May 2020","externalUrl":null,"permalink":"/posts/2020/introduction-to-webassembly-with-go/","section":"Posts","summary":"I will start this post with a quote from webassembly.org:\nWebAssembly (abbreviated Wasm) is a binary instruction format for a stack-based virtual machine. Wasm is designed as a portable target for compilation of high-level languages like C/C++/Rust, enabling deployment on the web for client and server applications.— webassembly.org That’s a lot of technical jargon. In very simple terms, Wasm provides a way to run code written in multiple languages on the web at near-native speed. Where multiple languages refers to all these languages.\n","title":"Introduction to WebAssembly with Go","type":"posts"},{"content":"","date":"25 May 2020","externalUrl":null,"permalink":"/tags/webassembly/","section":"Tags","summary":"","title":"Webassembly","type":"tags"},{"content":"","date":"18 May 2020","externalUrl":null,"permalink":"/tags/beanstalk/","section":"Tags","summary":"","title":"Beanstalk","type":"tags"},{"content":"In a previous post we discovered how can we run commands on each push and check if the test is passing against multiple versions of go. This is a continuation of the same concept. We are going to deploy our code to Elastic Beanstalk, straight from git push.\nElastic Beanstalk is nothing but a wrapper around it\u0026rsquo;s EC2 infrastructure. It is a developer-centric view of deploying an application on AWS. Developers just have to push the code and Beanstalk can take care of Auto Scaling Group, Elastic Load Balancer, databases etc. No need to manage infrastructure manually.\nHere is the outline of the post:\nPrerequisites Create an app on Elastic Beanstalk Obtain the access and secret key from AWS Console Create an HTTP Server Setup .travis.yml Please connect with me on LinkedIn.\nPrerequisites # I will assume that:\nYou have a working knowledge of git and TDD. You have an AWS account. You are hosting your code on GitHub, and Travis is enabled on repo. This setup must work on other providers, but I haven\u0026rsquo;t got time to test it. Let me know in the comments if you did it yourself and want to share it.\nCreate an app on Elastic Beanstalk # Before we automate the process of deployment, we must have an already running application on Beanstalk. If you don\u0026rsquo;t have one, head over to https://console.aws.amazon.com/elasticbeanstalk/home and look for a button saying Create a new application.\nSelect Docker for platform.\nCreate new beanstalk application This will create an app, an environment, and an S3 bucket where the code well be uploaded.\nHere is my already working instance of unittest application which we\u0026rsquo;ll be automating the CD workflow to.\nAlready working instance of beanstalk application Get the access and secret key from IAM Console # At this stage, you might want to create a user for this app and attach AWSElasticBeanstalkFullAccess policy to the user. But I am going to use my master account.\nYou can generate Access key ID and Secret access key by heading over to https://console.aws.amazon.com/iam/home?#/security_credentials.\nGet Secret key and AWS IAM Console You access key ID will look something like AKIABOOMLWMXVM5DUA4G and secret access key MdrNmw6Ub+vHp2cBZKZLBbtY54XaAMufNnfW1Rz5. Both will be needed by Travis to access your AWS programatically.\nLet\u0026rsquo;s create a simple app # main.go file with driver code:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 package main import ( \u0026#34;fmt\u0026#34; \u0026#34;net/http\u0026#34; ) func main() { mux := http.NewServeMux() mux.HandleFunc(\u0026#34;/\u0026#34;, hello) http.ListenAndServe(\u0026#34;:8080\u0026#34;, mux) } func hello(w http.ResponseWriter, r *http.Request) { fmt.Fprintln(w, \u0026#34;Hello, world!\u0026#34;) } main_test.go file with test:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 package main import ( \u0026#34;net/http\u0026#34; \u0026#34;net/http/httptest\u0026#34; \u0026#34;testing\u0026#34; ) func TestHelloResponse(t *testing.T) { request, _ := http.NewRequest(http.MethodGet, \u0026#34;/\u0026#34;, nil) response := httptest.NewRecorder() hello(response, request) got := response.Body.String() want := \u0026#34;Hello, world!\\n\u0026#34; if got != want { t.Errorf(\u0026#34;got %q want %q\u0026#34;, got, want) } } While this is a simple server. I\u0026rsquo;ll do a tutorial on multi-container deployment when I\u0026rsquo;m upto it.\nDockerfile # Dockerfile for our repo:\n1 2 3 4 5 6 7 8 9 10 11 12 FROM golang:alpine ENV CGO_ENABLED=0 WORKDIR /go/src/github.com/santosh/unittest COPY . . RUN go get -d RUN GOOS=linux GOARCH=amd64 go build -a -installsuffix cgo -o app . EXPOSE 8080 CMD [\u0026#34;./app\u0026#34;] Tune up the .travis.yml # Add deploy section to .travis.yml:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 sudo: required services: - docker before_install: - docker build -t sntshk/untittest . script: - docker run sntshk/untittest go test deploy: provider: elasticbeanstalk access_key_id: $AWS_ACCESS_KEY secret_access_key: secure: $AWS_SECRET_KEY region: \u0026#34;ap-south-1\u0026#34; app: \u0026#34;unittest\u0026#34; env: \u0026#34;Unittest-env\u0026#34; bucket_name: \u0026#34;elasticbeanstalk-ap-south-1-086705419055\u0026#34; As you can see, we haven\u0026rsquo;t put access key and secret key directly. Travis provides a way to pass environment variables to running jobs. You can find it in the project settings. Below I have set both those variables.\nSet environment variable in project settings. Set the region in which you have created the application. Fill in the app, env and bucket_name appropriately.\nSet environment variable in project settings. End of the blog post. Please list your queries in comments. Also, subscribe to the newsletter for updates.\n","date":"18 May 2020","externalUrl":null,"permalink":"/posts/2020/ci-and-deployment-to-beanstalk-with-docker/","section":"Posts","summary":"In a previous post we discovered how can we run commands on each push and check if the test is passing against multiple versions of go. This is a continuation of the same concept. We are going to deploy our code to Elastic Beanstalk, straight from git push.\nElastic Beanstalk is nothing but a wrapper around it’s EC2 infrastructure. It is a developer-centric view of deploying an application on AWS. Developers just have to push the code and Beanstalk can take care of Auto Scaling Group, Elastic Load Balancer, databases etc. No need to manage infrastructure manually.\n","title":"Continuous Integration and Deployment to Beanstalk With Docker","type":"posts"},{"content":"","date":"18 May 2020","externalUrl":null,"permalink":"/tags/travis/","section":"Tags","summary":"","title":"Travis","type":"tags"},{"content":"","date":"10 May 2020","externalUrl":null,"permalink":"/tags/boltdb/","section":"Tags","summary":"","title":"Boltdb","type":"tags"},{"content":" Why use BoltDB # Bolt plays pretty well with Go\u0026rsquo;s concurrency model, we can do multiple reads concurrently.\nFor pro and cons, I will contrast it with similar database SQLite. I have done SQLite when I worked with Xentrix to develop GUI tools for the artists.\nYou may want to connect with me on LinkedIn.\nPro # It is native go. This means you don\u0026rsquo;t need any database server running, like SQLite. In oppose to Redis and memcached.\nCross-compilation One thing I see a lot appearing on the internet about Bolt is it\u0026rsquo;s an ability to cross-compile.\nConcurrent reads. Read operation does not lock the database. But for writing, one transaction should end before next to process.\nAvailable on platforms where Go is available. One thing to note is, BoltDB is a key-value datastore unlike SQLite, which is an RDBMS.\nCons # Go only. The fact that Bolt is native go, you can\u0026rsquo;t use it with other languages. In contrast, SQLite works with approx 2 dozen languages.\nIf you are here, probably you are curious how to start using Bolt in your personal projects. BoltDB is also used in production at Shopify and HashiCorp (the company which made Consul). Bolt is as reliable as underlying infrastructure.\nBuckets # Bolt store data in buckets. A Bucket is similar to a table in RDBMS. And similar to a collection in Document based stores.\nBasically what we can do now is connect to the database.\nConnecting to a Database # 1 2 3 4 5 db, err := bolt.Open(\u0026#34;bolt.db\u0026#34;, 0600, nil) if err != nil { log.Fatal(err) } defer db.Close() From the docs:\n1 2 3 4 // Open creates and opens a database at the given path. // If the file does not exist then it will be created automatically. // Passing in nil options will cause Bolt to open the database with the default options. func Open(path string, mode os.FileMode, options *Options) (*DB, error) { Third argument to bolt.Open is Options. Most widely used is Timeout. From the documentation:\n1 2 3 4 // Timeout is the amount of time to wait to obtain a file lock. // When set to zero it will wait indefinitely. This option is only // available on Darwin and Linux. Timeout time.Duration Initialize the Database # Before we start writing to the database, we need to set things up. We must check if bucket exists or not. We\u0026rsquo;re gonna do that in our first transaction.\n1 2 3 4 5 6 db.Update(func(tx *bolt.Tx) error { b, err := tx.CreateBucketIfNotExists(\u0026#34;todo\u0026#34;) if err != nil { return fmt.Errorf(\u0026#34;create bucket: %s\u0026#34;, err) } }) Inside the closure, you have a consistent view of the database. You commit the transaction by returning nil at the end. You can also rollback the transaction at any point by returning an error. All database operations are allowed inside a read-write transaction (the Update func).\nWriting and Updating to the Database # The same db.Update API can also be used to write new entries to the bucket using Bucket.Put. Something like this:\n1 2 3 4 db.Update(func(tx *bolt.Tx) error { b := tx.Bucket([]byte(\u0026#34;todo\u0026#34;)) return b.Put([]byte(\u0026#34;your key\u0026#34;), []byte(\u0026#34;your value\u0026#34;)) }) Batch read-write transactions\nEach DB.Update() waits for disk to commit the writes. This overhead can be minimized by combining multiple updates with the DB.Batch() function:\n1 2 3 4 db.Batch(func(tx *bolt.Tx) error { // ... return nil }) If you are looking for an autoincrement solution, you can use the Bucket.NextSequence API. There is whole example at https://github.com/boltdb/bolt#autoincrementing-integer-for-the-bucket\nAs we know, Bolt stores keys and values in byte slices. And to convert from integer and string to byte slices have different difficulty level. While string do back and forth with []byte(str) and string(byt). Interger needs some special care. This is from the docs:\n1 2 3 4 5 6 // itob returns an 8-byte big endian representation of v. func itob(v int) []byte { b := make([]byte, 8) binary.BigEndian.PutUint64(b, uint64(v)) return b } Reciprocal of this func would be this:\n1 2 3 func btoi(b []byte) int { return int(binary.BigEndian.Uint64(b)) } Not to mention, same Put API can be used to update the existing pairs. It\u0026rsquo;s CRUD you know. \u0026#x1f937;\nReading from the Database # Until now, we were using db.Update, now while reading from there we\u0026rsquo;ll use db.View. It support concurrent reads.\nGetting Value by Key # We can use Bucket.Get to get value of specific keys. As always, pass a byte slice.\n1 2 3 4 5 6 db.View(func(tx *bolt.Tx) error { b := tx.Bucket([]byte(\u0026#34;todo\u0026#34;)) v := b.Get([]byte(\u0026#34;your key\u0026#34;)) fmt.Printf(\u0026#34;The value field for \u0026#39;your key\u0026#39; is: %s\\n\u0026#34;, v) return nil }) The Get() function does not return an error because its operation is guaranteed to work (unless there is some kind of system failure). If the key exists then it will return its byte slice value. If it doesn\u0026rsquo;t exist then it will return nil. It\u0026rsquo;s important to note that you can have a zero-length value set to a key which is different than the key not existing.\nIterating over the Keys in Bucket # In some situation you might want to iterate over all the items. To iterate over the keys, you first have to acquire a Bucket.Cursor. Something like this:\n1 2 3 4 5 6 7 8 9 10 db.View(func(tx *bolt.Tx) error { b := tx.Bucket([]byte(\u0026#34;todo\u0026#34;)) // we need cursor for iteration c := b.Cursor() for k, v := c.First(); k != nil; k, v = c.Next() { // do stuff with each key value pair } // should return nil to complete the transaction return nil }) At last, I must recommend reading the docs as there are more goodies there. Reading the source code is also not bad, infact it is full of documentation.\nFor more updates, please subscribe to this blog below.\nRelated readings:\nhttps://awmanoj.github.io/2016/08/03/using-boltdb-as-a-fast-persistent-kv-store/ https://medium.com/wtf-dial/wtf-dial-boltdb-a62af02b8955 ","date":"10 May 2020","externalUrl":null,"permalink":"/posts/2020/introduction-to-boltdb/","section":"Posts","summary":"Why use BoltDB # Bolt plays pretty well with Go’s concurrency model, we can do multiple reads concurrently.\nFor pro and cons, I will contrast it with similar database SQLite. I have done SQLite when I worked with Xentrix to develop GUI tools for the artists.\nYou may want to connect with me on LinkedIn.\nPro # It is native go. This means you don’t need any database server running, like SQLite. In oppose to Redis and memcached.\n","title":"Introduction to BoltDB","type":"posts"},{"content":"","date":"10 May 2020","externalUrl":null,"permalink":"/tags/key-value/","section":"Tags","summary":"","title":"Key-Value","type":"tags"},{"content":"","date":"3 May 2020","externalUrl":null,"permalink":"/tags/code-coverage/","section":"Tags","summary":"","title":"Code-Coverage","type":"tags"},{"content":"You can\u0026rsquo;t think of deploying your application to production without testing it.\nNeither you can manage a large codebase with confidence without it.\nLet us go through some basics of unit testing in golang.\nThis post is structured in the following manner:\nunit testing basics in golang (jump) inbuilt code coverage command understanding subtests and helper function (jump) Travis CI integration (jump) running test against multiple version of go Please connect with me on LinkedIn and let\u0026rsquo;s get started.\nUnit Testing in golang # Before I start, I must say that do not test code, test the behavior.\nBecause code can be written in spaghetti code as well as [clean code][].\nMoreover, a unit test gives you confidence in messing up the code to refactor it.\nOur hello world code below in the file hello.go:\n1 2 3 4 5 6 7 8 9 10 11 package main import \u0026#34;fmt\u0026#34; func Hello(hame string) string { return \u0026#34;Hello, \u0026#34; + name + \u0026#34;!\u0026#34; } func main() { fmt.Println(Hello(\u0026#34;Chris\u0026#34;)) } How do we test it? We\u0026rsquo;ll create a hello_test.go file in the same directory.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 package main import \u0026#34;testing\u0026#34; // TestHello: By convention this function will be called // Test + function we\u0026#39;re testing. func TestHello(t *testing.T) { got := Hello(\u0026#34;Chris\u0026#34;) want := \u0026#34;Hello, Chris!\u0026#34; if got != want { t.Errorf(\u0026#34;got %q want %q\u0026#34;, got, want) } } Given above is one of the basic kinds of a test function.\nOne test function is testing a single function in main.go code.\nBut there might come a time where one func is not enough to cover all the lines code. We\u0026rsquo;ll see that later, for now, let\u0026rsquo;s run the test by running go test -v in shell:\n$ go test -v === RUN TestHello --- PASS: TestHello (0.00s) PASS ok _/home/sntshk/repos/unittest 0.004s I have already talked about test coverage in my last post.\nIn simple words, test coverage shows how much code is covered by unit tests.\n$ go test -cover PASS coverage: 100.0% of statements ok _/home/sntshk/repos/unittest 0.004s More on test and coverage # As we can see, at the current state, our code has 100% of coverage by tests, which is a really good sign.\nAs we add some new features to our Hello function, we\u0026rsquo;ll see that changing.\nI\u0026rsquo;ve changed hello.go to print \u0026ldquo;Hello, World!\u0026rdquo; when no name is passed:\n+const prefix = \u0026#34;Hello, \u0026#34; + func Hello(name string) string { - return \u0026#34;Hello, \u0026#34; + name + \u0026#34;!\u0026#34; + if name == \u0026#34;\u0026#34; { + name = \u0026#34;World\u0026#34; + } + return prefix + name + \u0026#34;!\u0026#34; } Let\u0026rsquo;s run the test coverage again:\n$ go test -cover PASS coverage: 66.7% of statements ok _/home/sntshk/repos/unittest 0.003s All of a sudden our tests are only covering 66% of the code.\nThis is because our if name == \u0026quot;\u0026quot; section is not being covered by any test.\nLet\u0026rsquo;s write some test cases and run the coverage again to check the status.\nSubtests # In our test, we can extend our same test function with the use of subtests. If you are coming from Python world, you can relate it to an actual instance method you write for unittest.TestCase inherited class.\nI will refactor and extend the test:\n17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 func TestHello(t *testing.T) { t.Run(\u0026#34;saying hello to people\u0026#34;, func(t *testing.T) { got := Hello(\u0026#34;Chris\u0026#34;) want := \u0026#34;Hello, Chris\u0026#34; if got != want { t.Errorf(\u0026#34;got %q want %q\u0026#34;, got, want) } }) t.Run(\u0026#34;say \u0026#39;Hello, World\u0026#39; when an empty string is supplied\u0026#34;, func(t *testing.T) { got := Hello(\u0026#34;\u0026#34;) want := \u0026#34;Hello, World!\u0026#34; if got != want { t.Errorf(\u0026#34;got %q want %q\u0026#34;, got, want) } }) } Here I\u0026rsquo;ve used t.Run which takes name of the test as the first argument.\nThe second argument is a closure function. It is the function where the test body exists.\nName of the test is visible when you run the test. You\u0026rsquo;ll see this text in next test run.\nSubtests are also useful when multiple tests cases require same prequisites or post-requisites.\nIn some languages this can be achieved by using setup and teardown functions/methods.\nIn golang the code before first t.Run is setup and code after last t.Run teardown.\nWhich I personally find beautiful.\nLet\u0026rsquo;s see how our tests are doing.\n$ go test -v === RUN TestHello === RUN TestHello/saying_hello_to_people === RUN TestHello/say_\u0026#39;Hello,_World\u0026#39;_when_an_empty_string_is_supplied --- PASS: TestHello (0.00s) --- PASS: TestHello/saying_hello_to_people (0.00s) --- PASS: TestHello/say_\u0026#39;Hello,_World\u0026#39;_when_an_empty_string_is_supplied (0.00s) PASS ok _/home/sntshk/repos/unittest 0.003s Another reason to name test runners are that you can you can only run those runners by passing it to -run. As shown below:\n$ go test -v -run=TestHello/saying_hello_to_people === RUN TestHello === RUN TestHello/saying_hello_to_people --- PASS: TestHello (0.00s) --- PASS: TestHello/saying_hello_to_people (0.00s) PASS ok _/home/sntshk/repos/unittest 0.004s Let\u0026rsquo;s see the coverage.\n$ go test -cover PASS coverage: 100.0% of statements ok _/home/sntshk/repos/unittest 0.003s Our coverage is back up again.\nAs an end note for coverage, 100% test coverage is not always possible.\nBut don\u0026rsquo;t just simply ignore it. But more is good.\nYou can tell the effectiveness of the tests in the increasing codebase.\nHelper Functions # We can also use the helper function to dry up the code.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 func TestHello(t *testing.T) { assertCorrectMessage := func(t *testing.T, got, want string) { t.Helper() if got != want { t.Errorf(\u0026#34;got %q want %q\u0026#34;, got, want) } } t.Run(\u0026#34;saying hello to people\u0026#34;, func(t *testing.T) { got := Hello(\u0026#34;Chris\u0026#34;) want := \u0026#34;Hello, Chris\u0026#34; assertCorrectMessage(t, got, want) }) t.Run(\u0026#34;empty string defaults to \u0026#39;World\u0026#39;\u0026#34;, func(t *testing.T) { got := Hello(\u0026#34;\u0026#34;) want := \u0026#34;Hello, World\u0026#34; assertCorrectMessage(t, got, want) }) } We must use t.Helper() is those such functions. These helper functions can also reside outsite the main test function e.g. outside TestHello in this case.\nContinuous Integration with Travis CI # Continuous Integration is the practice of merging in small code changes frequently -\nrather than merging in a large change at the end of a development cycle.\nThe goal is to build healthier software by developing and testing in smaller increments.\nThis is where Travis CI comes in. This is not the only tool in the market.\nWhen you run a build, Travis CI clones your GitHub repository into a brand-new virtual environment, and carries out a series of tasks to build and test your code. If one or more of those tasks fail, the build is considered broken. If none of the tasks fail, the build is considered passed and Travis CI can deploy your code to a web server or application host.— Travis-CI Docs At this point you should have already pushed your repo to some remote like GitHub or GitLab.\nThe minimum .travis.yml should must have the language and script to run for test.\n1 2 3 language: go script: go test -v This is what happens when you git commit and git push your code.\nPassing test on Travis I consider adding some customizations for the sake of my optimization OCD.\n1 2 3 4 5 6 7 8 9 10 language: go go: - 1.14 git: depth: 1 # default is 50 quiet: true script: go test -v Builds can also fail. Below is an example of a failed build.\nFailing test on Travis The build is considered failed when a command in the script phase returned non-zero exit code.\nRun test against multiple version of go # To test against multiple go versions, I will pass more version to go key. This works the same way in other languages.\n3 4 5 go: - 1.13 - 1.14 Below is how it looks on Travis Dashboard. Our build launched 2 Jobs for each version.\nBuild with 2 Jobs on Travis There are many more things a CI provider can do. Like you can push a docker image if your test passes. Or you can deploy to Elastic Beanstalk or another cloud provider on a passed test. Or you can launch builds on each pull request which is sent on your public repo.\nThis is just to scratch the surface. I will post more as I use this service. To stay updated when new post comes, please subscribe to my newsletter below.\nMore information on enabling Travis in your repo is available at https://docs.travis-ci.com/user/tutorial/\n","date":"3 May 2020","externalUrl":null,"permalink":"/posts/2020/unit-testing-test-coverage-and-ci-in-go/","section":"Posts","summary":"You can’t think of deploying your application to production without testing it.\nNeither you can manage a large codebase with confidence without it.\nLet us go through some basics of unit testing in golang.\nThis post is structured in the following manner:\nunit testing basics in golang (jump) inbuilt code coverage command understanding subtests and helper function (jump) Travis CI integration (jump) running test against multiple version of go Please connect with me on LinkedIn and let’s get started.\n","title":"Unit Testing, Test Coverage and CI with Travis in Go","type":"posts"},{"content":"","date":"3 May 2020","externalUrl":null,"permalink":"/tags/unittest/","section":"Tags","summary":"","title":"Unittest","type":"tags"},{"content":" Me Before # 1 2 3 4 5 6 7 8 9 10 11 12 FROM golang:alpine WORKDIR /go/src/github.com/santosh/qagine COPY . . RUN go get -d RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -installsuffix cgo -o app . EXPOSE 8080 CMD [\u0026#34;./app\u0026#34;] Me building the image.\n$ docker build -t sntshk/qagine . Sending build context to Docker daemon 259.6kB Step 1/7 : FROM golang:alpine ---\u0026gt; 760fdda71c8f Step 2/7 : WORKDIR /go/src/github.com/santosh/qagine ---\u0026gt; Running in e2cf75dd1655 Removing intermediate container e2cf75dd1655 ---\u0026gt; 8b4955b2e885 Step 3/7 : COPY . . ---\u0026gt; 43cd9a41efcb Step 4/7 : RUN go get -d ---\u0026gt; Running in 55c01f67fa99 go: downloading github.com/gorilla/mux v1.7.4 go: downloading go.mongodb.org/mongo-driver v1.3.2 go: downloading github.com/dgrijalva/jwt-go v3.2.0+incompatible go: downloading golang.org/x/crypto v0.0.0-20200420201142-3c4aac89819a go: downloading github.com/pkg/errors v0.8.1 go: downloading github.com/xdg/scram v0.0.0-20180814205039-7eeb5667e42c go: downloading github.com/golang/snappy v0.0.1 go: downloading github.com/go-stack/stack v1.8.0 go: downloading github.com/klauspost/compress v1.9.5 go: downloading golang.org/x/sync v0.0.0-20190423024810-112230192c58 go: downloading github.com/xdg/stringprep v0.0.0-20180714160509-73f8eece6fdc go: downloading golang.org/x/text v0.3.2 Removing intermediate container 55c01f67fa99 ---\u0026gt; 038b29b80653 Step 5/7 : RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -installsuffix cgo -o app . ---\u0026gt; Running in 122df20d3329 Removing intermediate container 122df20d3329 ---\u0026gt; f4616dc3fe1c Step 6/7 : EXPOSE 8080 ---\u0026gt; Running in 4258f9aac8c6 Removing intermediate container 4258f9aac8c6 ---\u0026gt; b643c7e16c0b Step 7/7 : CMD [\u0026#34;./app\u0026#34;] ---\u0026gt; Running in 948fa65b093b Removing intermediate container 948fa65b093b ---\u0026gt; 621b8e5396ad Successfully built 621b8e5396ad Successfully tagged sntshk/qagine:latest The sweet size of the image is 525MB.\n$ docker images REPOSITORY TAG IMAGE ID CREATED SIZE sntshk/qagine latest 8e966b835d82 7 seconds ago 525MB That\u0026rsquo;s because we installed all the dependencies my application needs on top of the golang base image.\nMe After # 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 FROM golang:alpine WORKDIR /go/src/github.com/santosh/qagine COPY . . RUN go get -d RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -installsuffix cgo -o app . FROM alpine:latest RUN apk --no-cache add ca-certificates WORKDIR /go/src/github.com/santosh/qagine COPY --from=0 /go/src/github.com/santosh/qagine/app . EXPOSE 8080 CMD [\u0026#34;./app\u0026#34;] Me running my recipe of our latest Dockerfile. I\u0026rsquo;m proud that I made that choice. \u0026#x1f60b;\n$ docker build -t sntshk/qagine . Sending build context to Docker daemon 259.6kB Step 1/11 : FROM golang:alpine ---\u0026gt; 760fdda71c8f Step 2/11 : WORKDIR /go/src/github.com/santosh/qagine ---\u0026gt; Using cache ---\u0026gt; 8b4955b2e885 Step 3/11 : COPY . . ---\u0026gt; 824d443e02d4 Step 4/11 : RUN go get -d ---\u0026gt; Running in d6927d1b21f8 go: downloading github.com/gorilla/mux v1.7.4 go: downloading go.mongodb.org/mongo-driver v1.3.2 go: downloading golang.org/x/crypto v0.0.0-20200420201142-3c4aac89819a go: downloading github.com/dgrijalva/jwt-go v3.2.0+incompatible go: downloading github.com/go-stack/stack v1.8.0 go: downloading github.com/golang/snappy v0.0.1 go: downloading github.com/klauspost/compress v1.9.5 go: downloading github.com/pkg/errors v0.8.1 go: downloading github.com/xdg/stringprep v0.0.0-20180714160509-73f8eece6fdc go: downloading github.com/xdg/scram v0.0.0-20180814205039-7eeb5667e42c go: downloading golang.org/x/sync v0.0.0-20190423024810-112230192c58 go: downloading golang.org/x/text v0.3.2 Removing intermediate container d6927d1b21f8 ---\u0026gt; 9edacd1c2f9a Step 5/11 : RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -installsuffix cgo -o app . ---\u0026gt; Running in b6012db1fde8 Removing intermediate container b6012db1fde8 ---\u0026gt; 0591bf0229cf Step 6/11 : FROM alpine:latest ---\u0026gt; f70734b6a266 Step 7/11 : RUN apk --no-cache add ca-certificates ---\u0026gt; Using cache ---\u0026gt; 50dab5d0feaa Step 8/11 : WORKDIR /go/src/github.com/santosh/qagine ---\u0026gt; Running in 8136cd07627f Removing intermediate container 8136cd07627f ---\u0026gt; 900601534a07 Step 9/11 : COPY --from=0 /go/src/github.com/santosh/qagine/app . ---\u0026gt; 260353317972 Step 10/11 : EXPOSE 8080 ---\u0026gt; Running in b445d455c9c1 Removing intermediate container b445d455c9c1 ---\u0026gt; 284de5350f3e Step 11/11 : CMD [\u0026#34;./app\u0026#34;] ---\u0026gt; Running in 3d26b0b7ad1d Removing intermediate container 3d26b0b7ad1d ---\u0026gt; f63a288ada1e Successfully built f63a288ada1e Successfully tagged sntshk/qagine:latest I\u0026rsquo;m telling you, you\u0026rsquo;ll be surprised by the size of the image.\n$ docker images REPOSITORY TAG IMAGE ID CREATED SIZE sntshk/qagine latest a60a5c6a5722 About a minute ago 21.9MB Explanation # In the first Dockerfile we are taking a base image of golang:alpine which has its own weightage (128MB as listed on https://hub.docker.com/_/golang?tab=tags). On top op that dependency to the application is being installed, which must be quite an amount of space. Finally an executable is generated with its own space needs.\nIn the second run, we are doing the same. But at the last, we are extracting the executable from the first image and wrapping it inside a brand new alpine image. So this image has only the weight of alpine image (around 5MB?) and the executable.\nI am now smarter than yesterday.\nHope this helps you as well. If it did, please subscribe below to get updates about my new posts.\n","date":"1 May 2020","externalUrl":null,"permalink":"/posts/2020/create-lightweight-docker-images-with-multi-staged-build/","section":"Posts","summary":"Me Before # 1 2 3 4 5 6 7 8 9 10 11 12 FROM golang:alpine WORKDIR /go/src/github.com/santosh/qagine COPY . . RUN go get -d RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -installsuffix cgo -o app . EXPOSE 8080 CMD [\"./app\"] Me building the image.\n$ docker build -t sntshk/qagine . Sending build context to Docker daemon 259.6kB Step 1/7 : FROM golang:alpine ---\u003e 760fdda71c8f Step 2/7 : WORKDIR /go/src/github.com/santosh/qagine ---\u003e Running in e2cf75dd1655 Removing intermediate container e2cf75dd1655 ---\u003e 8b4955b2e885 Step 3/7 : COPY . . ---\u003e 43cd9a41efcb Step 4/7 : RUN go get -d ---\u003e Running in 55c01f67fa99 go: downloading github.com/gorilla/mux v1.7.4 go: downloading go.mongodb.org/mongo-driver v1.3.2 go: downloading github.com/dgrijalva/jwt-go v3.2.0+incompatible go: downloading golang.org/x/crypto v0.0.0-20200420201142-3c4aac89819a go: downloading github.com/pkg/errors v0.8.1 go: downloading github.com/xdg/scram v0.0.0-20180814205039-7eeb5667e42c go: downloading github.com/golang/snappy v0.0.1 go: downloading github.com/go-stack/stack v1.8.0 go: downloading github.com/klauspost/compress v1.9.5 go: downloading golang.org/x/sync v0.0.0-20190423024810-112230192c58 go: downloading github.com/xdg/stringprep v0.0.0-20180714160509-73f8eece6fdc go: downloading golang.org/x/text v0.3.2 Removing intermediate container 55c01f67fa99 ---\u003e 038b29b80653 Step 5/7 : RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -a -installsuffix cgo -o app . ---\u003e Running in 122df20d3329 Removing intermediate container 122df20d3329 ---\u003e f4616dc3fe1c Step 6/7 : EXPOSE 8080 ---\u003e Running in 4258f9aac8c6 Removing intermediate container 4258f9aac8c6 ---\u003e b643c7e16c0b Step 7/7 : CMD [\"./app\"] ---\u003e Running in 948fa65b093b Removing intermediate container 948fa65b093b ---\u003e 621b8e5396ad Successfully built 621b8e5396ad Successfully tagged sntshk/qagine:latest The sweet size of the image is 525MB.\n","title":"Create Lightweight Docker Images With Multi Staged Build","type":"posts"},{"content":"","date":"1 May 2020","externalUrl":null,"permalink":"/tags/images/","section":"Tags","summary":"","title":"Images","type":"tags"},{"content":"Hi, my name is Santosh Kumar. I’m am working in a startup called NuNet as a Fullstack Developer.\nAs a Fullstack Developer at NuNet, my task is to develop a distributed application which can distribute computing load on multiple hosts. The primary language we use is Go. Me with my team use Docker, Firecracker, IPv8 to develop this.\nBackstory # From the start, I have been self-taught. I started with writing static web pages at school. Then I felt the need to adding dynamic features to those pages, so I learned PHP which was dominating the market at that time.\nI grew a lot of attraction towards computer programming and open-source when I read How to Become a Hacker by Eric Raymond. I started practicing Python and Linux from way back then. That document also emphasizes on community practice and for so I have been active in communities like GitHub and StackOverflow. I got an enormous chance to enhance my Python skills in college since most of DCC packages already used it.\nWith passion for web development, I chased technologies related to web.\nIn recent years, I have picked up golang because its simplicity. I am also seasoned at Python. And I can also do client side programming in JavaScript using framework such as Vue and Vuetify.\nThis website is my personal space where I share experiences of my personal and professional life.\nContact # Below is my public key if you want to send me a secure message.\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 -----BEGIN PGP PUBLIC KEY BLOCK----- mQINBGFnNasBEADjFx9g7oaIIeAK8GgtvqnVFEZ9EubSi9q9F7idzB7JNm/mwS9c QnaqJu9rEpDxZOwKpu9n2meoin8dXNybFumDdrNAj75KNT9eW85PwvTDdn6peYu+ fxwqPey/oFVx1xNWcQqxKI3Uku8F1JvDoUjmG/zQJ0gFbhCYM+VeJ1/4J8yGPmxu T6AlhAsG9s1eENlgat0G+XfSPA4DblP9o9vTS9ljd4WitSe0DVy4Vkk6H71xgGsj h7o0V6ui8iYT2fFvh87GIopuXcsK6+gbFoyNIcFA42SjD9g/vil3JLSrqoDsMgpn I6aGfED2Gk2PaQQav9JfhrbPCdh/qW+yB3eVl2Dd+jDsILaSSitN/COr4S4UKYAR 9pvpRxtW3hY6yHtgeV75A1CgLuATL72kPIeZ+f36G7FU30912QX5MWGhqB239ITJ NrH1Ii46YjTte7q24+KAtl6L+FNdnjuHjpGIAYB2zV322XaK8xJDzmVFrcfC7uHd LVkLm4JwEkwbjesvtqGjN+C8Z2i5qK9crqnqpZX2+evOZozm3OamVAMp0gi0Nu3W YzllfjFgfJiEtgJDbtps8dVXasZif2cUpaGa0OETUDkQbIL1FNvlVuxegzX1F0tw p/JJPsYyZeCIg3E+GM9T10GUyminBBn6qAjnJ+1z+fHZckOqgVAzH01n4QARAQAB tCRTYW50b3NoIEt1bWFyIDxzbnRzaGttcjYwQGdtYWlsLmNvbT6JAk4EEwEIADgW IQRlz7pnyObmszh40HL/xlimwsKabQUCYWc1qwIbAwULCQgHAgYVCgkICwIEFgID AQIeAQIXgAAKCRD/xlimwsKabWAMD/wLO2b30IwuhGE3V52d5z46DapNEyeNWRIb BfZQgHl8zIp4Er3lfoOR3WUBxo+qXqdgB/Q/DtC1ZVIMlgsqHmTJx1Dmm2qB5mNT sQKwCeidqsWsJVNWzYj7ISXXcWyQrO4BfCbfyP69RIiFaSh3tjsLmqYFdV87V0Qs almBzW8RHQDkoDC5mRVhpgchKTLIPgtL+PeBy8HqZA/1HBmlr4erO7wqF7oMDGZh I7fnLRoOnT/b/jBGwg/8nZuXYMDO5WG+czk9gcg3gbAbuH9/Pkpq1bxS8xMB27zS YgMkHe/yzAahCVtvih7p9nuNRpTB8rcXCMQi/B3v+j5LEv9YWJaGHijfBl0/RTe6 SU2GG0f3IHsbO3IGYK5fhTRIimLW+/sMZ9P3SGDmU4kQ2uDXw/Y92aMwzwZhJ0Yg TJQhtFid4mf+hNqpY61vzfHRelz5q+isQd3zNluuCl5Y9nfoI/mQqFBTEdn2SmWw JFlHTSfcsbT5t/y+BFFUYCKVX0jf6dZwF+kPJRvGwIIJL+zeDWqQJlH1Q+XakDLr u1g1+RQ35633jDaFEYPm6/z5U2DHihxJjgyhCEtnfUp4rd7/PPcpQ4wHbWg1eGeT LWTAL78cZERt4DPTL2N0qgIpG5P85g3vznOskW6/UZaVlKvWg1VPTlI0CKlXhNcH 4TJxvDLPRLkCDQRhZzWrARAAs9rzGz+9KtdOYosfRib5edO9Cs9B+6npZZlPkN0l g4AFvrBpDYErZBaDBGTMaOFy+h1z3Ns6q9AXXCOqv60rv4pVi44orZHrkm0W5MI/ GFY4GhSQBHAR6j7eVB41ADZvevwlzTiB/XCZG6K4jboraZYVw6fUvCL2zaSzdtqN FmDDE8FlMG3vyi3dpTNyfFwGjLH1e5mIbd+XUomFfEC4v2wF6L3azYuVb2SMMI8k n2p1vN1Yz7jaFrw6UG9sU6ksLa4upjE67UPY25IQxwX2DgmavZ97Yyp1WICZGCzX +ct5mQRSj3o+TOB/AGvsSDF6ViguqGUcqC+dnZCzcOihYQlHzHlGSLS6Rt+22Lq1 VpAZ45ACf/WnNnnjdkz9zttHIWS+XG4pTuTsq83WSB1uurI2u6GscuT6eep7aAm7 hMfU/DnTjEmUEW4/1sYB2v1SdjjAs0dYUdXP4SVgaLv2mgHt1GWdjuUG+hYTbWXq QaWflVqx698L0rb+5spFb4XCfCHLzSB9wbsYKHf4E3VpXKLrS2+80vrBRfkdPji3 Gcoake3Vuh0t1KxJjtbnIgNQhPwfMimjpWhCTGg4oSqI8x6c6qlatdskhWGFF8EA /eR1TgXXq9FAp0NoO763QAaJri6SdT159MGjVUIbuNwgw6DzvfBeyhfBxa/ZzIDY GWUAEQEAAYkCNgQYAQgAIBYhBGXPumfI5uazOHjQcv/GWKbCwpptBQJhZzWrAhsM AAoJEP/GWKbCwppt9kUQAMLBO+5CVI3mb3eXC6T6CxwJvqg7boO0XD3e5ki5Nlwj lG5UyiN+GZVql4qqibisVtnkPRHs5bNLKfQ/GAazBC2mikzrZFqfoTmlk82MpIfF vvS/Gk1c4c2u3AGDbjCdf6GW0ludHUucsDFalJ6VR4/zRO7QEBtmTJi0QNQDaDUS jV+xTK8PQaury5O3zSXCSwGUVY2pPTymDHz3kxj8ZP+kctJLiH5goqdq2BKwve85 RkyL1PtMa25Ma/P+9bbhZJPkrsSsZa1xCPbZBcD9KbmWFda3azBdqQVDK9NcS/9s tVbmcZU0p7cbXpJWWsvSIEQjcjJsoXhvvwno3HufFa+E1qsENpbRAmHt790WrArf WWdx9oejnmtzRsnYZSxyspHqZAGheTg0xwYYZMocGJ8nfa4ziQTw40Q+sM2Wxi66 cCoOme0IijDq59ZRnDbsNHaF9bqSWfdahi+7U95+DFFnyxiEyE+9cqo2ycz5DNJL uCfZ9XpEGUwEM6nllYG5I0I6YowoAsx+ngb6GvS/PTMbVhQbI8qqtr7WHfZB5oRJ aLIknGwf674fkJt52XdpOXFbFkN1q+kyIVRirNlI+rVE7C8yt11bMI9OerNooAkQ B9sDl41Vscq24x+6Jver7W9oHsJWMN8pk0kCJtEBsXatCiaDSyVBVd6jOxh82YDU =Yp/w -----END PGP PUBLIC KEY BLOCK----- ","date":"4 January 2020","externalUrl":null,"permalink":"/about/","section":"Fullstack with Santosh","summary":"Hi, my name is Santosh Kumar. I’m am working in a startup called NuNet as a Fullstack Developer.\nAs a Fullstack Developer at NuNet, my task is to develop a distributed application which can distribute computing load on multiple hosts. The primary language we use is Go. Me with my team use Docker, Firecracker, IPv8 to develop this.\nBackstory # From the start, I have been self-taught. I started with writing static web pages at school. Then I felt the need to adding dynamic features to those pages, so I learned PHP which was dominating the market at that time.\n","title":"About Me","type":"about"},{"content":"This year was a transition period for me from a career perspective. I was hearing, understanding and practicing stuff like microservices, fault tolerance, cross-site deployment, REST APIs, software deployment system, agile development and attending meetings \u0026amp; training sessions with the people on the other side of the planet \u0026#x1f605; . 3 years back I never thought I would end up being some place like this. That version of me just wanted to be a compositor.\nEnough of blabbering, please connect with me on LinkedIn and come see what I\u0026rsquo;ve been through this year.\nCode Review # As a Junior Pipeline Developer at Method Studios, it was first time my code was going to be part of some multinational corporate codebase. Well, code reviews can be cruel at times, but I can say, getting hurt act as a catalyst in changing one\u0026rsquo;s life. Someone with 2-5x experience than me can see how my code can break create regression in the future which will create a headache in debugging.\nWriting any software is just 10% of the job. Rest 90% is maintaining it. Yes, you can write code and it will work. Have you documented your code? Have you written tests? Is your coding consistent over entire project? If your answer is no to all of these; no problem. Your code will still work. It will work even if you leave the company. But the person who comes after you will for sure curse you.\nFor me, one such case was to avoid the use of global variables. So the story is like this: I used as much global variables as possible\nCode review can prevent these headaches right at the start. It also gives us a chance to hone our skills, systematically.\nWhile writing code, make sure to keep the human being in mind who will be reading that in the future. Also, assume that they will be dumb (no offense).\nSeparate Client and Library Code # While reading through software packages this year, I started to see a pattern in almost every package. Packages used to have an api.py file that has no implementation. I learned to separate the library code and client code. This library code can be later be reused by other packages without caring about how the frontend has been developed. This pattern was always there in web application like Twitter, Facebook (or whatever services which open their API). They use the same API to to make their own application which all the public use. But also the API can be used from the third party like we developer. Better late than never.\nLesson learned: Expose the API so other similar packages can use it.\nSeparate Configuration from Application Code # Although I have seen this already in software like text editors, DCC softwares, terminals and pretty much every thing (which had some sort of configs system) had one thing in common, they had a system configuration, then there was a user configuration (on Linux this is in $HOME and $HOME/.config). This pattern is extended by text editors and IDEs to have a separate project configuration.\nAs a multinational company, Method has multiple facilities. And I saw this pattern being extended to have facility level configuration, so that each facility has some freedom in choosing optimal settings for them. Reasons may include limit access to resources, but it varies. This pattern is extended to fulfill some domain related needs which I not need to discuss.\nYou ship your code with default application settings, let people create their own. You maintain a map of settings and keep overwriting localized setting over the global one.\nBoltDB # BoltDB is analogical to what SQLite is for SQL servers but key/value databases. I stumbled upon it when I was making a URL shortener. I\u0026rsquo;ve heard some nice stuff about this and look forward to using it for smaller applications.\nBut there is one bad news. Currently, it\u0026rsquo;s Go only.\nTest Coverage # I don\u0026rsquo;t exactly remember by whom I was told to break larger functions into smaller ones. It was just too much of overkill to break them just for sake of readability. Now my mindset has changed a little bit. Now there\u0026rsquo;s one more reason to break them down, for test coverage. It is a lot easier to write tests for smaller units and increase the test coverage than to write tests for longer functions and sacrifice coverage.\nKeyboard Layout # The keyboard layout you are using was designed around the time light bulb was invented by Thomas Edison. At that time there were no computers, and the layout was used in typewriters. Later on, when computers came out, the same layout was used for keyboards.\nIt is 2020 now, and we have already sent so many probes in deep space. See, how far we have got.\nIf you are a keyboard guy, I would highly recommend switching to one of the modern layouts. It\u0026rsquo;s a one-time investment. Do mercy to your fingers. Typing on QWERTY is redundant. What\u0026rsquo;s the profit of so much of technological advancements if your fingers are living in the 1870s?\nI use Colemak, but there are other modern layouts out there. Each offers benefits over the other, but none of them are as easier as Colemak. It only swaps 17 from the QWERTY layout, means it is a lot easier to switch to Colemak than Dvorak (moves 33 keys) and Workman\u0026hellip;\nIf you liked this post, please subscribe to the newsletter below for more updates like this.\n","date":"3 January 2020","externalUrl":null,"permalink":"/posts/2020/2019-in-review/","section":"Posts","summary":"This year was a transition period for me from a career perspective. I was hearing, understanding and practicing stuff like microservices, fault tolerance, cross-site deployment, REST APIs, software deployment system, agile development and attending meetings \u0026 training sessions with the people on the other side of the planet 😅 . 3 years back I never thought I would end up being some place like this. That version of me just wanted to be a compositor.\n","title":"2019 In a Review: Colemak, Code Reviews and More","type":"posts"},{"content":"","date":"25 December 2019","externalUrl":null,"permalink":"/tags/migration/","section":"Tags","summary":"","title":"Migration","type":"tags"},{"content":"","date":"25 December 2019","externalUrl":null,"permalink":"/categories/stories/","section":"Categories","summary":"","title":"Stories","type":"categories"},{"content":"Finally I have decided to buy me a domain. But before that I must keep my content ready.\nEarlier on, I was decided on to giving my website a go backend, but I later thought it was an overkill to use golang as a backend for a personal website and I should keep that enthusiasm for a real app. Instead of manually creating a frontend for my site, I will be using already available Hugo theme. I will be migrating my blog which I\u0026rsquo;m maintaining at codicious.wordpress.com and fxcious.wordpress.com.\nConnect with on LinkedIn.\nReasons # Hugo has already done a great job to steal users from WordPress. Reasons are already well known, but I\u0026rsquo;ll jot it down briefly. Here are the points:\nIt\u0026rsquo;s free, no hosting to pay for. Not only Hugo is free, it can be hosted free too. I\u0026rsquo;ll be using GitHub Pages or Netlify. No server side dynamics. As there are no server side dynamics involved, there is less attack vector. I have no real gain to migrating to the Hugo, I\u0026rsquo;m just going with the trend. Also, by using Hugo, I\u0026rsquo;ll be able to keep close to the go community which already is a language of 2020s.\nConcerns # SEO Crisis - It\u0026rsquo;s not a big problem to me at the time of migration as I have not a lot of visitor. But if your site have a lot of traffic, you might wanna keep your old installation of WordPress inact to give search engines a transition to keep your SEO rank maintained.\nUpdate: Google has nice tools and tips to avoid this SEO. To avoid it, one should register your new domain on Google Webmaster Tools.\nImage Captions - Image captions would go away.\nUpdate: Use figure shortcode. You can always create custom shortcode.\nAnalytics - In wordpress hosted version, we can see how many people are visiting us, and where they are from and verious democratic data.\nUpdate: Use Google Analytics. There\u0026rsquo;s a inbuilt template which you can include in your header after writing anatytics id to config.toml. This has a drawback. When you see your website yourself (local development), you put a lot of shit out there on the Google Analytics dashboard. There are a lot of workaround though.\nRSS Feed -\nMigration Process # I went through a lot a blog posts to see what people have already faced during and after migration. But whatever you say man, writing blog in vim is fun. \u0026#x1f609;\nAmong others is a WordPress to jekyll blog engine converter exitwp.py (also works fine with Hugo). It is written in Python and works on exported xml file. Download the repo:\n1 2 git clone https://github.com/thomasf/exitwp.git cd exitwp Enable a virtual environment and install the requirements:\n1 2 3 pipenv --two pipenv shell pip install -r pip_requirements.txt Keep the xml file in wordpress-xml directory (can put multiple files), and:\n1 python ./exitwp.py Copy the created .markdown files in _posts directory to to /contents folder.\nI also renamed all .markdown extension to .md:\n1 find . -type f -exec rename .markdown .md {} \\; Other method includes wordpress-to-hugo-exporter. This method assumes a install of WordPress on your own server. One thing I was not aware of before migration was, even if you don\u0026rsquo;t host your blog at your own, you can download the xml from your wordpress.com website and import it to local install and proceed as a normal. I haven\u0026rsquo;t tried this method.\nIf you have a nodejs background, you might like blog2md which also works on exported xml file. Modify wordpressImport according to will.\nNo matter what script you use for migration, it will not give you the desired result. There will be some tune up needed, which I\u0026rsquo;ve discussed in cleanup and tuning process. But just to let you know, the site just runs if you run it in deployment. So maybe one you can proceed to deployment section and publish it first then continue back from here.\nAlso, I found out that drafts are not converted to hugo format. And I had to manually copy paste the content.\nNote on Comment Migration # Although I\u0026rsquo;ve not tried, Alessandro Bahgat in his blog post says:\nDisqus comments are associated with page URLs, so you will want to make sure your pages are served at the same URLs as before the migration. Alternatively, you can edit the URLs in the export file before importing it following the instructions above.\nNote on Image Migration # Downscale images. 1 2 find . -name \u0026#34;*.jpg\u0026#34; -exec jpegoptim -m80 -o -p --strip-all {} \\; find . -name \u0026#34;*.png\u0026#34; -exec optipng -o7 {} \\; Git LFS should be enabled to keep git repo efficient as git is not meant to track binary files. 1 2 git lfs install git lfs track \u0026#34;*.jpg\u0026#34; For me, images came up like this after conversion: 1 2 3 [caption id=\u0026#34;attachment_229\u0026#34; align=\u0026#34;aligncenter\u0026#34; width=\u0026#34;234\u0026#34;] [![Pivot Point Menu](https://fxcious.files.wordpress.com/2018/06/pivot-point-menu.jpg)](https://fxcious.files.wordpress.com/2018/06/pivot-point-menu.jpg) Pivot Point Menu [/caption] It needs to be converted like this:\n(Are you wondering why only the above piece of code is hosted on gist and rest natively? Seems like hermit does not support shortcode inside code blocks; or maybe it is by desigh, who knows).\nIf you have featured image per post, it goes like this in front matter: 1 2 images: - https://picsum.photos/1024/768/?random These images is also used in Twitter Cards and the Open Graph metadata.\nUse relative url from static folder. Cleanup Process # No matter what method you used above for data extraction, your isn\u0026rsquo;t going to be perfect. You might semi-automate some of these with tools like or text editor.\nFront Matter Cleanup # This is how a typical blog post looked like:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 author: sntshkmr60 showComments: true date: 2019-09-01 15:28:00+00:00 excerpt: \u0026lt;stripped\u0026gt; layout: post link: https://codicious.wordpress.com/2019/09/01/run-unit-test-before-every-commit-with-pre-commit-hook/ slug: run-unit-test-before-every-commit-with-pre-commit-hook title: Run Unit Test Before Every Commit with pre-commit Hook wordpress_id: 306 categories: - software-development tags: - deployment - git - webdev Obviously I don\u0026rsquo;t need all of them. Following are the things I\u0026rsquo;m planning to throw away:\nauthor - As I\u0026rsquo;m the only author of this blog, it\u0026rsquo;s redundant. wordpress_id, excerpt, link, layout - I don\u0026rsquo;t find any use of them. categories - categories are by default disabled, tags are enough though. slug - title is enough to generate URLs 1 sed -i \u0026#39;/^slug:/d\u0026#39; content/posts/* Janik Vonrotz has covered a lot of these automation if not all in his blog post: https://janikvonrotz.ch/2018/07/08/another-wordpress-to-hugo-migration/\nContent Cleanup # I checked if code blocks are converted properly and placed correctly (they were not). Convert youtube, twitter, image shortcodes to Hugo equivalent. See https://gohugo.io/content-management/shortcodes/ Check internal links, make them relative. For images, I\u0026rsquo;ve replaced https://codicious.files.wordpress.com/ and https://fxcious.files.wordpress.com/ with /uploads/ wherever image is being referenced. And downloaded and kept them inside /static/uploads folder. Inside uploads directory I\u0026rsquo;ve maintained YYYY and MM subdirectories so that one single directory is not cluttered up. We missed # I didn\u0026rsquo;t think of these parameters but these also matters. I have not implemented them yet. I\u0026rsquo;m planning to work on it when the site has started rolling.\nSearch (pending) # Many Hugo users employ Google Custom Search. But it contains ads (in the free edition). That’s why I’m using a simple HTML form and redirect to Google Search but restrict the search to my site.\n1 2 3 4 5 \u0026lt;form role=\u0026#34;search\u0026#34; method=\u0026#34;get\u0026#34; action=\u0026#34;https://www.google.com/search\u0026#34;\u0026gt; \u0026lt;input type=\u0026#34;search\u0026#34; placeholder=\u0026#34;Search...\u0026#34; value=\u0026#34;\u0026#34; name=\u0026#34;q\u0026#34; title=\u0026#34;Search for:\u0026#34;\u0026gt; \u0026lt;input type=\u0026#34;hidden\u0026#34; name=\u0026#34;sitesearch\u0026#34; value=\u0026#34;santoshk.dev\u0026#34;\u0026gt; \u0026lt;button type=\u0026#34;submit\u0026#34; value=\u0026#34;Search\u0026#34;/\u0026gt; \u0026lt;/form\u0026gt; Related Posts (pending) # Update: Since version 0.27, Hugo provides a powerful solution for related content out of the box. One needs to add a related content section to all single posts page, like below:\n1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 \u0026lt;div class=\u0026#34;related-posts\u0026#34;\u0026gt; \u0026lt;p class=\u0026#34;h3-like\u0026#34;\u0026gt;Related Posts\u0026lt;/p\u0026gt; {{ $related := .Site.RegularPages.Related . | first 4 }} {{ with $related }} \u0026lt;ul\u0026gt; {{ range . }} \u0026lt;li\u0026gt; \u0026lt;a href=\u0026#34;{{ .Permalink }}\u0026#34;\u0026gt; \u0026lt;img src=\u0026#34;{{ .Params.featuredImage }}\u0026#34; alt=\u0026#34;{{ .Title }}\u0026#34;/\u0026gt; \u0026lt;p\u0026gt;{{ .Title }}\u0026lt;/p\u0026gt; \u0026lt;/a\u0026gt; \u0026lt;/li\u0026gt; {{ end }} \u0026lt;/ul\u0026gt; {{ end }} \u0026lt;/div\u0026gt; Deployment Process # Initially I was going to host my website at Github Pages, but while this entire process I came over Netlify. It gives more control over the deployment process. It also works directly with my local Hugo setup and I don\u0026rsquo;t need another repo with generated html content.\nI added these lines in a file called netlify.toml in the root of the repo:\n1 2 3 4 5 6 7 8 [build] publish = \u0026#34;public\u0026#34; command = \u0026#34;hugo --gc --minify\u0026#34; [context.production.environment] HUGO_VERSION = \u0026#34;0.59.1\u0026#34; HUGO_ENV = \u0026#34;production\u0026#34; HUGO_ENABLEGITINFO = \u0026#34;true\u0026#34; I signed up to Netlify with my git hosting provider by which I was hosting your Hugo repo.\nSetting up a new Hugo site was pretty easy and straight forward. At last I selected my brach to which netlify will be looking for new updates.\nMore information about Netlify deployment can be found at https://gohugo.io/hosting-and-deployment/hosting-on-netlify/.\nThis was a journey from WordPress to Hugo, and was exaustive actually. Anyways, migrations are always painful than starting from scratch. And Hugo isn\u0026rsquo;t only used for blogging. I found out a theme which makes it easier to create dotumentation/book/tutorial.\nFor more updates like this one, please subscribe to this blog below.\n","date":"25 December 2019","externalUrl":null,"permalink":"/posts/2019/switching-to-hugo-this-new-year/","section":"Posts","summary":"Finally I have decided to buy me a domain. But before that I must keep my content ready.\nEarlier on, I was decided on to giving my website a go backend, but I later thought it was an overkill to use golang as a backend for a personal website and I should keep that enthusiasm for a real app. Instead of manually creating a frontend for my site, I will be using already available Hugo theme. I will be migrating my blog which I’m maintaining at codicious.wordpress.com and fxcious.wordpress.com.\n","title":"Switching to Hugo This New Year","type":"posts"},{"content":"","date":"25 December 2019","externalUrl":null,"permalink":"/tags/wordpress/","section":"Tags","summary":"","title":"Wordpress","type":"tags"},{"content":"archive page\n","date":"19 October 2019","externalUrl":null,"permalink":"/archive/","section":"Archive","summary":"archive page\n","title":"Archive","type":"archive"},{"content":"","date":"1 September 2019","externalUrl":null,"permalink":"/tags/deployment/","section":"Tags","summary":"","title":"Deployment","type":"tags"},{"content":" Introduction # Why wait for CI/CD services to run your test? And in case you\u0026rsquo;ve already submitted a pull request, you\u0026rsquo;ll have to push another commit to fix the cause of failed test held by CI provider. You can prevent that by running unit tests locally before every commit.\nI\u0026rsquo;m building my website using the golang. On every git push, Travis CI runs test and create a docker image and pushes it to DockerHub. My plan is to pull that image every time it is pushed and deploy it to some production server.\nFailed Travis build. One thing I\u0026rsquo;ve learned about this deployment system is, once I push my repo to GitHub and wait for the image to be pushed to DockerHub. I\u0026rsquo;m not sure if the test will pass and the image will be pushed. I have to open TravisCI every time I\u0026rsquo;ve to make sure.\nNow I\u0026rsquo;ll show what workflow I use to to get this done.\nStep by Step Setup # Inside your git repo, under .git/hooks, you\u0026rsquo;ll see git hook scripts. These are the scripts which run on a particular kind of event which git emits. The name of each file represents an event in git, and every time that event occurs, that script is run.\nContents of .git/hooks To make the hooks active, you\u0026rsquo;ll have to rename the file and remove the .sample extension.\nmv .git/hooks/pre-commit.sample .git/hooks/pre-commit mv .git/hooks/ The sample files might have already some content in them. You can safely remove them and start fresh.\nAlthough the default file type kept inside the hooks directory is bash. You can run whatever file type you want. For example, python, ruby. Basically, anything your development machine has available. Just make sure you put proper hashbang. Following is an example of a bash script.\n#!/usr/bin/sh echo Running pre-commit hook echo PWD is $PWD go test -v What Command do You Use to Run Tests # go test -v can be replaced with anything which you use to execute your test.\nMake sure the file is executable by running chmod +x \u0026lt;file\u0026gt;. The pre-commit hook works on the basis of exit codes. If the test failed, the exit code will be not 0. Which means something is wrong (test failed in this case). And the commit will halt there.\nPassed Travis build. Now I\u0026rsquo;m 100% sure my build will always pass and docker image will always be pushed to DockerHub.\nConclusion # Depending on your workflow, you might or not like way there is an additional time required to wait for the commit, but there are a plethora of possibilities with pre-commit hooks. Like linting your code, running a code format tool over it (go fmt), or anything else you would like to do with your life before commit.\nSubscribe for more updates by providing your email in form below.\nResources # https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks https://githooks.com/ ","date":"1 September 2019","externalUrl":null,"permalink":"/posts/2019/run-unit-test-before-every-commit-with-pre-commit-hook/","section":"Posts","summary":"Introduction # Why wait for CI/CD services to run your test? And in case you’ve already submitted a pull request, you’ll have to push another commit to fix the cause of failed test held by CI provider. You can prevent that by running unit tests locally before every commit.\nI’m building my website using the golang. On every git push, Travis CI runs test and create a docker image and pushes it to DockerHub. My plan is to pull that image every time it is pushed and deploy it to some production server.\n","title":"Run Unit Test Before Every Commit with pre-commit Hook","type":"posts"},{"content":"","date":"1 September 2019","externalUrl":null,"permalink":"/tags/webdev/","section":"Tags","summary":"","title":"Webdev","type":"tags"},{"content":"","date":"3 June 2019","externalUrl":null,"permalink":"/categories/animation/","section":"Categories","summary":"","title":"Animation","type":"categories"},{"content":"","date":"3 June 2019","externalUrl":null,"permalink":"/tags/blender/","section":"Tags","summary":"","title":"Blender","type":"tags"},{"content":"","date":"3 June 2019","externalUrl":null,"permalink":"/tags/checklist/","section":"Tags","summary":"","title":"Checklist","type":"tags"},{"content":"This is a textual version of Ways to Speed Up Blender Cycles Rendering, hope Andrew won\u0026rsquo;t mind me writing a textual version of his video. ;) This post is mostly for self-reference and does not contains all the methods shown by Andrew.\nWe\u0026rsquo;ll see how we can create better renders and how render times can be reduced by merely changing some factory settings, and also what prices we might have to pay for that optimization.\nConnect with on LinkedIn.\nReduce Light Bounces # While light bounces bring realism, it is a processor intensive task. You must keep it lower while still maintaining the realistic output. No light bounces (global illumination) can render the scene as unrealistic or cartoonish. One bounce is enough until you are rendering glass. In that case, you can set the global setting to 1 bounce and transparency to 3-4. This setting can be found under Render Tab \u0026gt; Light Paths.\nKeeping light bounces low yields similar render in less time. Use Portal # When you are making an interior scene, in which there is a window from where light (area) is coming. It\u0026rsquo;s better to use the area light as a portal. Doing it does not do all the calculations of light which is being outside the window and the building.\nThis can help keep samples 2x low, while grain still in control.\nUse of portal light in an interior render can reduce noise Use GPU # Not to remind, if you have GPU, use it. By default, this setting is set to CPU which most newcomers don\u0026rsquo;t notice.\ngpu settings If you have a GPU, but still can\u0026rsquo;t see GPU in these setting. You might have to enable it from system setting.\nenable gpu preferences Change Tile Size # When rendering with CPU, use lower tile size (because multiple threads can render different part of the frame simultaneously). While on GPU you have kinda 1 tile at a time to work on (Until you got two GPU, in that case, you\u0026rsquo;ll have 2 tiles at a given time). 512512 would be good for GPU. On the other hand, 3232 would be good for the CPU. When you buy a new GPU, do some tests to get to know your GPU\u0026rsquo;s optimal tile size.\nHigher tile size for GPU and low for CPU works better Reduce Samples # Reducing sample can give you grain. There\u0026rsquo;s a point where sample going any high is pointless. Lower it until you start to see the grain. Compensating samples for grain is not always the best. There could be another reason causing the grain. Increasing samples will just increase the render time.\nStart from low and go up. Denoise. Denoise # Starting from Blender 2.79, you can use the denoise feature in cycles. Tweak with settings to see what works for you better.\ndenoising Clamp # Clamping can get rid of fireflies. But with a cost. It kills lighting. It removes noise in fewer samples. Use it if all other noise reduction method fails. This setting can be found inside the Render \u0026gt; Light Path \u0026gt; Clamping.\nClamping reduces fireflies, use it as the last resort (affects lighting). Caustics # Make sure refractive and reflective caustics are turned off unless necessary. By default, it\u0026rsquo;s turned off. Settings can be found inside Render \u0026gt; Light Paths \u0026gt; Caustics.\nDon\u0026rsquo;t use caustics unless you\u0026rsquo;re rendering glass or water. Object Instancing # Use it whenever possible. Imagine multiple trees in a forest. GPU/CPU have to cache some texture and mesh data. And when it does it for every tree, there is goes resource hungry. Duplicating instead of instancing increases memory usage and render time. With only one tree, you can drop the resource usage drastically.\nRead more about it here and here.\nAdaptive Subdivision # Another option to enable to reduce resource usage. Based on how close objects to the camera, it decides to keep the meshes low poly and high poly. Meshes close the camera are high poly. Settings can be found inside the subdivision modifier.\nYou might have to enable experimental feature set in Cycles for this setting to appear.\nadaptive sampling See also: # https://docs.blender.org/manual/en/latest/render/cycles/optimizations/reducing_noise.html Subscribe to blog for more updates like this one.\n","date":"3 June 2019","externalUrl":null,"permalink":"/posts/2019/cycles-rendering-checklist/","section":"Posts","summary":"This is a textual version of Ways to Speed Up Blender Cycles Rendering, hope Andrew won’t mind me writing a textual version of his video. ;) This post is mostly for self-reference and does not contains all the methods shown by Andrew.\nWe’ll see how we can create better renders and how render times can be reduced by merely changing some factory settings, and also what prices we might have to pay for that optimization.\n","title":"Cycles Rendering Checklist for Faster and Clear Renders","type":"posts"},{"content":"In this post, I\u0026rsquo;ll show a glimpse of software testing and how test-driven development works in a nutshell.\nWhy Do We Test # Every little software grows big, trying to solve more problem regarding its particular domain. It faces bugs in the way which also needs to be taken care of.\nAs software grows big. It becomes harder to go and test if every part of the software is behaving the way it should. Take the example of a Django application. If the manual path is taken, one is supposed to test every layer of the application, including Model and Views against each if and else in the control flow.\nThis approach is fine until we have a smaller codebase. But imagine if the codebase inflates, we\u0026rsquo;ll be spending more time testing than implementing the actual code.\nSome developers even tend to first write the test and then write the software logic according to that. Then the software is improved to pass the new tests, only. The process goes like, Add a test \u0026gt; Run test and see if the test fails \u0026gt; Write the code \u0026gt; Run tests \u0026gt; Refactor code \u0026gt; Repeat.\nTypes of Testing # Now we know how test-driven development works. But the tests are itself classified into multiple variations. There are numerous types, levels, and classifications of tests and testing approaches. The most important automated tests are:\nUnit Tests # Verify the functional behavior of individual components, often to class and function level. You give a test case and check if the program the same results every time. This path is so common, major language has some implementation of unittest. Python has inbuilt unittest. JavaScript has their own. Even golang.\nRegression Tests # You fix a bug, which unconsciously results in another one. Regression tests are the tests that reproduce historic bugs. Each test is initially run to verify that the bug has been fixed, and then re-run to ensure that it has not been reintroduced following later changes to the code.\nEven if you send some project pull request, your patch is most likely will be first going through a test before request is merged into main codebase. If the test fails, maintainer might ask for the changes.\nIntegration Tests # Verify how groupings of components work when used together. Integration tests are aware of the required interactions between components, but not necessarily of the internal operations of each component. They may cover simple groupings of components through to the whole website.\nCode Coverage # When a code is tested. It might or might not go into every conditional branch.\nCode coverage is a metric which measures how much test covers the whole codebase. Tests are supposed to go through every each and else condition for a higher code coverage count. Higher is this measure, lower is the chance of encountering undetected software bugs.\nRelated readings # https://en.wikipedia.org/wiki/Test-driven_development https://developer.mozilla.org/en-US/docs/Learn/Server-side/Django/Testing Did you like the post? Is something missing? Please let me know in the comments. Also subscribe to this blog below.\n","date":"15 January 2019","externalUrl":null,"permalink":"/posts/2019/software-testing-and-test-driven-development/","section":"Posts","summary":"In this post, I’ll show a glimpse of software testing and how test-driven development works in a nutshell.\nWhy Do We Test # Every little software grows big, trying to solve more problem regarding its particular domain. It faces bugs in the way which also needs to be taken care of.\nAs software grows big. It becomes harder to go and test if every part of the software is behaving the way it should. Take the example of a Django application. If the manual path is taken, one is supposed to test every layer of the application, including Model and Views against each if and else in the control flow.\n","title":"Software Testing and Test Driven Development","type":"posts"},{"content":"","date":"8 October 2018","externalUrl":null,"permalink":"/tags/skills/","section":"Tags","summary":"","title":"Skills","type":"tags"},{"content":"I have been an independent programmer for a long time. Like 5-6 years? Recently been employed with Xentrix Studios, an animation studio based in Bengaluru, as a pipeline developer.\nToday I want to share what changes I\u0026rsquo;ve felt in these recent years. Please connect with on LinkedIn.\nWrite Readable Code # The code is read much more often than it is written. In fact, source code is written for us, the humans. A source code to achieve a certain task to be done can be written in dozens, if not hundreds of ways. Computers will still interpret it no matter in which way it is written. So at the end of the day, it\u0026rsquo;s the human to whom flow of source code matter.\nThis was something I wasn\u0026rsquo;t aware until some time back. This isn\u0026rsquo;t something you learn when you code for yourself. Because there is a high probability that you\u0026rsquo;ll understand your own code. But when you code in a team, it is not the same case.\nThis is where coding convention kicks in. Coding convention in a group is adhered to maintain interoperability between co-programmers.\nAdhere to the style guide of the language community or your company\u0026rsquo;s own; Python has their PEP-8, other might have their own. Google has its own style guide covering multiple languages.\nAvoid Tight Coupling # In my former days of programming, like any other self-taught programmer, I used to write procedural code.\nTight coupling is something when you write code which is highly dependent on another piece of code, most probably in the same script. When you change the code in one place, it breaks the entire script, like a ripple effect.\nBut when you break the code in different modules, you get a same place to look up into for similar problems.\nConceptual model of coupling (Wikipedia) User Experience # An analogy of UX can be taken from a door. There\u0026rsquo;s a door which has a handle. Would you push it or pull it? Now there\u0026rsquo;s another door which has no handle, would you push it or pull it?\nSame applies in interface design. Avoid distracting users attention with multiple components. Notice how having no handle on door makes up obvious to push the door?\nDon\u0026rsquo;t make the user click 5 buttons to do the stuff which can be done in 2 clicks.\nUnit Testing # Unit testing is something I\u0026rsquo;m not doing right now. But I anticipate it\u0026rsquo;s essential for large-scale development.\nFor more updates like this, please subscribe to my blog by filling the form below.\n","date":"8 October 2018","externalUrl":null,"permalink":"/posts/2018/what-ive-learnt-as-a-self-taught-programmer-till-now/","section":"Posts","summary":"I have been an independent programmer for a long time. Like 5-6 years? Recently been employed with Xentrix Studios, an animation studio based in Bengaluru, as a pipeline developer.\nToday I want to share what changes I’ve felt in these recent years. Please connect with on LinkedIn.\nWrite Readable Code # The code is read much more often than it is written. In fact, source code is written for us, the humans. A source code to achieve a certain task to be done can be written in dozens, if not hundreds of ways. Computers will still interpret it no matter in which way it is written. So at the end of the day, it’s the human to whom flow of source code matter.\n","title":"What I've Learnt as a Self Taught Programmer Till Now","type":"posts"},{"content":"","date":"8 July 2018","externalUrl":null,"permalink":"/tags/bash/","section":"Tags","summary":"","title":"Bash","type":"tags"},{"content":"","date":"8 July 2018","externalUrl":null,"permalink":"/tags/gnome/","section":"Tags","summary":"","title":"Gnome","type":"tags"},{"content":"","date":"8 July 2018","externalUrl":null,"permalink":"/tags/kde/","section":"Tags","summary":"","title":"Kde","type":"tags"},{"content":"It\u0026rsquo;s been a long time I\u0026rsquo;ve been fiddling around with Linux and Linux distros. The first Linux distro I ever used was Ubuntu. I like the beauty of Linux, its so customizable. One can change every aspect of interface and behaviour that end-user want. Like the desktop environments and shells, which you can\u0026rsquo;t really change on Windows operating systems.\nI myself belive in minimalism and not full of features products (which remains unused most of the time in my case). But I\u0026rsquo;m making this descision after using both of them for about a week.\nKDE # Most Linux distro, by default is equipped with GNOME by default. This is a pretty good Desktop Environment to hang around with, to get a taste of Linux. I never made a switch to desktop environment unti now, although I made switch to many distros.\nBut KDE is more customizable. I can make it behave the way I want. And I don\u0026rsquo;t have to install extension which bloat up my system to do that. The themes, the animation, the extra settings GNOME is not able to deliver by default is the reason I\u0026rsquo;m moving towards KDE. Other than that, KDE is written in Qt, which I\u0026rsquo;m working in my workplace _(Xentrix Studios). _It\u0026rsquo;s out of scope of this post here, but Python and Qt are the easiest and quickest if not best tools for cross platform GUI development.\nI\u0026rsquo;m not a big fan of packages (ecosystem) it comes with, but let\u0026rsquo;s see if I start to love them, or replace them with the GNOME equivalent.\nZSH # Just like GNOME, bash is default on all Linux. Bash made me learn shell scripting, which I feel pretty comfortable with now.\nThe default auto-completion which zsh provides attracted me towards it. Other features of zsh includes, it\u0026rsquo;s themable, sharing of command history among all running tabs/shells, extended file globbing.\nI don\u0026rsquo;t use all of them right now, but will get hang of it by time.\nLegacy # I know the consequences of making this change. I\u0026rsquo;ll still get bash when I ssh to remote server and the shell prompt which Git provides on Windows will still be bash. Maybe in future the company I\u0026rsquo;ll work with will have GNOME and I will have no control over it to modify these both. But personally, I\u0026rsquo;ll go with these both.\nBut Why # Now this paragraph is going to be highly subjective. GNOME and bash will still be my legacy. The knowledge I gained while using them will not be lost. But I think I\u0026rsquo;m ready to make a change now. Either they are limited or it\u0026rsquo;s hard to customize them.\nWhat are your thoughts on bash, zsh, GNOME and and KDE? Let\u0026rsquo;s discuss in comment\u0026rsquo;s section.\n","date":"8 July 2018","externalUrl":null,"permalink":"/posts/2018/making-a-switch-to-kde-and-zsh/","section":"Posts","summary":"It’s been a long time I’ve been fiddling around with Linux and Linux distros. The first Linux distro I ever used was Ubuntu. I like the beauty of Linux, its so customizable. One can change every aspect of interface and behaviour that end-user want. Like the desktop environments and shells, which you can’t really change on Windows operating systems.\nI myself belive in minimalism and not full of features products (which remains unused most of the time in my case). But I’m making this descision after using both of them for about a week.\n","title":"Making a Switch to KDE and zsh","type":"posts"},{"content":"","date":"8 July 2018","externalUrl":null,"permalink":"/tags/shell/","section":"Tags","summary":"","title":"Shell","type":"tags"},{"content":"","date":"8 July 2018","externalUrl":null,"permalink":"/categories/story/","section":"Categories","summary":"","title":"Story","type":"categories"},{"content":"","date":"8 July 2018","externalUrl":null,"permalink":"/tags/zsh/","section":"Tags","summary":"","title":"Zsh","type":"tags"},{"content":"This is kinda a manual for new Blender user which I compiled up when I was learning Blender.\nFirst of all, when you\u0026rsquo;re switching from Maya, you\u0026rsquo;ll find the interface a bit alien. Blender has an option to behave like Maya, but in long run, that\u0026rsquo;s not gonna be productive.\nConnect with on LinkedIn.\nThe Basics # Right-click selects any object. MMB rotates the scene (we\u0026rsquo;ll se pivot point later in the post). Shift+MMB+mouse movement pans the scene. Ctrl+MMB up and down zooms and zooms out of the level.\nG is for grabbing the poly, S for scale and R for rotate. In all of three functions, pressing x, y and z constrains the transform, scale and rotate. Right clicking and grabbing also move the poly. But there is another kind of rotation which is activated when R is pressed twice. Use it and feel it. You can also shilft select an axis and then grab to constraint to that axis.\nThe Navigation # Numpad 5 switches between perspective and orthographic view of the camera, no matter what view you are in. Numpad 1 goes to Front View, 3 to the Right, 7 to the Top. Pressing Ctrl while pressing those will take to the remaining counterpart of them, that is, Back, Left and Bottom. If you do not choose to hold on Ctrl, press 9 after one of those keystrokes.\n2, 4, 6, 8 will rotate the camera over the center axis. Numpad dot (.) will focus on the selected object, whereas pressing Home will focus the entire scene.\nAdditionally, pressing Shift+F takes to Fly Mode. When you press it, additional controls will be available in the status bar. In a nutshell, it takes to game mode and camera can be navigated with W, A, S, D. When in fly mode, see the statusbar for more info.\nPressing Ctrl+Alt+Q brings to quad view. Press again to return to perspective mode.\nThe Layout # Press T and toggle the sidebar (hold Ctrl and scroll), press N and do the same with another window which shows properties of the selected object.\nBlender\u0026rsquo;s Scripting Layout Ctrl+Left/Right switches between various predefined window layout.\nGo to the top-right of the layout component, the arrow pointer will turn into a crosshair, pull it to left to create new window. You can use that window for anything, just select from lower-left menu. To delete newly created window, just go to that crosshair section again and collapse it to opposite side.\nWorking on a dual monitor? Press Shift while on crosshair section, and it will pop out, fly it to another monitor. And when Ctrl is kept pressed, the window will swap. Also, there is a Maya like Ctrl+Space feature, but that is mapped to Shift+Space, it expands viewport.\nCreating Stuff # Shift+A pops up Add menu, where you can add various kind of elements in the scene.\nAdd Menu Press Z to switch between display modes, like textured, wireframe, shaded et cetra. Although g gives you a hand to grab, using Shift+S will give you snapping menu, which can be used to snap object according to various parameters.\nDifferent Modes # You can switch between different editing modes while working with an asset. Have a look below:\nEditing Mode Specific Context Menu If you change into another editing mode, the context menu would also change to give you extra commands for that specific mode.\nOther than editing mode, you can also go to other viewport shading mode.\nSwitch to different viewport shading mode Next to that is pivot point menu:\nPivot Point Menu Pivot point helps in navigation and modification of objects.\nThat\u0026rsquo;s enough for a Interface topic now. Without going out of scope, we should stop here. Feel free to add more in comments and I\u0026rsquo;ll mention it in here. Also, please subscribe to the blog feed by entering your email address below.\n","date":"10 June 2018","externalUrl":null,"permalink":"/posts/2018/advance-interface-topics-in-blender/","section":"Posts","summary":"This is kinda a manual for new Blender user which I compiled up when I was learning Blender.\nFirst of all, when you’re switching from Maya, you’ll find the interface a bit alien. Blender has an option to behave like Maya, but in long run, that’s not gonna be productive.\nConnect with on LinkedIn.\nThe Basics # Right-click selects any object. MMB rotates the scene (we’ll se pivot point later in the post). Shift+MMB+mouse movement pans the scene. Ctrl+MMB up and down zooms and zooms out of the level.\n","title":"Advance Interface Topics in Blender","type":"posts"},{"content":"","date":"10 June 2018","externalUrl":null,"permalink":"/tags/maya/","section":"Tags","summary":"","title":"Maya","type":"tags"},{"content":"In last post, I expressed how we can export a single file from a git repo the its own repo, preserving their commit history. In this post, I will tell how I dealt with extension-less file to ignore them in my repo.\nI have recently started learning C++ from Udemy, it\u0026rsquo;s a free course. I encourage you to take it if you are willing to learn C++. I was following the instructions on Linux. And special thing about working with C or C++ in Linux is that the executable you get has no extensions in it, unlike Windows, which has .exe extension.\nIt was a challenge for me to keep .cpp source files tracked, yet exclude the executable files. What would have I done? Without extension, I couldn\u0026rsquo;t do, for example, *.pyc, like I used to do in Python repositories. Keeping the only cpp files in staging area would have cluttered up my git status output every time a new executable was created, isn\u0026rsquo;t so?\nBut How # Remember what ! does in a boolean value? Correct, it negates it. The same could be applied in .gitignore\u0026rsquo;s entries too. Normally what we do in gitignore file is, we write a statement per line to blacklist files or directory we want to exclude from VCS. With regex support that\u0026rsquo;s even more powerful.\nHow about this in .gitignore?\n* !*.* !*/ So what does the above lines do? As most of you might be aware of, * is shorthand for select all in regex terms. So what we are saying is:\nignore/blacklist everything DO NOT blacklist any filename with any extension. Basically boils down to do not ignore any file. DO NOT blacklist any directory. The / at last means we are dealing with directory. Is there Another Option # Another option is to make a directory and move all the executable files there and exclude that directory with .gitignore. But I like this method more. It gives me the freedom to not mv files, again and again, everytime I compile a new file.\nContinue Reading # gitignore without binary files If you liked this post, the subscription form below is for you.\n","date":"5 June 2018","externalUrl":null,"permalink":"/posts/2018/how-would-you-exclude-all-extension-less-executable-from-your-repo/","section":"Posts","summary":"In last post, I expressed how we can export a single file from a git repo the its own repo, preserving their commit history. In this post, I will tell how I dealt with extension-less file to ignore them in my repo.\nI have recently started learning C++ from Udemy, it’s a free course. I encourage you to take it if you are willing to learn C++. I was following the instructions on Linux. And special thing about working with C or C++ in Linux is that the executable you get has no extensions in it, unlike Windows, which has .exe extension.\n","title":"How would you exclude all extension-less executable from your repo?","type":"posts"},{"content":"Today I will walk through how you can take out a single file from a git repository and create its own repository with all the file commit history preserved.\nWhen I started working at my workplace I initialized Maya script directory to git. At that time one repo was looking enough for entire folder. But when I started working with a tool, I realized an entire repo would be good for that tool. But it was too late to create another repo because I had already made 12-15 commits on that tool and I don\u0026rsquo;t wanted to lose the changes I made.\nI researched it on internet and found this.\n1 2 3 4 5 cd repository git log --pretty=email --patch-with-stat --reverse --full-index --binary -- path/ \u0026lt;/code\u0026gt;to/file_or_folder \u0026gt; patch cd ../another_repository git init git am \u0026lt; ../repository/patch This is an excerpt from a StackOverflow question.\nBasically what it is doing it is, creating patch based on the file history, along with the commiter and other data. And after creating another repository applying the patch. There is even no need to copy the files. The code is pretty self explanatary.\nIf you liked this post, please subscribe to my newsletter below.\n","date":"4 June 2018","externalUrl":null,"permalink":"/posts/2018/extract-a-file-from-git-repo-to-its-own-preserving-the-history/","section":"Posts","summary":"Today I will walk through how you can take out a single file from a git repository and create its own repository with all the file commit history preserved.\nWhen I started working at my workplace I initialized Maya script directory to git. At that time one repo was looking enough for entire folder. But when I started working with a tool, I realized an entire repo would be good for that tool. But it was too late to create another repo because I had already made 12-15 commits on that tool and I don’t wanted to lose the changes I made.\n","title":"Extract a File from Git Repo to its own, Preserving the History","type":"posts"},{"content":"Back in time when I used to see Photoshop\u0026rsquo;s layers section, there was a list of things named like Darken, Multiply, Screen and so on. They used to change the color when that particular layer was placed on some other layer.\nLater in my college days I came to know these were the well known blending modes. There are mathematical in nature and does maths on pixel by pixel basis.\nNow I\u0026rsquo;m going to go through some of the blending modes. This documentation assumes knowledge of channels. And terminologies are for Foundry Nuke; as same operation have different names over different packages, but the work is same.\nConnect with on LinkedIn.\nMultiply # Multiply multiplies each channel information of one image to another. The result is subtractive as we multiply..\nPlus # Plus adds pixel value from background to the foreground.\nScreen # The screen is a conditional operation. It multiplies if pixel if higher than 50% threshold and plus if the value if the value if lower than 50%.\nAverage # Average averages the range between both pixel channels. Mathematically A+B/2 (B=background, A=foreground).\nMin/Max # Min and Max keeps minimum and maximum per pixel value from both the images.\nWe can group above operation in one group and call it commutative. The operation in this group does not affect the result if background and the foreground are swapped, the result will be the same.\nOver # Over as the name implies, puts foreground over the background. Pixels are overwritten over the background.\nMinus # Minus subtract foreground pixel from the background pixel.\nFrom # This one is like minus, just it subtracts background pixel from foreground pixel.\nDivide # Divides foreground by background. A/B else 0.\nMatte # Matte is just like premultiplying foreground with its alpha. Otherwise it\u0026rsquo;s just over.\nDifference # Helpful in checking the difference between two states if comp.\nOverlay # Overlay is not to be confused with over. It multiply if B\u0026lt;0.5, screen if B\u0026gt; 0.5. Hard-light is with input swapped.\nFor more tech related updates, please subscribe to my newsletter.\n","date":"28 February 2018","externalUrl":null,"permalink":"/posts/2018/blending-modes-explained/","section":"Posts","summary":"Back in time when I used to see Photoshop’s layers section, there was a list of things named like Darken, Multiply, Screen and so on. They used to change the color when that particular layer was placed on some other layer.\nLater in my college days I came to know these were the well known blending modes. There are mathematical in nature and does maths on pixel by pixel basis.\n","title":"Blending Modes Explained","type":"posts"},{"content":"","date":"28 February 2018","externalUrl":null,"permalink":"/tags/compositing/","section":"Tags","summary":"","title":"Compositing","type":"tags"},{"content":"","date":"28 September 2017","externalUrl":null,"permalink":"/tags/nuke/","section":"Tags","summary":"","title":"Nuke","type":"tags"},{"content":"Nuke is a very powerful VFX tool in the industry. But at the time of learning, people often don\u0026rsquo;t pay attention to the optimization. Which leaves them with relatively longer render times.\nNote: Not every point can be followed. Do whatever is possible.\nConnect with on LinkedIn.\nHardware Related # Get Two Hard Drive # When you read footage from the hard drive, and write/render to the same hard drive, you actually make hard drive do double the work. Keep another drive (maybe a small one?) to render/write stuff.\nCache Management # This one is not for rendering; also not only applies for Nuke. Caching can make working in Nuke faster, by fasting up previews. It basically writes out files to disk and reads again without involving CPU or GPU again and again. This is the universal point of caching. It is preferred to change cache location to the _write _drive as said above.\nDisk caching in another drive for better performance. Node Graph Best Practices # Bounding Box (bbox) # Bounding Box is a virtual area which is calculated by render engine. bbox is not necessarily the resolution of the scene/project.\nAlthough Smaller is the bbox, faster CPU calculates the scene.\nThere\u0026rsquo;s node called CurveTool which we can export Auto Crop data to crop node. Which I\u0026rsquo;ll discuss in a later post. And why it\u0026rsquo;s important.\nAvoid Multiple Read Nodes # Read identical footages will just make the comp heavy. If there is need of reading the same file multiple times, pipe and branch out files with dot node. Dot node usually appears when Ctrl key is pressed over Node Graph.\nAvoid multiple read node. Masking/Bezier/Roto # Use roto node instead of rotopaint for masking (making bezier), there was a dedicated bezier node which became deprecated and later combined into roto node.\nBlur Instead of Defocus # Simply because Blur does less of the calculation than defocus, the same science involved in optics.\nRemove Channels after Use # There\u0026rsquo;s a Remove node specifically for this. Keeping channels can bog up the processing time. At least this is how Nuke works. There\u0026rsquo;s a Remove and Keep operation within this node. Keep will keep specified channels and delete all. Use it wisely.\nContinue Reading: # 10 Tips to Optimising Nuke\nTony Lyons\u0026rsquo;s Optimization in Nuke\nIf you liket this post, please subscribe to newsletter below for more tech related article.\n","date":"28 September 2017","externalUrl":null,"permalink":"/posts/2017/nuke-optimization-tips/","section":"Posts","summary":"Nuke is a very powerful VFX tool in the industry. But at the time of learning, people often don’t pay attention to the optimization. Which leaves them with relatively longer render times.\nNote: Not every point can be followed. Do whatever is possible.\nConnect with on LinkedIn.\nHardware Related # Get Two Hard Drive # When you read footage from the hard drive, and write/render to the same hard drive, you actually make hard drive do double the work. Keep another drive (maybe a small one?) to render/write stuff.\n","title":"Nuke Optimization Tips","type":"posts"},{"content":"","date":"28 September 2017","externalUrl":null,"permalink":"/tags/tips/","section":"Tags","summary":"","title":"Tips","type":"tags"},{"content":"When I first came to know about this, I was like:\n{{ #\u0026lt; twitter user=\u0026ldquo;sntshk\u0026rdquo; id=\u0026ldquo;816562426833289216\u0026rdquo; \u0026gt;}}\nWhat is Squashing # The process of merging commits together is called squashing. There are many commands to do the same thing, but I will discuss the one I learned. There\u0026rsquo;s a read more section below if you want to know more about this topic.\nsquashing multiple commit into one Don\u0026rsquo;t go far Deep # One thing to keep in mind, before we start is, we should avoid merging our pushed commits. It will create a problem to other developers if you\u0026rsquo;re working in a team, and things will kinda mess up.\nCodes and Explanations # Ariejan has already explained this in deep, so I will not do the same here but will do it for my reference. For merging last 3 commits, we can pass:\ngit rebase -i HEAD~3 This will open up your default editor configured with git with the following content:\npick hash Commit 1 pick hash Commit 2 pick hash Commit 3 The commits appear from older to newer. We\u0026rsquo;re probably gonna merge commit 2 \u0026amp; 3 into commit 1. In our code editor what we\u0026rsquo;ll do is replace pick from commit 2 \u0026amp; 3 lines to squash.\nSorry for the syntax highlighting. wordpress.com is preventing me from adding most external services like gist and all which has javascript embed (Twitter is working though).\nWhat are your thoughts on this post? How do you squash your changes? I\u0026rsquo;m eager to know. Please leave comments.\nSubscribe to the newsletter below.\nFurther Reading # http://stackoverflow.com/a/5201642/939986 https://davidwalsh.name/squash-commits-git https://ariejan.net/2011/07/05/git-squash-your-latests-commits-into-one/ ","date":"21 September 2017","externalUrl":null,"permalink":"/posts/2017/merge-more-than-one-commits-before-pushing/","section":"Posts","summary":"When I first came to know about this, I was like:\n{{ #\u003c twitter user=“sntshk” id=“816562426833289216” \u003e}}\nWhat is Squashing # The process of merging commits together is called squashing. There are many commands to do the same thing, but I will discuss the one I learned. There’s a read more section below if you want to know more about this topic.\n","title":"Merge more than one Commits before Pushing","type":"posts"},{"content":"Below is a list of procedures a compositor should follow. I also have a blog post compiled of optimization tips, which should work as side document to this post.\nDistortion # Evey footage has some sort of distortion. Not dealing with distortion before 3D integration can result in catastrophic disasters. It\u0026rsquo;s good to first undistort the background plate \u0026gt; do the CG integration \u0026gt; use the same undistortion data to re-distort the whole output. This way CG will be distorted as well. Giving a feel if it was shot with a camera.\nYou can find more about distortion workflow on Joe Raasch blog. Although distortion data is also handled by tracking software. That can be exported straight out from there.\nColor Matching aka Color Correction # Color correction has not the only place in VFX compositing but it is from the start of the pipeline. One output, which has two different images with different sources can\u0026rsquo;t look real until it is color calibrated to the base image. A good understanding of color, in general, is a plus.\nMatch Grain # If looked closely, footage plate contains some noise. But our readers usually comes at full render quality. So does the static images are not grainy. So we have to match the rate of noise Matching the level of the grain of background plate to CG element or the images we are using to give the sense of realism.\nEven there\u0026rsquo;s a node in Nuke for this. Inside the FurnaceCore plugin, named F_ReGrain.\nMotion/Edge Blur # Final outputs are not as sharp as they are rendered plates are (there can be exceptions). At 24fps, there is some motion blur, unlike games, which usually run at 60fps. The whole purpose that films still is broadcasted at 24fps is to give a feel realism to the eyes.\nMotion vector pass is generally used to do these motion blur things. Which can be obtained same way general passes are rendered. More on Motion Vector on Lester Bank\u0026rsquo;s blog.\nEdge blur on the other hand is used to blur the sharp edges of render objects. It blurs only those part of scene in which objects are moving, relative to speed of object and not the camera. This is what differentiates it from motion blur.\nChromatic Aberration # There\u0026rsquo;s no lens made in the universe, which has no chromatic aberration. Chromatic Aberration is a phenomenon in which channels of the image tend to offset by a certain amount of pixels. Just like distortion, this offset increases as we go from the middle of the footage.\nDepth of Field # Depth of Field is another factor which makes the plate look real.\nz-depth pass rendered out from Maya is used inside ZDefocus node of Nuke. Similar node must be there on other packages.\nLightWrap # When a key object is mixed over the background, it usually stands out. LightWrap helps there. The plugin was made by Red Giant for After Effects is also available in Nuke natively. A good tutorial about LightWrap in Nuke can be found at Huey Yeng\u0026rsquo;s Nuke blog.\nLens Flares? # Depending upon the lighting densities, there must be lens flare making the scene better. I find it less necessary of all listed here.\nIf you\u0026rsquo;ve read this whole post, and still reading, you should just give your opinions in the comments section.\nContinue Reading # Compositing Practices 101: https://www.nukepedia.com/written-tutorials/compositing-practices-101/ Tony Lyons Checklist: https://docs.google.com/document/d/1KL0mQUAcqCEONi7YJxAZq_4fZx_yfDn2Evim85C-Sms/edit ","date":"11 September 2017","externalUrl":null,"permalink":"/posts/2017/digital-compositor-checklist/","section":"Posts","summary":"Below is a list of procedures a compositor should follow. I also have a blog post compiled of optimization tips, which should work as side document to this post.\nDistortion # Evey footage has some sort of distortion. Not dealing with distortion before 3D integration can result in catastrophic disasters. It’s good to first undistort the background plate \u003e do the CG integration \u003e use the same undistortion data to re-distort the whole output. This way CG will be distorted as well. Giving a feel if it was shot with a camera.\n","title":"Digital Compositor Checklist","type":"posts"},{"content":"","date":"11 September 2017","externalUrl":null,"permalink":"/tags/distortion/","section":"Tags","summary":"","title":"Distortion","type":"tags"},{"content":"","date":"6 July 2017","externalUrl":null,"permalink":"/tags/lighting/","section":"Tags","summary":"","title":"Lighting","type":"tags"},{"content":"Although newer version of Maya have incorporated Solid Angle\u0026rsquo;s Arnold renderer. NVIDIA\u0026rsquo;s (formerly mental images) mental ray is still used in production quality rendering.\nPeople coming from Maya\u0026rsquo;s basic shader e.g. Phong, Blinn, Lambert don\u0026rsquo;t get daunted watching the mental ray shaders. So today I\u0026rsquo;ll explain what they actually mean.\n**mi - **mi is short for mental images, the facility in which mental ray was developed. You now know what is that mi in all the shaders.\nmib - b is for base. Are they base materials?\nmia - a is for architecture. Anything not containing any living organism maybe. More information can be found in mental ray documentation.\nmisss - sss is for subsurface scattering. Wikipedia has a good explanation of it.\nmila - l in mila stands for layering. So shaders are also layered to achieve a good result. More information can be found at elemental ray blog.\nNow you have basic knowledge of shaders. Now you can anticipate about car_paint and metallic_paint shader groups (libraries).\n","date":"6 July 2017","externalUrl":null,"permalink":"/posts/2017/mental-ray-shader-terminology-explained/","section":"Posts","summary":"Although newer version of Maya have incorporated Solid Angle’s Arnold renderer. NVIDIA’s (formerly mental images) mental ray is still used in production quality rendering.\nPeople coming from Maya’s basic shader e.g. Phong, Blinn, Lambert don’t get daunted watching the mental ray shaders. So today I’ll explain what they actually mean.\n**mi - **mi is short for mental images, the facility in which mental ray was developed. You now know what is that mi in all the shaders.\n","title":"mental ray shader terminology explained","type":"posts"},{"content":"","date":"6 July 2017","externalUrl":null,"permalink":"/tags/rendering/","section":"Tags","summary":"","title":"Rendering","type":"tags"},{"content":"Every videographer once in their career gets in a situation where they face flickering of video, this happens in a low light situation. This guide will help you get out of that situation to some extent.\nAdjust the Shutter # 180-degree shutter rule says to keep the shutter double of the fps. So that if we\u0026rsquo;re shooting at 1920x1080@24fps, our shutter should be minimum 48. Most camera won\u0026rsquo;t let you go down to 48 but you can round-up, safely.\nAlso note that slower shutter produces motion blur, which is bad for tracking.\nAre you Shooting a Screen # Then adjust the refresh rate if possible. For windows, this is located somewhere in the graphics driver setting. I\u0026rsquo;m worried about the Mac users as I don\u0026rsquo;t find any way to adjust it.\nEpilogue # I hope this quick tips helped you today. If it did, please follow this blog and share this post. Speak you mind in the comments section. :)\n","date":"13 January 2017","externalUrl":null,"permalink":"/posts/2017/get-rid-of-flickering-during-shoot/","section":"Posts","summary":"Every videographer once in their career gets in a situation where they face flickering of video, this happens in a low light situation. This guide will help you get out of that situation to some extent.\nAdjust the Shutter # 180-degree shutter rule says to keep the shutter double of the fps. So that if we’re shooting at 1920x1080@24fps, our shutter should be minimum 48. Most camera won’t let you go down to 48 but you can round-up, safely.\n","title":"Get rid of Flickering during Shoot","type":"posts"},{"content":"","date":"13 January 2017","externalUrl":null,"permalink":"/tags/photography/","section":"Tags","summary":"","title":"Photography","type":"tags"},{"content":"","date":"13 January 2017","externalUrl":null,"permalink":"/tags/quick-tip/","section":"Tags","summary":"","title":"Quick-Tip","type":"tags"},{"content":"","date":"13 January 2017","externalUrl":null,"permalink":"/tags/shooting/","section":"Tags","summary":"","title":"Shooting","type":"tags"},{"content":"","date":"17 December 2016","externalUrl":null,"permalink":"/tags/css/","section":"Tags","summary":"","title":"Css","type":"tags"},{"content":"","date":"17 December 2016","externalUrl":null,"permalink":"/tags/flexbox/","section":"Tags","summary":"","title":"Flexbox","type":"tags"},{"content":"","date":"17 December 2016","externalUrl":null,"permalink":"/categories/frontend/","section":"Categories","summary":"","title":"Frontend","type":"categories"},{"content":"In this tutorial, I\u0026rsquo;ll show how to make items of a flexbox to go to the next line on the window resize.\nThere comes a time in life when you need to create layouts where block snaps to next line when the window is too narrow. This is a best practice in responsive web design.\nThe Setup # Let\u0026rsquo;s get started creating a skeleton for our experiments:\n1 2 3 4 5 6 \u0026lt;div class=\u0026#34;flex\u0026#34;\u0026gt; \u0026lt;div class=\u0026#34;item\u0026#34;\u0026gt;HTML\u0026lt;/div\u0026gt; \u0026lt;div class=\u0026#34;item\u0026#34;\u0026gt;CSS\u0026lt;/div\u0026gt; \u0026lt;div class=\u0026#34;item\u0026#34;\u0026gt;JavaScript\u0026lt;/div\u0026gt; \u0026lt;div class=\u0026#34;item\u0026#34;\u0026gt;MongoDB\u0026lt;/div\u0026gt; \u0026lt;/div\u0026gt; As we have the skeleton, let\u0026rsquo;s start designing. We\u0026rsquo;ll add display: flex; to make that div flex or parent to flex items.\n1 2 3 4 5 6 7 8 .flex { display: flex; justify-content: space-around; } .item { border: 3px solid whitesmoke; } justify-content: space-around; tells the browser to maintain an equal amount of space between all the content.\nLet\u0026rsquo;s proceed giving our .item some width. So that when we resize the window we get the effect.\n1 width: 180px; At this stage, all items would appear on the same line. After we resize the window, it maintains the line.\nThe Magic # By default, flexboxes don\u0026rsquo;t allow items to be spread on multiple lines. This can be overridden flex-wrap property. We\u0026rsquo;ll add flex-wrap: wrap; property to the flex container.\n1 flex-wrap: wrap; This tells the browser to break to another line if there is no more space.\nSo if the viewport is 600px wide, then 600/180 i.e. only 3 flex items could fit on the first line. See this pen for the understanding. There are more topics in flex to talk about. But I would like to wrap up this post here.\nflex-wrap: wrap Please leave a comments, hugs or bugs all welcome. Also please subscribe to the newsletter below.\n","date":"17 December 2016","externalUrl":null,"permalink":"/posts/2016/move-item-to-next-row-on-resize-with-flexbox/","section":"Posts","summary":"In this tutorial, I’ll show how to make items of a flexbox to go to the next line on the window resize.\nThere comes a time in life when you need to create layouts where block snaps to next line when the window is too narrow. This is a best practice in responsive web design.\nThe Setup # Let’s get started creating a skeleton for our experiments:\n","title":"Move Item to Next Row on Resize with Flexbox","type":"posts"},{"content":"","date":"8 May 2012","externalUrl":null,"permalink":"/tags/java/","section":"Tags","summary":"","title":"Java","type":"tags"},{"content":" illegal characters in hello world Today I was reading an e-book on Java programming language and enjoying and learning from it.\nEvery thing was fine until I reached up to arrays section. The problem from that exercise was a bit long, so instead of typing the whole code I decided to copy and paste, and then compile them. As soon as I typed:\njavac arrayExample.java I got not one, nor two but a whole bunch of 100 errors. This was the first time when I received so much error in a such small Java source file. I was surprised seeing that. For fixing it I tried as much as I can, following are which I tried:\nChanged curly braces to square braces and compiled again to see if it works.\nChecked every whitespace to be clear if I have hit and RETURN key, But this also didn\u0026rsquo;t work.\nI disabled the text wrapping of my text editor to checked if it was an indentation error.\nNone of them helped. Then at last I took help of Google and found a list of java compiler error. and I found that I was using Unicode character 8220 (also known as \\u291c, 0x291c, “, left quote) and Unicode character 8221 (also known as \\u291d, 0x291d, ”, right quote) instead of just using a simple Unicode 34 character (also known as \\u0022, 0x22, \u0026ldquo;) which all programming language require.\nWhat Actually Happened # When the book/e-book was being made, a word processor was used (not a text editor or IDE which we use in programming) which uses_ left quote and right quote instead of simple quotes. And when I copy-pasted the left and right quotes remained as they were.\nMoral # Never copy paste anything! specially when you are programming or learning programming. Typing will not only make your typing speed fast but also will make you stop and think (and research the topics you don\u0026rsquo;t know).\nI have just started this blog, please share it and subscribe to the newsletter below.\n","date":"8 May 2012","externalUrl":null,"permalink":"/posts/2012/2012-05-08-what-is-illegal-character-8220-8221-and-how-to-fix-it/","section":"Posts","summary":" illegal characters in hello world Today I was reading an e-book on Java programming language and enjoying and learning from it.\nEvery thing was fine until I reached up to arrays section. The problem from that exercise was a bit long, so instead of typing the whole code I decided to copy and paste, and then compile them. As soon as I typed:\njavac arrayExample.java I got not one, nor two but a whole bunch of 100 errors. This was the first time when I received so much error in a such small Java source file. I was surprised seeing that. For fixing it I tried as much as I can, following are which I tried:\n","title":"What is Illegal Character  \\8220 \u0026 \\8221 Error Messages and How to fix it","type":"posts"}]