Today we are going to do a hands-on on how to kill a dangling EIP or in other words, deallocate an unassigned Elastic IP. But before that, let us know…
What is an Elastic IP and why do we need it?
Elastic IPs are simply IP addresses provided by Amazon AWS. These are usually attached to internet gateways such as load balancers, or even EC2 instances. I have been using it with EC2 instances directly for my testing use case.
Normally when you configure your EC2 instance to have a public IP but at the same time, you don’t use EIP. The problem is that when you stop and start your instance, your IP address is changed.
This is where Elastic IP shines. You attach an EIP to the EC2 instance and when you stop and start your instance, your IP won’t be changed.
This is extremely important if the EC2 you are running is hosting an internet-facing web server. Web servers shouldn’t generally frequently change their IPs.
But why do we need an EIP killer?
There is 1 good thing about EIP and 1 bad thing. The good thing is EIP is free to use. You don’t need to pay anything for the static IP you are using. How cool!
The bad thing is, that when EIP is not attached to an instance, it cost you money. How warm!
It won’t cost you much if you leave it detached for a few hours. But as a human, we tend to forget if EIP is in use and only get to know about it when we get our monthly bill.
In the Asia Pacific (Mumbai) region, if I forget to kill an EIP for a month, I get around a $4 bill. This was my main motivation to write this article.
With an EIP killer, I will be able to use Elastic IP with confidence by releasing it when not in use.
What we’ll work with:
- AWS Lambda
- AWS Go SDK
- AWS CLI
What we’re going to do:
- Run a Lambda on a predefined interval.
- This Lambda will use AWS SDK to check if any dangling EIP is lying around.
- If dangling EIP is found, deallocate, or in our language, kill it.
Initially, I was going to use Python for the Lambda runtime, but Go is faster than Python. We are going to benefit from this decision as we can run our function for a smaller time span. This is important because we plan to run our Lambda at frequent intervals.
With this post, I mostly plan to expand my knowledge in the field of serverless computing. I intend to touch Lambda, EventBridge, and related constructs.
Phase 1: Provision a Lambda function
The main task in this phase is to manually create a function. In the second phase, we’ll see what is the logic for killing the EIPs. Let’s get started.
- Go through the web UI and choose to create a function.
- Keep “Author from scratch” selected.
- Fill in the name. I liked to call it “eip-killer”.
- Select the runtime to be Go 1.x.
- Keep everything else on default. Hit Create function.
Note: Creating Lambda with default settings also creates an IAM role. This IAM role dictates how this Lambda interacts with other AWS services. By default, this role allows Lambda to talk with CloudWatch to put logs. You can add more policies to this role when working with other AWS services.
Let’s go ahead and do a quick invoke to test if the function is working.
Our EIP-killer is alive. But doesn’t know how to kill. Let’s teach it in the next phase.
Phase 2: Write the EIP killing logic
While testing, you might have got a view of the Code tab of the function which says, “The code editor does not support the Go 1.x runtime”. What this means for us developers is that we have to use our text editor to write the logic and upload the pre-built binary to Lambda.
The prime documentation for Go AWS SDK is https://docs.aws.amazon.com/lambda/latest/dg/lambda-golang.html, but don’t worry. I’ll filter out the required material required for this post.
Phase 2.1: The initial deployment process
Before we write the actual logic, let’s write a simple “Hello EIP Killer” code to get the awareness of the process required for uploading the code.
- Create a workspace for our lambda code.
mkdir eip-killer cd eip-killer
- Initialize a go module in the folder.
go mod init EIP-killer
You could also put the GitHub/GitLab URL of the repo where you intend to put your lambda code at.
- Create a main.go and put this code into it:
As we are using external dependency, get it with this command:
go get github.com/aws/aws-lambda-go/lambda
- Build a binary out of main.go:
GOOS=linux GOARCH=amd64 go build -o main main.go
Remember while creating the function we had selected architecture to be x86_64. So we need to reflect that with
- Zip the binary. Upload the zip to function.
zip main.zip main
Upload using this command:
aws lambda update-function-code --function-name eip-killer --zip-file fileb://main.zip
While this could have been enough, the default handler (where lambda begins executing) in function with factory settings is
hello. But our code uses
main. So we need to update the handler config to match that.
We need this update only once.
aws lambda update-function-configuration --function-name eip-killer --handler main
After this, when you run the func:
You can roam around this path: https://github.com/aws/aws-lambda-go if you want to dig deeper into Go+Lambda. But for now, I’m going to use AWS SDK to do our magic.
Phase 2.2: Logic to deallocate unused EIPs
Open up the main.go again and write this:
Quite a large source code. Let us understand it line by line.
Line 14-16 declares an event struct. This struct is going to be a parameter to the Handler function.
On line 18, we see function with signature
Handler(ctx context.Context, name MyEvent) (string, error).
Handler here is the name of the function.
The first parameter to the handler is the context object. If you are working with web services in golang, you must be aware of that.
Next is our
MyEvent which is defined above. This is as per Lambda’s spec.
Our handler function also returns a string and an error. Which is then bubbled to Lambda logs.
Line 20 deals with configuration. Please note that we are using AWS SDK. And the SDK can run on platforms other than Lambda. On the local system, the default config is located at ~/.aws/config. But on Lambda, it automatically loads the region it’s running in.
On line 26 we create a new EC2 client to interact with our AWS account. We initialize this client with the config instance we created on line 15.
On line 29 we have initialized a filter for IPs. This seems to be an added step. But this variable is going to be a mandatory parameter for our next command.
On line 32 we get a list of all Elastic IPs associated with our account. There are no filters to the result as we have not specified any filter on line 29.
It’s time to look at the
isThisEIPKillable function which is from 69 to 76. This is an extra feature for flexibility. This will forgive any EIP which has the tag
KILLERHOLD set to
true. So when you have an EIP you don’t want to kill, you set this tag to it. This is good for the experiment as our Lambda will be running on a 1 hours (configurable by user) interval and it might not be desired outcome to delete them every 1 hour.
Let us get back to our main flow. We have a
result variable with all the addresses in it. On line 40 we check if the IP is associated with any EC2 or not. Then we run the helper function we just discussed. If all is okay, meaning that KIP can be released, we go ahead and release the address back to the AWS pool.
On lines 57-59, if EIP is associated with any EC2, we print some metadata about the EIP association.
That’s it. It was not that hard to understand, isn’t it? Before we deploy it, let’s create some EIP and not allocate it to any EC2. We’ll also have some EIP having KILLERHOLD set to true.
Test our EIP killer Give Lambda permission to release and read EIPs
I hope you have updated your lambda function and pushed the zip file to the function. Review Phase 2.1 if you have not. Continue reading if you did.
Before I test it with actual EIPs, I wanted to show something else. Right now if you try to test the function, you’d see something like this:
If you clearly read the error message, it says that API call returned 403. Meaning that Lambda was not authorized to make the call to EIP APIs.
To overcome this problem, we need to give access to Lambda to read some metadata from EC2.
When no Lambda Web UI for the function. If you go to Configuration > Permission > Execution role as described in the image below. You’ll see a role that was created while we created the function.
Click on that and you’ll be taken to the IAM page for that role.
When on that page, look for a dropdown named Add permissions and click on Create inline policy.
Switch from Visual editor to JSON. And paste this policy.
As you can see, in the Action section, we have added permission for
Review policy and give it a name, and then Create policy.
When done, you’d see this policy in the list of policies attached to the role.
Phase 2.3: Test our EIP killer
We are not set for the actual testing.
We are going to allocate 3 Elastic IPs for our testing purpose.
One we’ll attach to the EC2 instance.
The second will be dangling.
And the third one will be also dangling but will have our KILLERHOLD set to true.
To actually invoke the function (apart from testing it), one way is to create a public URL. We can also invoke through the AWS CLI, but I’m not choosing that for now. You can do it.
Then I went ahead and opened that URL to execute the function.
You can configure AWS API Gateway in front of the function. But that’s out of the scope of this post.
Next, we see the result of the invocation.
That’s the end of Phase 2. We can kill EIPs now. How wonderful.
Phase 3: Trigger Lambda at a 1-hour interval
The main part of the EIP killer is already developed. Let’s now execute it at a 1-hour interval as planned before.
Phase 3.1: Configure EventBridge aka CloudWatch Events
We are going to emit an event at our 1-hour interval and we are going to use EventBridge for this.
I have recorded a screencast gif to preserve some space on this post.
Once done, this event should appear in the Trigger section of the Lambda function; as seen in the below image
Phase 3.2: Create an EIP and go to sleep
As I’m writing this post at 4 am. I’m going to create an EIP and go to sleep. Will check in the morning if our setup is working.
A few hours later…
I woke up this morning with only 1 EIP.
And this is how we gonna achieve this killing at 1 hour. You can configure the interval, but I think this is optimal. You also don’t need to have a public IP for this. So if you have, you can delete it now from the web interface.