Elastic Beanstalk: The Struggle

By: on February 28, 2019

Recently, I found myself tasked with modifying a bunch of AWS Elastic Beanstalk environments despite having little prior DevOps/AWS experience.

For the subsection of developers that, like me, now find themselves in the wide world of deploying their own code, I’ll describe a couple of the problems that I faced and how I solved them. Many are actually described in the documentation: I thought it important to collate them in a place for myself and others to find in the future.

Just give me the ‘t2.medium’

EB uses .ebextensions to configure the EB environment as it is created or deployed. I wanted to use ‘t2.medium’ instances because my application was somewhat CPU-intensive. But no matter how hard I tried with these configuration files, my instances always turned out to be ‘t2.micro’.

Honestly after a bit, I gave up on trying. I just moved on with my life (and my other tickets)…until only day or two later, I came across this:

“Options specified in ‘.ebextensions’ have the lowest level of precedence and are overridden by settings at any other level.” (source)

That was interesting. It turns out that EB defines an order of precedence for its configuration options. It also turns out that the EBCLI passes its own defaults when you run ‘eb create’, which override anything in .ebextensions due to the order of precedence.

To get around this, I ran the ‘eb create’  with the instance type specified:

To me, the sneaky bit is that EBCLI is setting values for you before looking at what you have defined in the configuration. Just make sure to take a look at the values that EBCLI sets during the `create` and override them yourself at the earliest stage.

To remove the default options after they are set you can run ‘eb config’, modify the file, save and EB will update the environment configuration for you.

All this manual work really bypasses the good practice of configuration-as-code, but c’est la vie.

Configuring Nginx

Elastic Beanstalk instances use Nginx to proxy requests from the load balancer to your application. Sometimes you may find yourself wanting to change how Nginx is configured. In my case, I wanted to increase the maximum allowed body size on incoming client requests which. Doing so would prevent Nginx from automatically returning a 413 when the client uploaded a large file.

Configuring Nginx on your Elastic Beanstalk changes based on the application type that you are using. If you have a java program or a node program, you may have to configure nginx differently than if you are using docker, which I was.

With java, you can add any configuration files to a specific directory in ‘.ebextensions’. With node, you have to rewrite the entire configuration and replace the one that EB creates for you.

But with docker, you need to use ‘.ebextensions’ to create a file in the correct location and then restart the Nginx server.

This is the default Nginx config file located at ‘/etc/nginx/nginx.conf’ on your EB instance:

If you look at line 22 you will see that Nginx is importing any configuration files matching ‘/etc/nginx/conf.d/*.conf’. To modify the configuration you need to use ‘.ebextensions’ to create a file in matching this pattern and then restart the Nginx service.

In the ‘.ebextensions’ directory in your EB root folder, create a file called ’01_nginx.config’ (the name of this file doesn’t matter, but do note that file commands are executed in alphanumerical order–– it is good practice to guarantee order by prepending numbers):

Lines 1-6 create the configuration file in the appropriate directory where Nginx will pick it up. 6-7 describe the configuration parameters that I wanted to update.

Lines 9-11 restart the Nginx service so that it will pick up the new config.

And, that’s it. Deploy your EB environment and Nginx will now be configured the way you want. Remember that you can always ‘eb ssh’ and try the changes out manually first.

Shutting Things Down

EB will creates a lot of AWS services around your instances. Out of the box, you get a load balancer, multiple security groups (one of them for the load balancer), and the instance itself. If you find yourself running EB a s a single instance, you won’t get the load balancer, but you will be assigned a dynamic IP. One gotcha is that AWS only allows you to have 5 dynamic IP addresses by default, so if you have a bunch of single instance EB environments, you can quickly run out (oops).

Because of all the surrounding paraphernalia, it sometimes was a pain to terminate an EB environment. If there is anything outside of the environment referencing it, such as an external security group, the terminate will stall for 20 to 30 minutes.

To avoid this, try to keep anything outside the environment from using any EB-created resources.

The End

I liked using Elastic Beanstalk, honestly. I’m a developer new to DevOps and in a lot of cases was trying to make EB do things that it wasn’t really meant to do. EB is a good existing solution to many problems. The kind of place where you deploy when you just need to get it up and running quickly. If you use it simply and don’t try to change too many things, it will work like a charm.


Leave a Reply

Your email address will not be published.

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>