Customizing the Deployment - AWS Elastic Beanstalk
The application deployment in AWS Elastic Beanstalk environment can be customized using the configuration files and you can read Amazon's documentation on it here.The configuration file provided here could be used to customize AWS Elastic Beanstalk environment whose container type could be Tomcat, Python, PHP or Ruby.
We need to make a source bundle containing our application and provide it to Elastic Beanstalk using its console or command line utility for deployment. The source bundle must have a directory ".ebextensions" at the top most level. It is in this ".ebextensions" directory we need to have the configuration file. We are free to have any number of directories and files under ".ebextensions". At the time of deployment when the configuration file is executed the current working directory is set to ".ebextensions" and all the files and directories under it become available to us.
hudku.com is a Tomcat application and the source bundle is nothing but a war file and it contains ".ebextensions" folder at the top level. We have our config file inside the ".ebextensions" directory.
We have named our configuration file "elastic-beanstalk.config" and this is the main and only configuration file we use. The syntax of the file is YAML compliant and various sections and commands that can be used are covered in the Amazon's documentation.
Our objective is to just do enough in this configuration file so that it finally runs our main setup bash script "app-setup.sh". Once we get control inside a bash script, then we are in our familiar turf where we know how to get our job done completely using all the wonderful features provided by bash and fleet of commands provided by Linux. In effect all the setup tasks are performed by "app-setup.sh" so that we could run it as a standalone on any Linux machine.
You can take the configuration file provided here, change the application name from "hudku" to the name of your choice and you should be able to get going with AWS Elastic Beanstalk right away.
Here are the contents of the "elastic-beanstalk.config", our main and only configuration file.
# Errors get logged to /var/log/cfn-init.log. See Also /var/log/eb-tools.log packages: yum: dos2unix: [] container_commands: 01-command: command: rm -rf /hudku/.ebextensions 02-command: command: mkdir -p /hudku/.ebextensions 03-command: command: cp -R .ebextensions/* /hudku/.ebextensions/ 04-command: command: echo "ELASTICBEANSTALK_CMD_LEADER=true" > /hudku/.ebextensions/copy-to-slash/root/.elastic-beanstalk-cmd-leader leader_only: true 05-command: command: dos2unix -k /hudku/.ebextensions/scripts/main/app-setup.sh 06-command: command: chmod 700 /hudku/.ebextensions/scripts/main/app-setup.sh 07-command: command: bash /hudku/.ebextensions/scripts/main/app-setup.sh The only package we install using the configuration file is "dos2unix" to convert the line endings of files from Windows (CRLF) to Linux (LF or '\n' character). Without this the main script "app-setup.sh" cannot be run if the file has been edited in Windows environment.Then we copy the entire ".ebextensions" directory to a known location. In our case we copy it under the directory "/hudku" after removing any existing ".ebextensions" directory. I did not know how to use a variable such as APP_NAME so that file could have been made more generic. Would appreciate if anybody can suggest a solution in the comments section of this post.
The configuration script gets executed on every EC2 instance available in the environment. If a particular command has to be executed only for the leader of the auto-scaling group then we need to specify "leader_only: true" for that command. We do that and create a file ".elastic-beanstalk-cmd-leader" so that at any point when we execute a bash script later we could identify whether the current EC2 instance is just a member or the leader of the auto-scaling group.
Next we fix the line endings of the main bash script file "app-setup.sh" using the "dos2unix" package we had just installed.
Set the proper permissions for the script file and execute the main setup script file. Once the script starts executing we get the full control to accomplish all the tasks we need to. As it is bash we are no longer in an unfamiliar territory of a YAML compliant configuration file and constricted by its restrictions.
We are Free as we have gained our Freedom. We are Free as in not being dependent. We are just set Freeeeeee.
You might ask what is so big deal about this config file whose contents are displayed above?
You started it. You just started it!! The config file is rolling on the floor laughing at me. It is ridiculing me because it knows all the torture it has inflicted on me. Let me start explaining everything for your as well as everybody's benefit.
I also thought the same when I first read the AWS documentation "Customizing and Configuring AWS Elastic Beanstalk Environments". I was really happy to know that there is some mechanism to customize. Having worked with batch files in Windows and the shell scripts under Linux I thought it is not going to be a sweat and we could start our customization easily.
Big mistake. The fact is that is the only single page of documentation available on the configuration file format. Even a simple copy command in Linux has pages of documentation explaining its different options. This configuration file has various sections and we need to do several things as part of setup and without detailed examples of each syntax it was very difficult to get started. Another couple of links I could find on the Net were just reproducing the same information without any additional help or details. It took almost 3 days of searching for information before we decided to go ahead and learn on our own.
Obviously we could not get it working straight away. To know which command had failed we had to look in the log files. But if you do not know which log files had to be looked into then imagine the plight. We started looking at all the files available in the system barring the system swap file, but to no avail. Then in the config file we had this command echo "This line is not supposed to be in any log file" and then after deploying the application with that modification we tried to search using grep for that exact line. Still we could not find it because we had a bug even before the echo could get executed. Finally somehow managed to figure out which files are supposed to contain the output of our commands. Immediately what we did was to have a comment in the very first line containing the names of the log files that need to be looked at so that at least from now on either us or you do not have to go through this ordeal of figuring out the names of the log files.
The YAML format was also not very easy for me to get hold of. What I understood is we need to be careful with the indendations of the lines and wherever necessary the blank lines are also important. I can't count how many online YAML parsers and validators I found online and every time I changed anything in the file I used to ensure that the changes pass the tests of those online parsers. Now you do not have to do any of those if you just use the file presented in this post.
The config file is sporting a challenging smile. It is thinking whether I will be able to recollect and list each one of the problems it had caused. I am not looking at it and trying to concentrate hard.
You might be wondering why I have only one command per line instead of just having a nice heading such as "commands" and have all the commands we need listed under it. I did that only to figure out that the commands are NOT executed in the order they are listed in the file. Instead all the commands get sorted and are executed in "alphabetical order". Don't ask me which character code page is used for the sorting and whether it is accent sensitive. All I can say is this file is extremely insensitive to a hapless developer coming across it for the first time and I happened to be born as one such unfortunate soul.
Even after knowing this I happened to make a mistake. After some copy / paste operations and writing some more commands, one of the commands listed down below had a lesser number and it got executed much before than it is supposed to because of the "alphabetical order" execution. By this time I was starting to get a grip of which log files to look at and figure out what went wrong.
Soon I realized that for every deployment I was deploying our real application whose size is large to upload. It dawned on me to create an application with a single file "index.html" which simply displayed "Hello World - Elastic Beanstalk Configuration Script". Then I made the war file which contained only this html file along with the entire ".ebextensions" folder so that uploading the very small war file became easy. That way we could completely concentrate on getting the configuration scripts right and we never used the real application until we reached the complete automation which was the destination. This made it easy to carry out the cycle of upload, deploy, test, fix or make enhancements and upload again.
The file has started grinning again and this time it is laced with a sadistic satisfaction. It is extremely happy that it hit me with this particular problem. Again due to human mistake I had used the same number for two commands. But fortunately one was trying to execute "chown" and the other "chmod". If it had executed them in any order it would have been ok. But had no such luck. Looks like the sorting filter used cannot handle duplicates. It dropped dead. I was staring at the contents of the file for a long time before I was able to figure out that I am using the same number for two different commands.
It took a good 15 days to become aware of all the land mines this file possessed and then I started cruising with the tasks we had to do. But that period was hell, extremely agonizing and left me badly battered and bruised. It actually changed my attitude towards the other problems we face in our profession. Here is a list containing few samples.
Blue screen of death has started appearing like a beautiful damsel.
I am finding those hour glass and loading icons so cute that I can keep watching them to eternity.
core-dump feels like a cotton candy - very light and sweet.
I find StackOverflow (pun intended) to be highly educative. It teaches us that a recursive function must have a terminating condition and it better be near by and not lie somewhere half way across the globe.
Fortunately there is no stack-underflow problem. One less critter to deal with. Thank God for his mercy.
over-flow and under-flow errors seem to be kind. They do not occur together as long as the blood-flow to the brain is normal.
Buzzing of CPU seems to have some rhythm whenever it gets stuck in an infinite loop.
off-by-one error does not put me off any more.
divide-by-zero does not seem to be an infinite problem.
NaN sounds like the name of a caring person.
Deadlock need not be a deadly embrace if we carefully pay attention to its prevention techniques.
Memory overwrite and other memory related problems are under control with the advent of techniques and tools like _malloc_dbg, DEBUG_NEW, Valgrind, Jmap, MAT, etc.
Wherever I need to, I diligently assert the steps I take and the assumptions I make.
You see, no other problem looks nasty any more. But I wonder why I was not spared by not even one and instead got hit by all the possible ways this file could bite. What? Is it so that I could write this blog post and alert everybody?! How cruel!!
Now when I think about it, may be that's the right way to look at. As developers, we can't lose heart. Let us turn this horrific experience into our advantage. Let my bloodshed be the last this devil is ever going to get. From now on let us deny this sucker even a drop of sweat of another developer. If you are planning to work with AWS Elastic Beanstalk, just take the file provided here, change the app name from "hudku" and you should be fine.
Wait. I see that this config file is laughing uncontrollably tapping its butt with all the hands it has got. I do not like this file at all. Whether I like it or not I seem to have become good in interpreting its wicked thoughts. I found the reason. I tell you this config file is evil, cunning, crooked and wicked.
This time it is thinking that anybody using the file presented here could get going easily. But what about those developers who have not read this post? It is smacking its lips in anticipation of their blood.
I am sure like all the previous readers before you, you also agree to alert all our fellow developers, friends, colleagues, batch mates and everybody whom we care for by clicking those social buttons available below this article.
On a more serious note, for anybody already using AWS Elastic Beanstalk, about to start using or might use it in future, by using this file and the information provided here, if some valuable man hours could be saved then that is exactly what we seek from this blog post as well as our commitment to do serial of posts covering AWS Elastic Beanstalk Configuration and Automation.
As more and more developers become aware of this file and its nefarious ways this devil's teeth would get blunted and its blood thirsty tongue would dry up. I think with our discussion we have managed to wipe the smile from this file. I can see its expression changing to defeatist.
Oh no. I spoke too early. The config file is wearing some seriously wicked grin. Let me see what I have left out and why is this file trying to regain its hope to hurt?
Oh yes. I remember now. This is a killer. You will just not believe it. Can you believe if I say this file can cause damage even if there are no bugs in this file? How about that? I had never come across such a thing before. Have no bugs and can still get bitten?
We happened to carry a file by name "myWebApp.config" in one of the directories under ".ebextensions" and the file contained some XML. Our objective was to copy it to the appropriate location where it was required. But the deployment resulted in an error and when I looked into the log file, it had a line something like "$$##**fffffff**_sob@http.tcp.com##$$" which gave me no clue about what went wrong. Later figured out that the YAML wizard after executing the main config file had tried to execute this XML file also just because its name also happened to end in ".config". If you read the Amazon's documentation, under the subheading "Using Configuration Files", in the first point it is stated that we can have multiple config files and they all get executed.
I am so thankful that Windows during boot up executes only one batch file "autoexec.bat", instead of executing all the batch files present in the hard disk.
So please do not have any file with its extension as ".config" anywhere at any level in the ".exextensions" directory. In case if you must, then have it with a different extension and at the time of copying that file to its destined location do the copy and rename.
Steve Maguire in his book "Writing Solid Code", in the chapter "Candy Machine Interface" and in the sub chapter "The One-Function Memory Manager" warns us how the C memory library function "realloc" tries to do too many things being a single function and advises us why an interface should not be designed that way. I wonder what he would say about this configuration file which also can do too many things, that too each one being more evil than the other.
I am heaving a sigh relief. I just shared with you all the things I had learnt about this config file.
This config file is not an unassailable devil after all. For we developers, taming the beasts is a pastime. Isn't it? If this happens to be our profession do we require anything more? If the heaven exists, just burn it down as we do not need it because for us the heaven is right here on this earth.
If you want to experiment further by sticking semi-colon at the end and full-colon at the beginning of every line or every alternate line and see what happens, I am not joining that game. I am getting off here.... with the message
Stay Away! Please Stay Away from this Config file!!!