The two main environment variables that contain AWS account's security credentials are AWS_ACCESS_KEY and AWS_SECRET_KEY. Almost all the command line utilities of AWS make use of the values of these two environment variables if they are defined. If they are not defined then you will have to specify them as command line arguments while invoking each utility.
There exists one more method to provide these security credentials and that is by defining an environment variable AWS_CREDENTIAL_FILE whose value is the complete path to a file containing the security credentials. At hudku.com we have called this file ".aws-credentials" and carry it in the home directory of the user. We allow only the root user or the user with super-user privileges to access these information and execute the command line utilities.
The format of ".aws-credentials" file is as below.
AWSAccessKeyId= AWSSecretKey= There are few more utilities which we use and they also require the same credentials in different format. We found the tool s3cmd to be extremely useful for downloading or uploading files to Amazon S3 and you can read more about s3cmd here. It can make use of the settings specified in the file ".s3cfg" if that file is present in the user's home directory. The config file gets created by executing the command "s3cmd --configure". I shall list only those two lines that carry AWS credentials.
access_key = secret_key = We use another tool "dnscurl.pl" to interact with Amazon Route53 and you can read more about this tool here. This tool expects AWS credentials to be in a file ".aws-secrets" and located in the home directory of the user. Please note that there is already a AWS command line utility route53 that is installed by default and we make use of this tool too in our scripts. Here is the format of the file ".aws-secrets".
%awsSecretAccessKeys = ( "myAppName" => { id => "", key => "", }, ); We carry all the files discussed above as part of our application war file and during deployment we copy them to the user's home directory. But as you might have noticed they do not contain the value of the AWS credentials as they are left empty in all the files.
The way we make those credentials available is when the very first time we deployed our application, we provided those credential values using AWS Elastic Beanstalk console as described here. This link is applicable for only Java applications but in that page itself you can find links for PHP, Python and Ruby applications. As per those instructions, using AWS Elastic Beanstalk console, only for the very first time we provide the values for the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_KEY. We all know that a person with valid authorization only can access AWS consoles. Once these credentials are provided, from then on our scripts remember them and make them available whether we create a new development or staging environments or if we deploy a new version of application to any of the existing environments.
Thus the security credentials are not part of our source code, we do not have to provide it for every deployment, but all the EC2 instances in the environment have them and remain with them. When we access EC2 using ssh, unless a person knows the root user password or has super-user privileges, these security credentials cannot be accessed.
Apart from AWS credentials we may have some more security credentials such as user name and password for database, git, etc. What we have done is to put them in a file and store it in a known location in one of the Amazon S3 buckets. Using the AWS credentials, every EC2 instance can access the S3 bucket and download that file. We have called this file ".elastic-beanstalk-app" and we carry it in our source bundle. Only the first three lines carry actual values which are the application name, the private S3 bucket name and the location of the deployment related files. During deployment, we copy this file to the user's home directory. We carry the same file in Amazon S3 also which contains valid values for all the environment variables. As part of the deployment we download that file containing all the values from S3 and overwrite the template file in the home directory. The template of the file ".elastic-beanstalk-app" is provided below.
#!/bin/bash # AWS Elastic Beanstalk Application Configuration Settings # Elasticbeanstalk app main settings to bootstap the deployment export ELASTICBEANSTALK_APP_NAME="myAppName" export ELASTICBEANSTALK_APP_PRIVATE_S3_BUCKET="bucket.private" export ELASTICBEANSTALK_APP_DEPLOY_DIR="folders/deployment" # # Only the above information is mandatory. Information below has to be present in the file # named .elastic-beanstalk-app contained in the zip file secrets.zip. At the time of # deployment that zip file gets retrieved from S3 and unzipped overwriting any # existing file. Please see the script aws-credentials-setup.sh which does the retrieval # # # Below settings are provided for only illustrative purpose. # # Apache settings export APACHE_DIR="/var/www/myApacheWebAppDir" # Tomcat settings export TOMCAT_DIR="/usr/share/tomcat7" export TOMCAT_PRIMARY_INSTANCE_NAME="tomcat7" export TOMCAT_SECONDARY_INSTANCE_NAME="myTomcatSecondaryInstanceName" # AWS credentials export AWS_CREDENTIAL_FILE="/root/.aws-credentials" export AWS_ACCOUNT_KEY_NAME="myKeyName used in file .aws-secrets" # RDS settings export RDS_INSTANCE_NAME="myRDSInstaceName" # Route53 Zone and ResourceRecord settings export ROUTE53_ZONE_NAME="myDomainName.com." export ROUTE53_RR_PRODUCTION_NAME="$ELASTICBEANSTALK_APP_NAME-prod.$ROUTE53_ZONE_NAME" export ROUTE53_RR_STAGING_NAME="$ELASTICBEANSTALK_APP_NAME-staging.$ROUTE53_ZONE_NAME" export ROUTE53_RR_DEVELOPMENT_NAME="$ELASTICBEANSTALK_APP_NAME-dev.$ROUTE53_ZONE_NAME" # Git settings export GIT_USER_NAME="myGitUserName" export GIT_PASSWORD="myGitPassword" Then there are few more environment variables that are required by AWS command line utilities. AWS_REGION is one such important variable. EC2 command line utilities need EC2_REGION and EC2_URL variables. If we have them exported then we could use the command line utilities without any hassle.
Apart from these we could have some private environment variables that are extremely useful and we could make use of them in our scripts or in our application. For example, it would be nice to know the current EC2 instance type, instance id, etc. If the instance is connected to a load balancer then it would be handy to have the ELB endpoint URL. Similarly we have identified several such useful variables and in the home directory we have all these variables declared in the file "env-result.sh". Interesting point is this file is "auto generated". Here are the contents of "env-result.sh".
# This file is generated by running ~/env.sh and contains all export statements with values. export ELASTICBEANSTALK_APP_NAME="" export ELASTICBEANSTALK_APP_PRIVATE_S3_BUCKET="" export ELASTICBEANSTALK_APP_DEPLOY_DIR="" export APACHE_DIR="" export TOMCAT_DIR="" export TOMCAT_PRIMARY_INSTANCE_NAME="" export TOMCAT_SECONDARY_INSTANCE_NAME="" export AWS_CREDENTIAL_FILE="" export AWS_ACCOUNT_KEY_NAME="" export RDS_INSTANCE_NAME="" export ROUTE53_ZONE_NAME="" export ROUTE53_RR_PRODUCTION_NAME="" export ROUTE53_RR_STAGING_NAME="" export ROUTE53_RR_DEVELOPMENT_NAME="" export AWS_ACCESS_KEY="" export AWS_SECRET_KEY="" export ELASTICBEANSTALK_APP_DIR="" export ELASTICBEANSTALK_APP_EXT_DIR="" export ELASTICBEANSTALK_APP_TMP_DIR="" export ELASTICBEANSTALK_APP_DATA_DIR="" export ELASTICBEANSTALK_APP_SCRIPT_DIR="" export AWS_REGION="" export EC2_REGION="" export EC2_URL="" export EC2_INSTANCE_ID="" export EC2_INSTANCE_TYPE="" export EC2_INSTANCE_URL="" export ELB_NAME="" export ELB_HOSTEDZONE_NAME="" export ELB_HOSTEDZONE_ID="" export ELB_URL="" export ELASTICBEANSTALK_CMD_LEADER="" export ELASTICBEANSTALK_URL="" export ELASTICBEANSTALK_S3_BUCKET="" export ELASTICBEANSTALK_ENV_ID="" export ELASTICBEANSTALK_ENV_NAME="" export ELASTICBEANSTALK_CNAME="" export ROUTE53_ZONE_ID="" export MYSQL_USER="" export MYSQL_PASSWORD="" export GIT_USER_NAME="" export GIT_PASSWORD="" Finally we bring all the environment variables alive through the file ".bashrc". We overwrite the default file provided by Beanstalk environment with our version. First we modify the PATH to include some of the AWS command line utilities that we use. The last two sections contain some interesting code. We first check if the file "env-result.sh" exists. If not then we run "env.sh" which generates it. From then on that file would exist and there is no need to generate it every time. In our tests we found that it approximately takes around 10 seconds to generate that file. We do not have to pay this penalty every time as the values that are calculated do not change.
The information about Security Credentials could be used by anybody using EC2 and not just by Elastic Beanstalk users. In our next post we shall see how "env.sh" generates "env-result.sh" and also the script that downloads more credentials from Amazon S3.
Here are the contents of ".bashrc"
# .bashrc # User specific aliases and functions alias rm='rm -i' alias cp='cp -i' alias mv='mv -i' # Source global definitions if [ -f /etc/bashrc ]; then . /etc/bashrc fi export AWS_ELASTIC_BEANSTALK_HOME=/opt/aws/AWS-ElasticBeanstalk-CLI-2.2 export AWS_CLOUDFORMATION_HOME=/opt/aws/AWSCloudFormation-1.0.11 export AWS_SNS_HOME=/opt/aws/SimpleNotificationServiceCli-1.0.3.3 if [ -e $AWS_ELASTIC_BEANSTALK_HOME ]; then export PATH=$PATH:$AWS_ELASTIC_BEANSTALK_HOME/api/bin fi if [ -e $AWS_CLOUDFORMATION_HOME ]; then export PATH=$PATH:$AWS_CLOUDFORMATION_HOME/bin fi if [ -e $AWS_SNS_HOME ]; then export PATH=$PATH:$AWS_SNS_HOME/bin fi # Run env.sh and generate env-result.sh if it does not exist if [ ! -f ~/env-result.sh ]; then if [ -f ~/env.sh ]; then bash ~/env.sh fi fi # Use env-result.sh if present if [ -f ~/env-result.sh ]; then # Get all the environment variables source ~/env-result.sh fi