Continuing from the previous post, let us see how we obtain the values for all the environment variables. Just to recap, almost all the AWS command line utilities make use of the environment variables AWS_ACCESS_KEY, AWS_SECRET_KEY and AWS_CREDENTIAL_FILE. The variable AWS_REGION, if set, it becomes the default region and we do not have to specify it explicitly on the command line.
In the same way EC2 (Elastic Compute Cloud) command line utilities make use of EC2_REGION and EC2_URL and you can read Amazon's documentation on it here.
Please note that these get used as default values. We can always override them by providing the appropriate option and its value on the command line.
We at hudku.com felt we needed some more private environment variables that we could make use of in our scripts. For example we compute EC2_INSTANCE_ID, EC2_INSTANCE_TYPE, EC2_INSTANCE_URL, etc. which we believe could be useful for any EC2 user. To compute these we make use of EC2 command line utilities as well as the tool "ec2-metadata" which you can read about it here. You can also refer to Amazon's documentation on obtaining metadata of an EC2 instance here.
Similarly we compute few ELB (Elastic Load Balancer) variables, Route53 variables and if the instance is part of Elastic Beanstalk Environment then we compute ELASTICBEANSTALK_ENV_ID, ELASTICBEANSTALK_ENV_NAME, ELASTICBEANSTALK_CNAME, etc. which we make extensive use of them in our scripts.
As mentioned in the previous post, we compute these values and store it in the file env-result.sh. The computation is done by the script env.sh. Please note that the methods used for computing various values are tested and are working correctly at present. In future if things change then we might have to adapt the script accordingly.
Last few lines of the script file env.sh are responsible for generating the file env-result.sh. Those lines scan the current script file env.sh, extract only "export" statements and then generate the output file. Any suggestion on improving the script is highly appreciated.
Here is the source code of env.sh.
#!/bin/bash # Execute "export DEBUG=1" to debug this script. # Set value to 2 to debug this script and the scripts called within this script. # Set value to 3,4,5 and so on to increase the nesting level of the scripts to be debugged. [[ $DEBUG -gt 0 ]] && set -x; export DEBUG=$(($DEBUG - 1)) # This file contains all the environment variables used by the scripts of elastic beanstalk application # Displays error in case the file does not exist source /root/.elastic-beanstalk-app # export all the variables sourced above from the app config file so that they are copied to the file env-result.sh # Elasticbeanstalk app main settings export ELASTICBEANSTALK_APP_NAME=$ELASTICBEANSTALK_APP_NAME export ELASTICBEANSTALK_APP_PRIVATE_S3_BUCKET=$ELASTICBEANSTALK_APP_PRIVATE_S3_BUCKET export ELASTICBEANSTALK_APP_DEPLOY_DIR=$ELASTICBEANSTALK_APP_DEPLOY_DIR # Apache related export APACHE_DIR=$APACHE_DIR # Tomcat related export TOMCAT_DIR=$TOMCAT_DIR export TOMCAT_PRIMARY_INSTANCE_NAME=$TOMCAT_PRIMARY_INSTANCE_NAME export TOMCAT_SECONDARY_INSTANCE_NAME=$TOMCAT_SECONDARY_INSTANCE_NAME # AWS credentials settings export AWS_CREDENTIAL_FILE=$AWS_CREDENTIAL_FILE export AWS_ACCOUNT_KEY_NAME=$AWS_ACCOUNT_KEY_NAME # RDS settings export RDS_INSTANCE_NAME=$RDS_INSTANCE_NAME # Route53 settings export ROUTE53_ZONE_NAME=$ROUTE53_ZONE_NAME export ROUTE53_RR_PRODUCTION_NAME=$ROUTE53_RR_PRODUCTION_NAME export ROUTE53_RR_STAGING_NAME=$ROUTE53_RR_STAGING_NAME export ROUTE53_RR_DEVELOPMENT_NAME=$ROUTE53_RR_DEVELOPMENT_NAME # AWS Security Credentials source $AWS_CREDENTIAL_FILE export AWS_ACCESS_KEY=$AWSAccessKeyId export AWS_SECRET_KEY=$AWSSecretKey # Set up few derived environment variables of the app export ELASTICBEANSTALK_APP_DIR="/$ELASTICBEANSTALK_APP_NAME" export ELASTICBEANSTALK_APP_EXT_DIR="$ELASTICBEANSTALK_APP_DIR/.ebextensions" export ELASTICBEANSTALK_APP_TMP_DIR="$ELASTICBEANSTALK_APP_DIR/tmp" export ELASTICBEANSTALK_APP_DATA_DIR="$ELASTICBEANSTALK_APP_EXT_DIR/data" export ELASTICBEANSTALK_APP_SCRIPT_DIR="$ELASTICBEANSTALK_APP_EXT_DIR/scripts" # EC2 environment variables EC2_ZONE=$(/opt/aws/bin/ec2-metadata -z | grep placement: | awk '{print $2}') if ([ -f /etc/elasticbeanstalk/.aws-eb-stack.properties ]) then EC2_REGION=$(cat /etc/elasticbeanstalk/.aws-eb-stack.properties | grep "region=" | cut -d= -f2) else EC2_REGION=$(curl -fs http://169.254.169.254/latest/dynamic/instance-identity/document | grep "\"region\"" | awk '{print $3}' | sed s/\"//g) fi EC2_URL=https://$(/opt/aws/bin/ec2-describe-regions $EC2_REGION | grep $EC2_REGION | awk '{print $3}') EC2_INSTANCE_ID=$(/opt/aws/bin/ec2-metadata -i | grep instance-id: | awk '{print $2}') EC2_INSTANCE_TYPE=$(/opt/aws/bin/ec2-metadata -t | grep instance-type: | awk '{print $2}') EC2_INSTANCE_URL=$(/opt/aws/bin/ec2-metadata -p | grep public-hostname: | awk '{print $2}') export EC2_REGION export EC2_ZONE export EC2_URL export EC2_INSTANCE_ID export EC2_INSTANCE_TYPE export EC2_INSTANCE_URL # AWS environment variables AWS_REGION=$EC2_REGION export AWS_REGION # ELB environment variables allRecords=$(/opt/aws/bin/elb-describe-lbs --show-xml) # Setup XPath query to obtain the ELB record containing the current EC2_INSTANCE_ID elbRecordSearch="DescribeLoadBalancersResponse/DescribeLoadBalancersResult/LoadBalancerDescriptions/member[ Instances/member/InstanceId = \"$EC2_INSTANCE_ID\" ]" elbNameSearch="$elbRecordSearch/LoadBalancerName" ELB_NAME=$(echo $allRecords | xpath "$elbNameSearch" 2> /dev/null | cut -d">" -f2 | cut -d"<" -f1) elbHostedZoneNameSearch="$elbRecordSearch/CanonicalHostedZoneName" ELB_HOSTEDZONE_NAME=$(echo $allRecords | xpath "$elbHostedZoneNameSearch" 2> /dev/null | cut -d">" -f2 | cut -d"<" -f1) elbHostedZoneIdSearch="$elbRecordSearch/CanonicalHostedZoneNameID" ELB_HOSTEDZONE_ID=$(echo $allRecords | xpath "$elbHostedZoneIdSearch" 2> /dev/null | cut -d">" -f2 | cut -d"<" -f1) elbDNSNameSearch="$elbRecordSearch/DNSName" ELB_URL=$(echo $allRecords | xpath "$elbDNSNameSearch" 2> /dev/null | cut -d">" -f2 | cut -d"<" -f1) export ELB_NAME export ELB_HOSTEDZONE_NAME export ELB_HOSTEDZONE_ID export ELB_URL # ELASTICBEANSTALK environment variables if ([ -f /root/.elastic-beanstalk-cmd-leader ]) then source /root/.elastic-beanstalk-cmd-leader fi export ELASTICBEANSTALK_CMD_LEADER=$ELASTICBEANSTALK_CMD_LEADER if ([ -f /etc/elasticbeanstalk/.aws-eb-stack.properties ]) then ELASTICBEANSTALK_URL=$(echo $EC2_URL | sed s/ec2\./elasticbeanstalk./) ELASTICBEANSTALK_S3_BUCKET=$(cat /etc/elasticbeanstalk/.aws-eb-stack.properties | grep "environment_bucket=" | cut -d= -f2) ELASTICBEANSTALK_ENV_ID=$(cat /etc/elasticbeanstalk/.aws-eb-stack.properties | grep "environment_id=" | cut -d= -f2) ELASTICBEANSTALK_ENV_NAME=$(/opt/aws/bin/ec2-describe-instances $EC2_INSTANCE_ID | grep "elasticbeanstalk:environment-name" | awk '{print $5}') envInfo=$(elastic-beanstalk-describe-environments -e $ELASTICBEANSTALK_ENV_NAME -j) ELASTICBEANSTALK_CNAME=$(echo -e "$envInfo" | grep -ioP "CNAME\":.*?elasticbeanstalk\.com\"" | cut -d, -f1 | cut -d: -f2 | sed s/\"//g) fi export ELASTICBEANSTALK_URL export ELASTICBEANSTALK_S3_BUCKET export ELASTICBEANSTALK_ENV_ID export ELASTICBEANSTALK_ENV_NAME export ELASTICBEANSTALK_CNAME # ROUTE53 environment variables export ROUTE53_ZONE_ID=$(route53 ls | sed -n -e '/'"$ROUTE53_ZONE_NAME"'/{g;1!p;};h' | awk '{print $3}') # MYSQL environment variables if ([ -f /etc/my.cnf ]) then MYSQL_USER=$(cat /etc/my.cnf | grep "user=" | cut -d= -f2) MYSQL_PASSWORD=$(cat /etc/my.cnf | grep "password=" | cut -d= -f2) fi export MYSQL_USER export MYSQL_PASSWORD # GIT credentials in case provided in the file .elastic-beanstalk-app export GIT_USER_NAME="$GIT_USER_NAME" export GIT_PASSWORD="$GIT_PASSWORD" # Generate envResult.sh using the settings from 'this' file envResult=$(cat /root/env.sh | grep -o "^[ \t]*export[ \t]*[^=]*" | sed "s/export[ \t]*\(.*\)$/export \\1=\\\\\"$\\1\\\\\"/") envResult=$(echo $envResult | sed "s/ export/\\\\nexport/g") envResult="echo -e \"$envResult\"" # By copying we get the same file permissions of env.sh to env-result.sh cp -f ~/env.sh ~/env-result.sh echo -e "# This file is generated by running ~/env.sh and contains all export statements with values.\n" > ~/env-result.sh eval $envResult >> ~/env-result.sh Now we shall see the script aws-credentials-setup.sh that fetches additional security credentials from Amazon S3 bucket. As explained in the previous post the main AWS credentials are set using the Elastic Beanstalk console. After that our scripts remember them and pass them on to any new Beanstalk environments we may create. Thus the environment variables AWS_ACCESS_KEY, AWS_SECRET_KEY should remain always set. In our case as hudku.com is a Tomcat application, we found that we could also obtain it from the tomcat configuration file used by the Beanstalk. Then we access the credential files which contain empty values such as ".aws-credentials", ".aws-secrets" and ".s3cfg" and update them with actual values. After that we can start using the command line utilities which make use of these credential files.
Please refer to ".elastic-beanstalk-app" explained in the previous post where we need to specify the name of the Amazon S3 private bucket and the name of the deployment folder in that bucket that is supposed to contain all the deployment related files.
We use the utility s3cmd to access the private bucket at Amazon S3 and download the "secrets.zip" file from the folder "secrets" which should exist under the deployment folder. The contents of the zip file are unzipped to "/" folder overwriting any existing contents. Thus if the secrets.zip file contains "root/.elastic-beanstalk-app" file then we would be overwriting the template file with the downloaded file having real contents. In the same way for example you could store "etc/my.cnf" file if you use MYSQL. Thus all the passwords and confidential information can remain in Amazon S3 private bucket and need not be part of the source bundle.
Please note that we always do the unzip as every deployment would result in template files with blank values overwriting the existing files. By doing unzip we overwrite them again with valid files. But downloading the zip file from Amazon S3 need not be done if the zip file has been downloaded before and exists in the expected location.
Hope you find our way of handling AWS Security Credentials useful. Please leave your opinion in the comments section.
In our next post we shall move on to setup tasks that we do whenever we deploy an application to AWS Elastic Beanstalk environment.
Presenting the source code of the script aws-credentials-setup.sh.
#!/bin/bash # Execute "export DEBUG=1" to debug this script. # Set value to 2 to debug this script and the scripts called within this script. # Set value to 3,4,5 and so on to increase the nesting level of the scripts to be debugged. [[ $DEBUG -gt 0 ]] && set -x; export DEBUG=$(($DEBUG - 1)) # # Setup AWS credentials so that all AWS command line utilities can work # # Obtain the AWS credentials from beanstalk tomcat configuration file if ([ -f /tmp/deployment/config/tomcat7 ]) then AWS_ACCESS_KEY=$(grep -o "DAWS_ACCESS_KEY_ID=[^ ]*" /tmp/deployment/config/tomcat7 | cut -d'"' -f2 | cut -d '\' -f1) AWS_SECRET_KEY=$(grep -o "DAWS_SECRET_KEY=[^ ]*" /tmp/deployment/config/tomcat7 | cut -d'"' -f2 | cut -d '\' -f1) fi # Update all the appropriate files with AWS credentials so that we can start using all AWS command line utilities sed -i -e "s/AWSAccessKeyId=.*/AWSAccessKeyId=$AWS_ACCESS_KEY/" -e "s/AWSSecretKey=.*/AWSSecretKey=$AWS_SECRET_KEY/" /root/.aws-credentials sed -i -e "s/id\s*=>.*/id => \"$AWS_ACCESS_KEY\",/" -e "s/key\s*=>.*/key => \"$AWS_SECRET_KEY\",/" -e "s/myAppName/$ELASTICBEANSTALK_APP_NAME/" /root/.aws-secrets sed -i -e "s/access_key.*/access_key = $AWS_ACCESS_KEY/" -e "s/secret_key.*/secret_key = $AWS_SECRET_KEY/" /root/.s3cfg # Obtain the secrets zip file from S3 containing passwords and other confidential information if ([ ! -z $ELASTICBEANSTALK_APP_PRIVATE_S3_BUCKET ] && [ ! -z $ELASTICBEANSTALK_APP_DEPLOY_DIR ]) then if [ ! -e ~/secrets/secrets.zip ]; then mkdir -p ~/secrets # As AWS credentials are setup, access S3 and obtain some more secured and confidential information pushd ~/secrets s3cmd --config=/root/.s3cfg get s3://$ELASTICBEANSTALK_APP_PRIVATE_S3_BUCKET/$ELASTICBEANSTALK_APP_DEPLOY_DIR/secrets/secrets.zip popd chmod -R 400 ~/secrets/ fi # Unzip the contents of the secrets zip file and overwrite any existing files unzip -q -o ~/secrets/secrets.zip -d / fi