Now we are ready to start the secondary instance of tomcat using the script "tomcat-start-secondary.sh".
In all the previous scripts I had been mentioning about only 3 ports that tomcat uses by default which are 8080, 8009 and 8443. For secondary instance we increase their numbers by 100 and use the ports 8180, 8109 and 8553. But in the previous script file if you had noticed we also changed another port 8005 to 8105. This is called Shutdown port used by tomcat. Tomcat primary instance and secondary instance run as two separate processes. No two processes can use the same port. That is fundamental which we all know. That's why we need to change that port also for the secondary instance. But we should not be redirecting that port. Only the other three ports we would be redirecting. This is what I wanted to share about the shutdown port.
Before we start the secondary instance let us do some preliminary checks in order to ensure that everything is proper.
First we need to check that some application files exist in the secondary instance directory. If they don't it means that previous script has not been run to copy the application files from the primary instance.
Then we check whether secondary instance is already running. If it is already running, but still we are trying to again start the secondary instance then may be we are working hard staying late. Just to remind ourselves that we should be going home before causing any major damage during such a tired state, we are doing these checks :-)
Then we check whether the AJP port 8109 is available. Just to ensure that secondary instance does not have any trouble while coming up.
Please leave your opinion whether any more useful checks that can be made or have we covered ourselves enough?
To do these checks I have implemented few bash functions and it should be obvious from the source code that they are all implemented in "include.sh" script file. More on that file in my future post. Here is the source code snippet of the functions we use in our current script file.
Now we are completely ready to start the secondary instance which we do by the following line.
Then the script waits for some time to allow the secondary instance to come up.
We check whether the application has come up properly by using Linux command "curl" to retrieve a HTML web page. The web page could be just the home page of the application. But at hudku.com we actually try to do a business search by executing a search query. As this is the first search query, it causes our application to initialize its search engine, do database access and execute all the important logic so that the application gets warmed up. Then when we do the port redirection and make it available to our users, the application is completely ready to service the user's search requests at full speed.
If the web page retrieval check looks good then we do the port redirection using "iptables". We first remove any existing redirection just in case and then redirect all the three important ports, 8080 ==> 8180, 8009 ==> 8109 and 8443 ==> 8543.
The above code works and is tested. I am not an expert on "iptables" and so far I have hardly scratched its surface. iptables with its various options such as OUTPUT, PREROUTING, SNAT, DNAT, etc., is quite vast and you could read about it elsewhere.
Let me share with you the information on iptables what I know and what is applicable to our current discussion. As I am quite confident of that much, I am sharing it with you. Otherwise I would have been happy to just provide the set of working commands and keep quiet.
In our application hudku.com it is the Apache server that listens to the HTTP port 80, receives packets from the internet and delivers content back to the internet. But internally Apache talks to Tomcat running at port 8080 (AJP 8009). Tomcat does not communicate to the outside world. That's why we use the iptables "OUTPUT" chain as we are only dealing with locally generated packets.
When we used to run our website hudku.com on a dedicated server we did not have Apache web server. The primary instance of Tomcat itself was directly listening to HTTP port 80. That means Tomcat was interacting with the outside world. The packets were coming into the server from the Internet and going out of the machine to the outside world. Tomcat's communication was not limited to within the machine. With this setup we used to bring up our secondary instance of tomcat with port other than 80, say 8080 and do the port redirection from 80 to 8080. To do such a redirection when the communication is not restricted to the local machine, we cannot use the "OUTPUT" chain. Instead we need to use the "PREROUTING" chain and an example of doing such a redirection is given below.
iptables -A PREROUTING -t nat -p tcp --dport 80 -j REDIRECT --to-port 8080
Let us thank Mehar, my colleague who provided us the above syntax.
If you are using the Tomcat server alone, listening to HTTP port 80 without having Apache web server which is also a common usage then you just need to do only one redirection command provided above. But remember, in this case we are using "PREROUTING" chain.
That's it. If the redirection rules are added using "iptables -A" option then the secondary instance becomes active and handles all the requests. You delete the redirection rules using "iptables -D" option, then secondary instance becomes dummy and the primary tomcat instance starts handling all the requests. This way you can switch back and forth until you have a stable, properly working version of your application in the primary tomcat instance. Once you have that then you can delete the redirection rules and shut down the secondary instance.
As I have been mentioning, these scripts can be used to setup and run secondary instance of tomcat on any linux machine to do "Zero Downtime" deployment of new version of a tomcat web application.
Here is the complete source code of the linux bash script "tomcat-start-secondary.sh"
In all the previous scripts I had been mentioning about only 3 ports that tomcat uses by default which are 8080, 8009 and 8443. For secondary instance we increase their numbers by 100 and use the ports 8180, 8109 and 8553. But in the previous script file if you had noticed we also changed another port 8005 to 8105. This is called Shutdown port used by tomcat. Tomcat primary instance and secondary instance run as two separate processes. No two processes can use the same port. That is fundamental which we all know. That's why we need to change that port also for the secondary instance. But we should not be redirecting that port. Only the other three ports we would be redirecting. This is what I wanted to share about the shutdown port.
Before we start the secondary instance let us do some preliminary checks in order to ensure that everything is proper.
if ( ! $(exists $tomcatSecondaryInstanceDir/webapps/ROOT/*) ) then echo "Error: Did not find any valid file in the directory $tomcatSecondaryInstanceDir/webapps/ROOT/" echo "Run tomcat-copy-app-to-secondary.sh and try again" exit 1 fi if ( $(isProcessRunning "$tomcatSecondaryInstanceName") ) then echo "Error: Tomcat secondary instance $tomcatSecondaryInstanceName seems to be already running. Please check." exit 1 fi if ( $(isPortOpen 8109) ) then echo "Error: Port 8109 is open and is being used by some unknown process. Please check." exit 1 fi First we need to check that some application files exist in the secondary instance directory. If they don't it means that previous script has not been run to copy the application files from the primary instance.
Then we check whether secondary instance is already running. If it is already running, but still we are trying to again start the secondary instance then may be we are working hard staying late. Just to remind ourselves that we should be going home before causing any major damage during such a tired state, we are doing these checks :-)
Then we check whether the AJP port 8109 is available. Just to ensure that secondary instance does not have any trouble while coming up.
Please leave your opinion whether any more useful checks that can be made or have we covered ourselves enough?
To do these checks I have implemented few bash functions and it should be obvious from the source code that they are all implemented in "include.sh" script file. More on that file in my future post. Here is the source code snippet of the functions we use in our current script file.
# Usage: getFileOrDirCount Path-or-FileName # Description: bash -f or -e fail if wildcard is used in the name and it results in more than one entry # See Also: exists function getFileOrDirCount() { local count count=$(ls -1 $1 2> /dev/null | wc -l) echo $count } # Usage: exists Path-or-FileName # See Also: getFileOrDirCount function exists() { [ $(getFileOrDirCount "$1") -gt 0 ] } # Usage: getHTTPResponseStatus URL function getHTTPResponseStatus() { local result result=$(curl -s -o /dev/null -I -w "%{http_code}" "$1") echo $result } # Usage: getProcessPID processName function getProcessPID() { echo $(ps -ef | grep "$1" | head -1 | grep -v grep | awk '{print $2}') } # Usage: isProcessRunning processName function isProcessRunning() { [ ! -z "$(getProcessPID \"$1\")" ] } # Usage: isPortOpen portNumber function isPortOpen() { local result if [ ! -z "$1" ]; then result=$(netstat -ln --tcp | grep "$1\s.*\sLISTEN") fi [ ! -z "$result" ] } Now we are completely ready to start the secondary instance which we do by the following line.
service $tomcatSecondaryInstanceName start Then the script waits for some time to allow the secondary instance to come up.
echo "Waiting for $waitingTimeInSeconds seconds to allow the tomcat instance $tomcatSecondaryInstanceName to start" sleep $waitingTimeInSeconds echo "Trying to access the URL $urlToTest" status=$(getHTTPResponseStatus "$urlToTest") if ([ $status -ne 200 ]) then echo "Error: http://localhost:8180 failed with status $status" exit 1 fi We check whether the application has come up properly by using Linux command "curl" to retrieve a HTML web page. The web page could be just the home page of the application. But at hudku.com we actually try to do a business search by executing a search query. As this is the first search query, it causes our application to initialize its search engine, do database access and execute all the important logic so that the application gets warmed up. Then when we do the port redirection and make it available to our users, the application is completely ready to service the user's search requests at full speed.
If the web page retrieval check looks good then we do the port redirection using "iptables". We first remove any existing redirection just in case and then redirect all the three important ports, 8080 ==> 8180, 8009 ==> 8109 and 8443 ==> 8543.
# Remove the redirection in case if they exist iptables -D OUTPUT -t nat -p tcp -d 127.0.0.1 --dport 8080 -j REDIRECT --to-port 8180 2> /dev/null iptables -D OUTPUT -t nat -p tcp -d 127.0.0.1 --dport 8009 -j REDIRECT --to-port 8109 2> /dev/null iptables -D OUTPUT -t nat -p tcp -d 127.0.0.1 --dport 8443 -j REDIRECT --to-port 8543 2> /dev/null # Set up the internal redirection of primary instance tomcat ports to secondary instance iptables -A OUTPUT -t nat -p tcp -d 127.0.0.1 --dport 8080 -j REDIRECT --to-port 8180 iptables -A OUTPUT -t nat -p tcp -d 127.0.0.1 --dport 8009 -j REDIRECT --to-port 8109 iptables -A OUTPUT -t nat -p tcp -d 127.0.0.1 --dport 8443 -j REDIRECT --to-port 8543 The above code works and is tested. I am not an expert on "iptables" and so far I have hardly scratched its surface. iptables with its various options such as OUTPUT, PREROUTING, SNAT, DNAT, etc., is quite vast and you could read about it elsewhere.
Let me share with you the information on iptables what I know and what is applicable to our current discussion. As I am quite confident of that much, I am sharing it with you. Otherwise I would have been happy to just provide the set of working commands and keep quiet.
In our application hudku.com it is the Apache server that listens to the HTTP port 80, receives packets from the internet and delivers content back to the internet. But internally Apache talks to Tomcat running at port 8080 (AJP 8009). Tomcat does not communicate to the outside world. That's why we use the iptables "OUTPUT" chain as we are only dealing with locally generated packets.
When we used to run our website hudku.com on a dedicated server we did not have Apache web server. The primary instance of Tomcat itself was directly listening to HTTP port 80. That means Tomcat was interacting with the outside world. The packets were coming into the server from the Internet and going out of the machine to the outside world. Tomcat's communication was not limited to within the machine. With this setup we used to bring up our secondary instance of tomcat with port other than 80, say 8080 and do the port redirection from 80 to 8080. To do such a redirection when the communication is not restricted to the local machine, we cannot use the "OUTPUT" chain. Instead we need to use the "PREROUTING" chain and an example of doing such a redirection is given below.
iptables -A PREROUTING -t nat -p tcp --dport 80 -j REDIRECT --to-port 8080
Let us thank Mehar, my colleague who provided us the above syntax.
If you are using the Tomcat server alone, listening to HTTP port 80 without having Apache web server which is also a common usage then you just need to do only one redirection command provided above. But remember, in this case we are using "PREROUTING" chain.
That's it. If the redirection rules are added using "iptables -A" option then the secondary instance becomes active and handles all the requests. You delete the redirection rules using "iptables -D" option, then secondary instance becomes dummy and the primary tomcat instance starts handling all the requests. This way you can switch back and forth until you have a stable, properly working version of your application in the primary tomcat instance. Once you have that then you can delete the redirection rules and shut down the secondary instance.
As I have been mentioning, these scripts can be used to setup and run secondary instance of tomcat on any linux machine to do "Zero Downtime" deployment of new version of a tomcat web application.
Here is the complete source code of the linux bash script "tomcat-start-secondary.sh"
#!/bin/bash # Execute "export DEBUG=1" to debug this script. # Set value to 2 to debug this script and the scripts called within this script. # Set value to 3,4,5 and so on to increase the nesting level of the scripts to be debugged. [[ $DEBUG -gt 0 ]] && set -x; export DEBUG=$(($DEBUG - 1)) # # Start the secondary instance # waitingTimeInSeconds=10 tomcatPrimaryInstanceName="$TOMCAT_PRIMARY_INSTANCE_NAME" tomcatSecondaryInstanceName="$TOMCAT_SECONDARY_INSTANCE_NAME" tomcatPrimaryInstanceDir="/usr/share/$tomcatPrimaryInstanceName" tomcatSecondaryInstanceDir="/usr/share/$tomcatSecondaryInstanceName" urlToTest="http://localhost:8180/search/business_listings/Resorts%20in%20Goa" # include all the utility scripts source $ELASTICBEANSTALK_APP_SCRIPT_DIR/include/include.sh if ( ! $(exists $tomcatSecondaryInstanceDir/webapps/ROOT/*) ) then echo "Error: Did not find any valid file in the directory $tomcatSecondaryInstanceDir/webapps/ROOT/" echo "Run tomcat-copy-app-to-secondary.sh and try again" exit 1 fi if ( $(isProcessRunning "$tomcatSecondaryInstanceName") ) then echo "Error: Tomcat secondary instance $tomcatSecondaryInstanceName seems to be already running. Please check." exit 1 fi if ( $(isPortOpen 8109) ) then echo "Error: Port 8109 is open and is being used by some unknown process. Please check." exit 1 fi # Start the secondary instance service $tomcatSecondaryInstanceName start if ([ $? -ne 0 ]) then echo "Error: Failed to start the instance $tomcatSecondaryInstanceName" exit 1 fi echo "Waiting for $waitingTimeInSeconds seconds to allow the tomcat instance $tomcatSecondaryInstanceName to start" sleep $waitingTimeInSeconds echo "Trying to access the URL $urlToTest" status=$(getHTTPResponseStatus "$urlToTest") if ([ $status -ne 200 ]) then echo "Error: http://localhost:8180 failed with status $status" exit 1 fi # Remove the redirection in case if they exist iptables -D OUTPUT -t nat -p tcp -d 127.0.0.1 --dport 8080 -j REDIRECT --to-port 8180 2> /dev/null iptables -D OUTPUT -t nat -p tcp -d 127.0.0.1 --dport 8009 -j REDIRECT --to-port 8109 2> /dev/null iptables -D OUTPUT -t nat -p tcp -d 127.0.0.1 --dport 8443 -j REDIRECT --to-port 8543 2> /dev/null # Set up the internal redirection of primary instance tomcat ports to secondary instance iptables -A OUTPUT -t nat -p tcp -d 127.0.0.1 --dport 8080 -j REDIRECT --to-port 8180 iptables -A OUTPUT -t nat -p tcp -d 127.0.0.1 --dport 8009 -j REDIRECT --to-port 8109 iptables -A OUTPUT -t nat -p tcp -d 127.0.0.1 --dport 8443 -j REDIRECT --to-port 8543 echo "Secondary tomcat instance $tomcatSecondaryInstanceName started successfully"