What does Reservation id mean when an AMI instance is provisioned? And what is it’s use?

If you do not know what I’m talking about, see the following command ec2-run-instances (and it’s output) which is used to launch any number of instances of a specified Amazon AMI:

roshan@roshan:~/.ec2$ ec2-run-instances ami-cddf39a4

In the above output the RESERVATION ID is r-06acd26e.

At the meeting I attempted to answer that question by waving my arms in a very mumbo jumbo sort of way. Pretty much like how those marketing/sales guys you meet do (j/k)
I did not make sense at all. Therefore I decided to spend some time putting my thoughts down in this post.

Reservation
A reservation in the above context is an atomic launch of one or many instances. One reservation does not imply a single reserved instance. For example, if you launch 1 instance, you get 1 instance ID and 1 reservation ID. If you launch 2 instances in a single ec2-run-instances command, you get 2 instance IDs, but still 1 reservation ID. Likewise if you launch N instances in a single ec2-run-instances command, you still get only 1 reservation ID.

Use
Since a reservation is an atomic launch and the reservation id serves as an identifier to a group of instances, you could use it, for instance, to terminate the entire reservation of instances. The attached Scala script does just that! It takes a reservation id as a parameter, gets the list of instances in that reservation and then terminates those instances. This script saves you the trouble of terminating each instance manually.

Let’s see this script (using reservation id) in action. Let’s start 5 instances of an AMI.

roshan@roshan:~/.ec2$ ec2-run-instances ami-cddf39a4 -n 5

Now, to delete all instances under reservation id r-72feba1a, all I have to do is the following:

roshan@roshan:~/.ec2$ ec2-terminate-reservation r-72feba1a

As you can see from the above output of ec2-terminate-reservation, the instances are now in the shutting-down state

ec2-describe-instances confirms that the instances are being shutdown

roshan@roshan:~/.ec2$ ec2-describe-instances

How to use ec2-terminate-reservation
Download the attached ec2-terminate-reservation.sh script to your $EC2_HOME/bin directory, rename it to ec2-terminate-reservation and make it executable.

roshan@roshan:~/.ec2$ chmod +x ec2-terminate-reservation

You should have Scala installed and the $SCALA_HOME/bin appended to your PATH for the script to work.

ec2-terminate-reservation

But before you jump into virtualization, how can you find out if your existing infrastructure/machines support it?

In Linux, you can do this by checking for the following flags in the /proc/cpuinfo file:

• vmx (for intel)
• svm (for amd)

Run the following command:

egrep '(vmx|svm)' --color=always /proc/cpuinfo

If your processors supports virtualization, you will see an output like the following:

Since I have 2 processors, the above output shows 2 “flags” section.

Note the above command only indicates that your processors support virtualization. You will need to make sure that it is not disabled in your machine’s BIOS.

The three features I really like about PuTTY-CM are:

• Ability to re-size and arrange windows (SSH sessions to multiple Linux machines) within the parent window
• Ability to execute a command across multiple active SSH sessions at the same time

The ability to execute the same command on multiple SSH sessions is HUGE for me given that I am generally working with either VMs or multiple physical machines at any given point of time.

I have googled quite a bit to find an equivalent of PuTTY-CM for the Mac. And the closest I have come is iTerm.
I say closest, since iTerm does not have the feature of re-sizing/re-arranging the SSH sessions (tabs) within a parent window.

But hey, the ability to “Send Input to all tabs” (Shift + Command + I) within a window is still HUGE and an excellent feature over the Terminal app that comes bundled with the Mac OS X.

I generally categorize my projects into multiple Eclipse workspaces and at times like having them open at the same time. In Windows/Linux OSes, this is a no-brainer – just click on the shortcut and this will start up a new process. But in Mac OS X, clicking on the Eclipse launcher in the taskbar just brought the existing Eclipse process into the foreground.

Finally I found out that I could accomplish what I wanted with the open command. Add it to a script as shown below and bingo, starting up multiple Eclipse workspaces is a piece of cake.

#!/bin/bash

ECLIPSE_HOME=~/tools/editors/eclipse35-JEE-x86_64/eclipse
open -n $ECLIPSE_HOME/Eclipse.App

Since Ubuntu 6.10, the default system shell, /bin/sh, was changed to dash. This change will cause /bin/sh scripts to fail if bash extensions exist.

There is a tool called checkbashisms that is shipped along with the “devscripts” package, which is able to find several kind of bashisms in /bin/sh scripts and makefiles.

Install checkbashisms
First, check if checkbashisms is installed using the following command:

$ aptitude show devscripts

Since in our case it is not installed, we will install checkbashisms using the following command:

$ sudo aptitude -y install devscripts

Using checkbashisms
Once you have the devscripts package installed, you can check for bashisms in scripts using the the following command:
$ checkbashisms [-n] [-f] [-x] script_filename
For example, to check for bashisms in the start-up script for Appistry CloudIQ cloud platform, use the following command:

$ checkbashisms -f /etc/init.d/fabric_keeper

The above command returns the following output:

Checkbashisms provides a good starting point to find bash extensions in /bin/sh scripts. Using this tool, you should be well on your way to getting rid of these bash extensions and towards creating portable scripts.

To learn more about bashisms see https://wiki.ubuntu.com/DashAsBinSh

Note: These instructions should work with other Debian flavors as well.

Download Appistry-CloudIQ
Download the rpm package for Appistry-CloudIQ from the Community site. The free Community edition is great to start building a small enterprise cloud on-premise. For the purpose of this post, I’m Appistry-CloudIQ Platform 4.2 Community edition for RHEL 5.x, x86 32-bit version.

After you have downloaded, you should see something like this:

Create .deb package
Once you have the package downloaded, use alien to convert the rpm to a .deb package using the commands below. The package alien allows you to convert rpm packages to debian packages and then install them via dpkg. See how-to install alienhow-to.

Convert the rpm package to the debian package using the following command:

roshan@ubuntuWorkerB:~$ sudo alien -kc appistry-cloudiq-4.2.1.2-rhel5.rpm

Install the .deb pacakge
Now that you have the .deb package, install it using dpkg.

roshan@ubuntuWorkerB:~$ sudo dpkg -i appistry-cloudiq_4.2.1.2-rhel5_i386.deb

The install fails to create the user “fabricuser”. So let’s create it and add it to group “fabricuser”.

roshan@ubuntuWorkerB:~$ sudo useradd -s /bin/bash -g fabricuser fabricuser

You can confirm that the user was created by running the following command:

roshan@ubuntuWorkerB:~$ cat /etc/passwd | grep fabricuser

Change ownership of /usr/local/appistry/cloudiq to fabricuser
Change ownership of the /usr/local/appistry/cloudiq directory recursively to fabricuser using the following command:

roshan@ubuntuWorkerB:~$ sudo chown -R fabricuser:fabricuser /usr/local/appistry/cloudiq

Update scripts and create /bin/arch script
First, update the start-up scripts – fabric_keeper and fabric_system_service to explicitly use the /bin/bash interpreter. These two scripts are located under /etc/init.d directory.

roshan@ubuntuWorkerB:~$ sudo sed -ie 's/\/bin\/sh/\/bin\/bash/g' /etc/init.d/fabric_system_service
roshan@ubuntuWorkerB:~$ sudo sed -ie 's/\/bin\/sh/\/bin\/bash/g' /etc/init.d/fabric_keeper

The reason for doing this is because of the bashisms that exist in the two scripts.

Next, we need to create an executable script arch under the /bin directory.

roshan@ubuntuWorkerB:~$ echo "uname -m" > arch
roshan@ubuntuWorkerB:~$ sudo mv arch /bin/.; sudo chmod +x /bin/arch

The reason for doing this is because the start-up scripts source in /etc/fabric_env references /bin/arch which does not exist in Ubuntu since version 7.10. This causes the start-up scripts to fail. See https://bugs.launchpad.net/ubuntu/+source/util-linux/+bug/148511 for more information.

Set up symlinks for libssl.so.6 and libcrypto.so.6 shared libraries
Since the start up scripts attempt to load the libssl.so.6 and libcrypto.so.6 shared libraries, create symlinks with these names under /usr/lib directory.

roshan@ubuntuWorkerB:/usr/lib$ sudo ln -s libssl.so.0.9.8 libssl.so.6
roshan@ubuntuWorkerB:/usr/lib$ sudo ln -s libcrypto.so.0.9.8 libcrypto.so.6

If you do not create the above symlinks you will get errors like:

and

Reboot to start Appistry CloudIQ
Reboot Ubuntu. You should see the following messages when Ubuntu is restarting.

Also, start log_monitor. You should see heartbeat messages.

roshan@ubuntuWorkerB:~$ . /etc/fabric_env
roshan@ubuntuWorkerB:~$ log_monitor 239.255.0.1

You now have the world’s best cloud platform – Appistry CloudIQ – running on Ubuntu.

In future posts, we will see how to create/migrate/manage infrastructure and applications to the “cloud” using Appistry CloudIQ

Downloads
Download the following script and place it alongside the appistry-cloudiq-X.x.rpm that you downloaded from the Appistry Community website.

cloudiq-ubuntu-install.sh

Next, run the script as follows:

roshan@ubuntuWorkerB:~$ sudo sh cloudiq-ubuntu-install.sh appistry-cloudiq-4.2.1.2-rhel5.rpm

where, appistry-cloudiq-4.2.1.2-rhel5.rpm is the version that I downloaded.

Install alien
First check if alien is installed

roshan@ubuntuWorkerB:~$ alien

Since alien is not installed, use the following command to install it

roshan@ubuntuWorkerB:~$ sudo aptitude -y install alien

roshan@ubuntuWorkerB:~$ alien

Using alien

If you have an rpm package, you can convert it to a debian (deb) package using the following command:

roshan@ubuntuWorkerB:~$ sudo alien -kc appistry-cloudiq-4.2.1.2-rhel5.rpm

All Windows OSes with the exception of Windows 95 (I think) come with WSH pre-installed. And if you need it for Windows 95, you could download it from the Microsoft site.

To display the version of WSH installed on Windows, type cscript at the command prompt and then press ENTER. It should display an output like the following:

C:\Documents and Settings\xxxx>cscript
Microsoft (R) Windows Script Host Version 5.7
Copyright (C) Microsoft Corporation. All rights reserved.

Usage: CScript scriptname.extension [option...] [arguments...]

Options:
//B Batch mode: Suppresses script errors and prompts from displaying
//D Enable Active Debugging
…
…

To read more about WSH: http://www.microsoft.com/technet/scriptcenter/guide/sas_wsh_overview.mspx

Enough talk. Let’s see an example script that sets an environment variable (say, ”environSet.vbs”):

'Set JAVA_HOME, JRE_HOME and update PATH
set shellObj = WScript.CreateObject("WScript.Shell")
set envObj = shellObj.Environment("System")
envObj("JAVA_HOME")="C:\tools\jdk\jdk1.6.0_10"
envObj("JRE_HOME")="C:\tools\jdk\jdk1.6.0_10\jre"
envObj("PATH")=envObj("PATH") & ";%JAVA_HOME%\bin;%JAVA_HOME%\jre\bin\client"

To run the above script, you can open a command shell, type cscript environSet.vbs (or just environSet.vbs) and press ENTER or just double-click the environSet.vbs file.

In the above example, the JAVA_HOME and JRE_HOME system environment variables are set to “c:\tools\jdk\jdk1.6.0_10″ and “c:\tools\jdk\jdk1.6.0_10\jre” respectively. The PATH system environment variable is updated and appended with “%JAVA_HOME%\bin;%JAVA_HOME%\jre\bin\client”.

Notes:

• If the variables do not exist, they are created. If they already exist, they will be over-written unless you take care to append them as shown in the above example (see PATH)
• When you run the above script, the updates are also reflected in the registry. In the above case since we have set the System environment variables you could browse to HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\Environment\ in regedit to see the system environment variables.
• Some pre-defined system variables such as PATH may not be picked by Windows services and will require a Windows Reboot.

Pete Koomen, the product manager for Google App Engine showed interested in the CloudIQ Engine piece of CloudIQ especially given that Goople App Engine is one of the key players in the Cloud Platform area – Hosted Cloud Platforms or PaaS, to be precise.
Some folks were interested in whether CloudIQ supports scaling out .NET components. While some asked if C, C++ code was supported. Yes and Yes.

I’m glad I used Sun’s VirtualBox to provision VMs installed with only CloudIQ and the sample application for the demo. I call it the cloud-in-a-box. Also it doesn’t hurt when VirtualBox is part of Sun Cloud.
Anyway, one of the slides in the presentation was that

CloudIQ complements Virtualization

And here is how you can do this with VirtualBox:

1. Use VirtualBox (or Sun Cloud when it is released this summer) to provision VMs with an OS and CloudIQ installed
2. Use CloudIQ Manager to provision and manage applications and middleware such as Tomcat, GlassFish, JBoss, etc.
3. And once you are comfortable with this setup, deploy (”touchless” or otherwise) your compute intensive code to CloudIQ Engine to achieve linear scalability and reliability at the component level

I will blog about my experience with VirtualBox and running Appistry CloudIQ platform on Ubuntu VMs soon. And when Sun’s cloud offering is released, I will tweak it to use Sun’s Cloud APIs. But if you cannot wait until then, sign up for the Appistry CloudIQ 4.0 beta program or download the community edition of Appistry CloudIQ from here and start creating your very own cloud-in-a-box now!

Back to the session, I do wish I had time for Q&A. My boss had warned me about that when he gave me feedback about my presentation slides. What I’m thinking is that people must have been hungry since my session was just before lunch-break and the guy waving the “time-remaining” sign pitied the crowd and decided to cut my session short by 5 minutes. Kidding!

Anyway great crowd, good times.

I attended the Birds-of-a-feather talk hosted by Tim Bray, Craig McClanahan and Lew Tucker where they presented the APIs and answered questions. Both Tim and Craig and their team have done an excellent job of coming up with these APIs that adhere so well to the REST principles. Kudos! Tim has blogged about the Sun Cloud APIs here. And so has Craig here.

During the session, one of the questions was

how do you upgrade middleware or applications or framework libraries (say, Hiberate) used by applications that are deployed in the cloud?

Answer: Use Cloud Application Platforms such as Appistry’s CloudIQ.

In my humble opinion, it would be great if the folks at Project Kenai designed the APIs with extensibility in mind to support Cloud Application Platform APIs such as Appistry’s CloudIQ APIs (which are also REST based) to deploy, install, update, uninstall applications and middleware. I say this because infrastructure level APIs are limiting.

With infrastructure level APIs you could create a VM, clone it with application servers/applications installed and go live. But what happens when you want to deploy a new version of say, GlassFish (perhaps with some critical updates) to your existing production cloud? Now suddenly you are back to home-grown scripting or creating additional VMs with the new updates. Now take it to the next level where you have to create and manage these scripts for various OS’s. Ewwwwww!

The solution is to use CloudIQ which abstracts out the virtual machines. Using CloudIQ you could easily deploy the new version of GlassFish to your existing production cloud without resorting to crazy, complex scripts. Save the time from creating additional VMs for every update and spend that time focusing on your business!

Another great example to drive home the importance of CloudIQ in the Cloud Computing arena:
Assume you are using Hibernate 2.x in a couple of your several web applications running on Tomcat in your production cloud. And now you want to upgrade to the 3.x version of Hibernate for only one of your web applications. So how would you do this easily using infrastructure?

1. Clone a production VM
2. Replace Hibernate 2.x with 3.x jars in the target web application
3. Clone that VM
4. Deploy the new VMs to the production cloud and delete the existing VMs

Ewwwwwwww!

Using CloudIQ, you could do this in half the steps and so fast that it will make your VMs head spin.

1. Replace Hibernate 2.x with 3.x jars in the target web application
2. Deploy the web application using CloudIQ Manager.

Sweeeeeeeet!

The point I’m trying to make is that CloudIQ virtualizes virtualization It gives you more fine-grained control or application level control.

Therefore it would be great to have Project Kenai allow for extensibility to support popular Cloud Application Platforms APIs.

What do you think?