Thanks to the typical demo demons, I’ve been unable to get everything functioning perfect for instantaneous clustering on EC2 by tomorrow.
Oh well.
But here’s the first draft of some slides I’m taking with me to the Raleigh RubyCamp.
It’s a BarCamp style event, so I anticipate the slides probably changing throughout the day. I’ll published updates if needed.
Trying to debug why my init scripts weren’t initing on EC2, I checked all the normal places.
The chkconfig tool reported that my services should be on at runlevel 3:
jboss 0:off 1:off 2:off 3:on 4:off 5:on 6:off
My /etc/inittab says I was booting to runlevel 3:
id:3:initdefault:
Yet the console did not show my things to be spinning. A little googling around, I found the runlevel command (I’m not a linux expert…) which explained to me I was in runlevel 4.
N 4
4? Really? 4? I wasn’t expecting that.
Apparently this is just how EC2/Xen work. They force you up into runlevel 4.
What does 4 mean? Asking the Wikipedia, you’ll see a pattern:
I had no idea that runlevels were all over the map. I also had no idea that EC2 would disregard my inittab and make up their own ideas.
Thanks, guys. Another hour-long round of RPMing, kickstarting, and image-creation ahead of me…
On my path towards clustering a Rails app on JBoss on EC2, I stumbled across Bryan Kearney and the other Thincrust guys. With their help, I now have a JBoss AS5 + jboss-rails “appliance” ready to roll.
Grab the raw image or the VMWare image, and play along at home.
Fire up the image in your favorite virtualization environment. I give my virtual machine at least a gig of RAM. Marvel at the pretty Grub splash screen, courtesy of James Cobb (JBoss.org designer).
Let it boot on up, and you’ll notice a handful of things:
You can login with root password of thincrust.
The login prompt will tell you the IP address of the appliance, since it probably booted off DHCP. JBoss will be up and running at http://<IP_ADDRESS>:8080/.
You can su jboss, whose $HOME is /opt/jboss/jboss-as5, which coincidentally is $JBOSS_HOME. The default configuration is used to start the AS. Logs are under /opt/jboss/jboss-as5/server/default/log/. And to deploy, just drop something into /opt/jboss/jboss-as5/server/default/deploy/ and it’ll hot-deploy.
To control the service, as root:
service jboss stopservice jboss startservice jboss statusI’ll make the RPMs used to build this available sometime soon. Until then, you can poke around the bits I use to create the RPMs ultimately used by Thincrust to build the appliance image. They are packaged in a way that makes my Red Hat brethren throw up in their mouths a little bit.
Also, once I test’em on EC2, I’ll throw out public AMIs for testing.
By no means is this complete. This just marked a nice spike of a milestone along the way. There’s still plenty of things that’ll poke you in the eye if you’re not careful. Always wear your safety harness. Drink plenty of fluids. Keep away from children.
It can sometimes be funny, the paths we take.
Once I’d verified that a Rails app deployed on JBoss would indeed cluster, I sat in my farmhouse, looking at my lone little Mac. Not much of a cluster to play with.
So I started looking at Amazon EC2, which is truly very nice, particularly when paired with Elasticfox.
Of course, firing up a cluster on EC2 requires a nicely-produced, ready-to-boot machine image or a lot of manual configuration on each node.
Sacha pointed me to another group within Red Hat: Thincrust.
Thincrust configures a Fedora disk image with “just enough” OS bits, and provides a way to add/update applications on it. It’s a happy mixture of Kickstart, Puppet and Yum.
The goal is to make it easy to produce “appliance” images that could be flung onto machines, real or virtual. Need a Git server? Fire up the Git appliance on the cloud. Need a Drupal server? Fire one of those up. Need a JBoss server or cluster? We’re working on that..
Thincrust is built upon Fedora. My Mac is not. But VMWare Fusion lets me run Fedora. So I can create Fedora-based Thincrust images. Which I’ll deploy on EC2. Which is running Xen, probably on RHEL. And ultimately allowing you to run your Ruby-on-Rails applications on a Java stack.
There are times I can’t remember exactly which OS, language, or virtualization environment I should be thinking in.
Elasticfox is a nice extension for managing your relationship with EC2 from within Firefox. It’s XUL-based, somewhat homely and odd. But it’s so much nicer than dealing with the ec2-* scripts directly and copying Amazon’s wacky hash identifiers hither and thither.
But on OSX, the “ssh to this instance” button is wired up to Terminal.app. I can’t stand Terminal.app. I much prefer iTerm.
So, a little AppleScript hacking, and Elasticfox now uses iTerm to connect to instances through ssh.
Instead of jamming it all into a dozen -e options on osascript, I just put it in a file, and changed my configuration.
And the script itself:
on run argv
tell application "iTerm"
activate
set cmd to "ssh -i " & item 1 of argv & " " & item 2 of argv
set myterm to the last terminal
tell myterm
launch session "Default Session"
tell the last session
write text the cmd
end tell
end tell
end tell
end run
Don’t forget, in a few weeks at the Red Hat offices in Raleigh, there will be an unconference-style Ruby BarCamp on October 18th.
My plan is to put together a few clumps of slides, prepare a cluster on EC2, and see if anyone wants to hear about JBoss Rails. I’ll of course put them online sometimes before/during/after the camp, since that’s part of the rules.
Rysiek asked me to elucidate my clustering on OSX…
By default, it seems that OSX provides exactly 1 localhost address, unlike RHEL. I normally make use of 127.0.0.1/24 when I’m deploying on Linux, and figured I’d do the same on OSX.
To accomplish this, you need to create some new localhost IPs to play with.
sudo ifconfig lo0 alias 127.0.0.10 up
sudo ifconfig lo0 alias 127.0.0.11 up
sudo ifconfig lo0 alias 127.0.0.12 up
Now, when you run your AS with ./run.sh, just pass -b 127.0.0.10 or -b 127.0.0.11 etc, and the entire stack will bind to that IP address.
At this point, I’m allowing each AS instance to share a work directory and such, which is probably not exactly the safest thing to be doing. Normally each cluster node would be a discrete machine with its own JBOSS_HOME.
For the apache config, in /private/etc/apache2/httpd.conf, I uncommented the line to allow vhost file loading:
# Virtual hosts
Include /private/etc/apache2/extra/httpd-vhosts.conf
Then, in the httpd-vhosts.conf file I fixed it up to include my virtual host for the head-end load-balancing stuff.
The <VirtualHost> is only barely conformant, and even the DocumentRoot is gratuitous, really. It looks like this:
<VirtualHost *:80>
ServerName app.local.ballast
DocumentRoot "/Users/bob/public_html"
RewriteEngine on
RewriteMap lb 'prg:/Users/bob/bin/load_balancer local.ballast 1 3'
RewriteRule ^/(.*)$ ${lb:$1} [P,L]
</VirtualHost>
What that is doing is passing each requested path to the load_balancer script I’ve borrowed from I-don’t-know-’where. It returns an augmented URL pointing to one of my localhost-bound nodes.
#!/usr/bin/env perl
##
## lb.pl -- load balancing script
##
$| = 1;
$name = $ARGV[0]; # the hostname base
$first = $ARGV[1]; # the first server (not 0 here, because 0 is myself)
$last = $ARGV[2]; # the last server in the round-robin
$cnt = 0;
while (<STDIN>) {
$server = sprintf("node%d.%s", $cnt+$first, $name);
$cnt = (($cnt+1) % ($last+1-$first));
print "http://$server:8080/$_";
}
If a path of /foo/bar is handed to it on STDIN, it’ll return something in the format of http://node1.local.ballast:8080/foo/bar or http://node2.local.ballast:8080/foo/bar.
Through the magic of /etc/hosts those friendly names point to the localhost bound AS cluster nodes, along with app.local.ballast pointing to good old traditional 127.0.0.1, where httpd is normally listening. That’s my head-end from the httpd-vhosts.conf.
127.0.0.1 localhost app.local.ballast
127.0.0.10 node1.local.ballast
127.0.0.11 node2.local.ballast
127.0.0.12 node3.local.ballast
Fire up a browser and I surf to http://app.local.ballast/, apache answers, and immediately and invisibly proxies the request to port 8080 of one of my localhost-bound JBoss AS instances.
Nice.
JBoss on Rails will indeed cluster!
After modifying and dropping my jboss-rails.deployer into an ‘all’ configured server of JBoss AS 5, and firing up 3 instances on my localhost (non-trivial on OSX…):
10:43:28,409 INFO [RPCManagerImpl] Received new cluster view: [127.0.0.10:63740|2] [127.0.0.10:63740, 127.0.0.11:63747, 127.0.0.12:63749]
10:43:28,435 INFO [RPCManagerImpl] Cache local address is 127.0.0.12:63749
10:43:28,469 INFO [ComponentRegistry] JBoss Cache version: JBossCache 'Poblano' 2.2.0.GA
And I’ve got 3 nodes running the same Rails app, all sharing a cookie and a JBossCache cache. Nick Sieger’s JRuby-Rack handles binding the Rails session to the actual servlet session, and JBossCache takes care of the rest.
A little 8-line perl round-robinning load-balancer is wired up through mod_rewrite in my Apache httpd.conf to throw requests to each of the nodes. Anything set in the session is immediately available at the next request which lands at a different node.
Further down the line, we can look at a clustered cache for caching AR models and view fragments. Not too shabby.
It should be fairly easy to create a nice Amazon EC2 AMI with Fedora+AS5+jboss-rails, plus some better Rake/capistrano tasks, and make for quick cluster deployment. Any EC2 experts wanting to jump in?
Since JBoss AS 5 is built on top of Microcontainer, it’s effectively a network of beans just doing their respective jobs. You have already probably noticed that it ships with 3 included configurations: minimal, default, and all.
Unfortunately, they’re awefully far apart along the spectrum of configuration options. The minimal configuration barely gets the container running, while the all configuration includes, well, everything. For most apps, it’s safe to start with the default configuration. But default to who? It’s the 80% case. But the 80% of the world who gets by with default, it still probably includes way more than they need. Of course, each user needs a different subset of that 80%.
Given that the MC is managing a graph, it seems like we should be able to actually perform a solve to determine what is or is not required. Or at least get a good idea. Bonus points if someone can then twiddle conf/, deploy/ and deployers/ to tidy things up.
Anyhow, for the jboss-as-rails project, I’m kicking it old-school, and going with default. I’m jamming it into git, and hopefully with some help, we can get it pared down to what we need.
The jboss-as-rails project is simply a configuration of AS that includes only what we need, along with the jboss-rails.deployer from the jboss-rails project (notice the subtle naming difference…).
Here’s a picture to see how it’ll all ultimately fit together.
The plugin will offer rake tasks for managing the included AS, deploying your app. I’ll also produce an AS-free version of the plugin, assuming you want to manage your own AS separately.
And poke around jboss-as-rails, see what we can rip out.
Tomorrow is my first real status update call with my boss, Sacha Labourey. I’ve been anxious to deliver something, to prove I hadn’t gone completely pudding-brained during my tenure as management.
This morning, it all finally came together in a pleasing fashion, causing me to hoot and holler loud enough to scare the cats and probably some cows.
I’ve just pushed an ugly-but-working deployer targeting JBoss-AS 5.0.0.CR2 (the latest and greatest!)
It’s not very consumable at this point, as it’s just a deployer, not a nice Rails plugin with a set of Rake tasks. Heck, it doesn’t even undeploy yet.
But adding the deployer to your server’s deployers/ directory allows you symlink live RAILS_ROOTs into your deploy/ directory, and be running on JBoss.
Live. In-situ. Edit your controllers or views as you like, and your changes are immediately reflected in the running instance. Just like with ./script/server. It does not even have to redeploy your app. The rails framework is handling the magic reloading.
It’s taken me some time to dig through the innards of JBoss-Microcontainer, and a few false starts, but I finally figured out a super simple deployment process.
I’d previously been trying to manipulate a RAILS_ROOT into a synthetic Java WAR archive, and shoe-horn things around that. But I have the freedom to go lower than that, so the jboss-rails deployer just sets up a Catalina context appropriately, without regard to WEB-INF or other non-Rails stuff. There’s no need for that cruft. Likewise, I can directly control and manipulate the classpath, so the RAILS_ROOT does not even have to have any JRuby bits in it.
The example application (src/test/ballast) is a virgin rails app with ActiveRecord disabled so I don’t have to deal with database-driver gems just yet.
Once deployed, a Rails app looks like pretty much any other web-app. The jboss.rails.deployment domain contains deployment objects for each rails app. And jboss.web contains all the webby bits floating around.
I need to go back and remove the dead-end code I’ve left in my wake, and update the tests I’d disabled while in a coding flury (bad Bob!) I plan to put together an easy-to-consume plugin gem which contains an nicely-configured AS along with the jboss-rails deployer pre-installed, along with rake tasks to start/stop AS, and deploy your app. I’d also like to give clustering a whirl, and see what we can do.
It’s been an excellent 3 weeks back as an engineer.