I signed up for a new AWS account and ran their sample LogAnalyzer Job Flow using the AWS console. That was easy enough. Next I attempted to run the same sample from the command line using the Amazon Elastic MapReduce Ruby Client.

Note: The Ruby Client README turns out to be very helpful.

Next I downloaded the source and looked at. Seems simple enough. I notice that this sample uses a library called Cascading, which appears to be a way to simplify common job flow tasks.

After adding the elastic-mapreduce app to my path and setting up my credentials file, I ran:

elastic-mapreduce –create –jar  s3n://elasticmapreduce/samples/cloudfront/logprocessor.jar –args  “-input,s3n://elasticmapreduce/samples/cloudfront/input,-output,s3n://folksemantic.com/cloudfront/log-reports,-start,any,-end,2010-09-07-21,-timeBucket,300,-overallVolumeReport”

It produced:

INFO Exception Retriable invalid response returned from RunJobFlow: {”Error”=>{”Details”=>#<SocketError: getaddrinfo: nodename nor servname provided, or not known>, “Code”=>”InternalFailure”, “Type”=>”Sender”}} while calling RunJobFlow on Amazon::Coral::ElasticMapReduceClient, retrying in 3.0 seconds.

After some poking around, I realized that I specified “west-1″ as my region when it should have been “us-west-1″. This resulted in the client trying to contact a non-existent server I’m guessing.

So now, my jobs started, but failed immediately. I logged into the AWS console and clicked on one of the failed job flows to see the reason for the failure (Last State Change Reason):

The given SSH key name was invalid

Googling found: http://developer.amazonwebservices.com/connect/message.jspa?messageID=166768

Which at first confused me, then I went ahead and followed the link (while logged in) and did what it said to. (Amazing how that works sometimes ) It prompted me to create a new key and to assign it a name.

After I had generated the key and put its name in the credentials.json, things worked like a charm. It turns out that if you run a job from scratch, it has to fire up an EC2 instance in order to run the job, and that can take a while. To avoid that start up time, you can run:

elastic-mapreduce –create –alive –log-uri s3://my-example-bucket/logs

As mentioned in the README.TXT

My next steps are to:

1. Modify the job flow and run that job flow.
2. Run the job flow locally.
3. Debug the MapReduce portion of the job flow.

http://mydomain.com:8080/lms/index.jsp

Some firewalls block port 8080 so I wanted my site to be available on port 80 so that it uses urls like:

http://mydomain.com/lms/index.jsp.

One option was to modify the Tomcat configuration to listen on port 80. However, I already have Apache installed and listening on port 80 (to serve other content) so I couldn’t that. Instead I configured Apache to route requests for my web application to Tomcat. I’ve been through this process a number of times before, and it never seems to go smoothly, so I document it here. I am running Apache 2.2 and Tomcat 6 on 32 bit Ubuntu Linux.

Overview

The steps are:

1. Download the Apache jk connector module (mod_jk.so).
2. Create Apache module configuration files for the jk connector (jk.load and jk.conf) and enable the module.
3. Create a worker.properties file to configure the Tomcat worker for the connector.
4. Define an AJP connector in your Tomcat configuration file (server.xml)
5. Assign urls to Tomcat in your Apache virtual hosts file.

Download the Apache jk connector module (mod_jk.so)

Apache uses the jk connector module to talk to Tomcat. I downloaded it from a subdirectory of http://www.apache.org/dist/tomcat/tomcat-connectors/jk/binaries/. I wasn’t sure which OS I was running and whether or not I was running a 32 bit version (i586 directory) or a 64 bit version (x86_64). To find this out, I ran:

file /usr/bin/file
/usr/bin/file: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.15, stripped

So I downloaded: http://www.apache.org/dist/tomcat/tomcat-connectors/jk/binaries/linux/jk-1.2.28/i586/mod_jk-1.2.28-httpd-2.2.X.so

I chose that version because I am running Apache 2.2. There are different versions for different versions of Apache.

I put the file in my Apache modules directory (/usr/lib/apache2/modules/) and renamed it to mod_jk.so.

Create Apache module configuration files for the jk connector (jk.load and jk.conf) and enable the module

In order to get Apache to load and configure the jk connector module, I created jk.load and jk.conf files (in /etc/apache2/mods-available/) and then enabled them. jk.load just tells Apache where to find the module:

LoadModule jk_module /usr/lib/apache2/modules/mod_jk.so

jk.conf ties everything together by configuring the jk connector module:

# Where to find workers.properties
# Update this path to match your conf directory location (put workers.properties next to httpd.conf)
JkWorkersFile /etc/apache2/workers.properties

# Where to put jk shared memory
# Update this path to match your local state directory or logs directory
JkShmFile     /var/log/apache2/mod_jk.shm

# Where to put jk logs
# Update this path to match your logs directory location (put mod_jk.log next to access_log)
JkLogFile     /var/log/apache2/mod_jk.log

# Set the jk log level [debug/error/info]
JkLogLevel    info

# Select the timestamp log format
JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "

For more information about the Apache jk connector module configuration, see the Tomcat Connector – Apache Webserver HowTo.

Initially I set the JkLogLevel to debug, so I could see any error messages, but then changed it to info once I had everything working.

After creating the files, I enabled the module using:

sudo a2enmod jk

That creates symlinks to the jk.load and jk.conf files in the mods-enabled directory where my Apache is configured to look for modules to load.

Define an AJP connector in your Tomcat configuration file (server.xml)

AJP is an efficient protocol that Apache and Tomcat can be configured to use to talk to each other. I set up an AJP connector in my Tomcat configuration file (/etc/tomcat6/server.xml). The default configuration file has the connector defined but commented out, so I uncommented it:

<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />

For details see the Tomcat AJP Connector documentation.

Create a worker.properties file to configure the Tomcat ajp worker for the connector

“A Tomcat worker is a Tomcat instance that is waiting to execute servlets or any other content on behalf of some web server”.

Note: this quote from the documentation is a bit curious, because, nowhere in the Tomcat configuration files do I tell Tomcat about the worker. I think that the worker is actually a process that the jk connector spawns.

I configured a worker to listen to Apache requests by creating a worker.properties file in the same directory as the Apache configuration file (/etc/apache2/workers.properties).

# Define 1 real worker using ajp13
worker.list=worker1

# Set properties for worker1 (ajp13)
worker.worker1.type=ajp13
worker.worker1.host=localhost
worker.worker1.port=8009

The jk connector knows how to talk to this worker, because the file name is specified in the Apache jk connector configuration file (/etc/apache2/mods-available/jk.conf). For more information see the Tomcat Connector Quick Start or the Tomcat Connector Reference Guide.

Assign urls to Tomcat in your Apache virtual hosts file

After I configured a Tomcat worker to listen to AJP requests and configured Apache to use the jk connector module to talk to that worker, the last thing that was needed was to configure my web site’s virtual host (/etc/apache2/sites-available/default) to route urls to Tomcat:

<VirtualHost *:80>
...
        JkMount /lms/* worker1
...
</VirtualHost>

Note that worker1 is the name I gave to the worker I set up in the workers.properties file. Note that by using a * mask, I routed all requests (including static files) through Tomcat. Alternatively I could have configured only jsp requests to be routed to Tomcat, using:

<VirtualHost *:80>
...
        JkMount /lms/*.jsp worker1
...
</VirtualHost>

Restart Tomcat and Apache

sudo /etc/init.d/tomcat6 restart
sudo /etc/init.d/apache2 restart

Lifecycle

To the best of my understanding, the relevant lifecycle is:

1. When Tomcat starts up, it begins listening for AJP requests on port 8009 (because the connector is defined in /etc/tomcat6/server.xml).
2. When Apache starts up, it loads the jk connector module (because it is defined in /etc/apache2/mods-enabled/jk.load).
3. When Apache loads the jk connector, its configuration file (/etc/apache2/mods-enabled/jk.conf) tells it to send requests to the specified Tomcat worker and to use shared memory to do that.
4. It is not clear to me whether or not the Tomcat worker gets spawned when Apache starts up or on each request. I don’t see how it could get spawned when Tomcat starts up since Tomcat has no way of knowing about it.
5. Apache receives a request for a url that is mapped to Tomcat (in the virtual host file – /etc/apache2/sites-enabled/default).
6. Apache uses the jk connector module (mod_jk.so) to generate a request to send to Tomcat via a Tomcat worker.
7. The Tomcat worker communicates with Tomcat using the protocol (AJP) and port (8009) defined in the workers configuration file (/etc/apache2/workers.properties).
8. Tomcat processes the request and returns the response back through the worker and connector to Apache which returns it to the client.

Kind of complicated, huh? And of course, this is just my best guess.

Questions and problems

This past week I began updating the NLVM Application to support switching between multiple languages. On the screen that displays applets I added a JComboBox below the applet containing a list of languages that a user can select. Everything seemed to work fine until I noticed that once I clicked on the list box the applets stopped updating. After much pain and suffering and digging through the Swing source code I figured out the problem:

When JComboBox gets focus it (via javax.swing.plaf.basic.BasicPopupMenuUI.grabContainer) calls addMouseListener, addMouseMotionListener, and addMouseWheelListener on the applet, which are implemented in java.awt.Component. This is where the problems begin. All of those methods contain an innocuous line:

newEventsOnly = true;

There is the evil. The effect of this flag being set bears fruit in java.awt.Component.dispatchEventImpl where there is a set of nested if statements that begins:

if (newEventsOnly) {
if (eventEnabled(e)) {
processEvent(e);
}
}
...

The result is that none of the old AWT events (that the applets depend on) are sent after one of those listeners is added and the flag is set. It turns out, there are more methods that set the flag as well: addComponentListener, addFocusListener, addHierarchyListener, addHierarchyBoundsListener, addKeyListener, and addInputMethodListener, but the JComboBox never calls those methods.

My simple hack was to override those addXXXListener methods in my base class to prevent the calls from reaching down into Component where the flag gets set. I did this by writing a new J14Applet base class which extends java.applet.Applet and extending my former J10Applet base class from J14Applet instead of Applet. I’m sure this may have side effects but it seems to work fine for now.

As to why JComboBox was adding those listeners in the first place? I think it is so it can handle mouse messages when it has the focus but the mouse is not over the component, so I have defeated that behavior, but I can live with it. It is not as clean a hack as I would like because I want to be able to compile the applets for running on machines that don’t have a 1.4+ JVM. It would be nice to be able to have my Swing app dynamically add those methods like you can do in Ruby, but I don’t quickly see how to do that. For now, I will just change what J10Applet extends depending on what target I’m compiling for.