I signed up for a new AWS account and ran their sample LogAnalyzer Job Flow using the AWS console. That was easy enough. Next I attempted to run the same sample from the command line using the Amazon Elastic MapReduce Ruby Client.

Note: The Ruby Client README turns out to be very helpful.

Next I downloaded the source and looked at. Seems simple enough. I notice that this sample uses a library called Cascading, which appears to be a way to simplify common job flow tasks.

After adding the elastic-mapreduce app to my path and setting up my credentials file, I ran:

elastic-mapreduce –create –jar  s3n://elasticmapreduce/samples/cloudfront/logprocessor.jar –args  “-input,s3n://elasticmapreduce/samples/cloudfront/input,-output,s3n://folksemantic.com/cloudfront/log-reports,-start,any,-end,2010-09-07-21,-timeBucket,300,-overallVolumeReport”

It produced:

INFO Exception Retriable invalid response returned from RunJobFlow: {”Error”=>{”Details”=>#<SocketError: getaddrinfo: nodename nor servname provided, or not known>, “Code”=>”InternalFailure”, “Type”=>”Sender”}} while calling RunJobFlow on Amazon::Coral::ElasticMapReduceClient, retrying in 3.0 seconds.

After some poking around, I realized that I specified “west-1″ as my region when it should have been “us-west-1″. This resulted in the client trying to contact a non-existent server I’m guessing.

So now, my jobs started, but failed immediately. I logged into the AWS console and clicked on one of the failed job flows to see the reason for the failure (Last State Change Reason):

The given SSH key name was invalid

Googling found: http://developer.amazonwebservices.com/connect/message.jspa?messageID=166768

Which at first confused me, then I went ahead and followed the link (while logged in) and did what it said to. (Amazing how that works sometimes ) It prompted me to create a new key and to assign it a name.

After I had generated the key and put its name in the credentials.json, things worked like a charm. It turns out that if you run a job from scratch, it has to fire up an EC2 instance in order to run the job, and that can take a while. To avoid that start up time, you can run:

elastic-mapreduce –create –alive –log-uri s3://my-example-bucket/logs

As mentioned in the README.TXT

My next steps are to:

1. Modify the job flow and run that job flow.
2. Run the job flow locally.
3. Debug the MapReduce portion of the job flow.

• Duplicate entries. Search and recommendation results that list multiple entries for the same resource.
• Catalog pages. Search and recommendation results that link to catalog pages for resources (people would rather go directly to the resource, but the metadata providers want people to go to their catalog entry for the resource).
• Dead links. Results that link to resources that no longer exist.
• Urls without metadata. When someone shares a resource or inserts the recommender widget in a page for which we don’t have metadata, we need to be able to generate metadata.

Duplicate entries show up because:

• Two feeds specify entries with the same permalink.
• The same feed gets added twice (maybe different formats for the same feed, eg. RSS, Atom)
• Multiple catalogs provide metadata for the same resource.

Dealing With Duplicate Feeds

Problem: In folksemantic a user can enter the url of their blog and we will detect the feeds from the page and add them. We use the feeds to generate personal recommendations. The problem is, a blog typically has 3 or more feeds all of which contain the same content, just provided in different formats (e.g. RSS, Atom etc). So we really don’t want all of the feeds to be generated.

Solution 1: One approach to solving this is to try to detect the duplicate feed the first time we harvest it, don’t add its entries to the index, and then flag the feed as “duplicate” so that we don’t harvest it again. Store in the feed the id of the feed it duplicates. One potential problem with this is that if someone registers a feed that has just the entries tagged a certain way (e.g. all of the entries tagged apple on the gizmodo feed), then if the main feed is already registered, all of the entries on the filtered feeds duplicate the entries in the main feed, so the entries are duplicate, but the feed is not. If we want to use the feed as a basis for making recommendations to the user, we don’t want to use the main feed.

Solution 2: Another approach to the problem is to just add the feed, and harvest it, but then flag the entries as duplicates. Our thought about doing this is to store in each entry a list all of the feeds that the entry belongs to. We need to verify that this won’t slow down our Lucene queries.

It seems that Solution 2 may be best and make it up to the app to avoid adding duplicate feeds (like the 4 feeds for the same blog that Folksemantic does).

Dealing with Catalog Entries

A number of NSDL and other projects such as OER Commons  have created large catalogs of online resources. Sometimes their metadata is harvested directly from the resource websites. Sometimes they enhance that metadata with new information. Sometimes they create metadata for resources that don’t provide their own metadata. The catalog websites often provide services such as rating, discussion, and other valuable services and so they want people to come to their websites and use them. While, these services are nice, when people are searching for resources, they likely want to look at the resource first and make their own judgement if that is possible, and then read more about it if they are interested. I think this is because the cost of looking at an online resource is minimal (as compared to buying something or attending a course, for example). So the catalog issue leads to two problems:

Problem: When people see search results, they likely want to go directly to the resource instead of to a catalog page.

Solution: When a catalog page is the only entry for a resource, that entry is flagged “primary”. As soon as we create an entry that goes directly to the resource, we flag that new entry primary, and the catalog entry as not primary; we also store the id of the catalog entry in the list of duplicate entries that we store in the new entry. When searching, by default return only primary entries unless the application explicitly requests all entries. Return a flag indicating that an entry has catalog entries. Provide an API for requesting catalog entries for a specific entry.

Problem: In most cases, catalog metadata does not provide the url of the resource it is cataloging.

Solution: Initially flag the entry as “primary” so it will show up in search results. Later, asynchronously crawl the catalog pages to find the url of the catalogued resource. Once the direct url is known, create a new entry for the resource and store the id of the catalog entry in the list of “related entries” that we store for the new entry. Flag the catalog entry as not primary and the new entry as primary. Copy the metadata from the catalog entry into the new entry. Use the resource domain as the key for the feed to add the new entry to. If the feed does not already exist, create one for it.

Problem: If there are multiple entries (catalog etc for a resource), which metadata should we use to calculate the recommendation for the resource?

Solution: Options might be: (a) the metadata provided by the resource, (b) metadata generated by a crawl of the resource – I think this is bad because frequently metadata is more descriptive than the page itself, (c) the first catalog entry found for the resource, (d) the largest set of metadata for the resource. My thought it to always use the largest set of metadata for the resource unless there is no catalog entries (like in the case of where we crawl a website), in which case we must use the metadata generated by the crawl. In order to facilitate this approach, we: (1) for entries, we store whether or not the metadata came from that resource itself or not, (2) whenever we detect a new catalog entry for a resource that already has an entry, we look to see if the metadata in the existing entry was copied from a catalog entry; if it was, compare the size of the metadata from the two entries and update the metadata with the new catalog entry metadata if it is larger. For the purpose of calculating recommendations it might make sense to use all of the metadata from all of the sources.

Problem: When a website requests recommendations for a url, normally we want to return non-catalog entries, but when a catalog requests recommendations for one of its urls, they likely want their own catalog entries back if they exist.

Solution: When generating recommendations, for recommended entries that have catalog entries, check those and recommend those catalog entries instead.

Detecting and Handling Feed Entry Deletions

Problem: OAI has a way to tell you that an entry has been deleted, but RSS does not. How can you detect when an entry has been deleted, and what should you do when it is deleted?

Solution: My thought is that this is just part of what our dead link handler does. It finds entries with dead links and flags them deleted or actually deletes them. When we re-index we remove items from the index that have been flagged deleted.

Dealing with Dead Links

Problem: Many times the resources in our indexes get taken down or moved without notification (the source of the metadata doesn’t get updated or it doesn’t get updated for a while). What should we do in that situation?

Solution: We will write a bot that will flag entries dead. Once entries are dead they won’t show up in search or recommendation results. Should they still be used as the basis for recommendations? Probably not. Maybe we create another process that looks for the new location of the dead entries?

Generating Metadata for a URL

Problem: When someone adds an entry but doesn’t provide metadata, we need to be able to generate metadata for the entry. We also need to know which feed to put it into.

Solution: The application should give us a feed id, or a display url for the feed along with the entry URL. If it does not send a feed ID, we will look for feed using the host portion of the entry permalink. If one does not exist, it will create one and specify the host the display url for the feed, that way future entries for that feed will always go into that feed.

Generating Recommendations for Web Pages We Haven’t Indexed Yet

This is similar to the previous issue, we want people to be able to add the OER Recommender widget to their pages and have them just start working, removing the requirement that they add their resources to our index before we can provide recommendations. We can analyze and provide recommendations in real time, but that tends to bury our server if it gets a bunch of requests for real time recommendations  all at once.

Problem: Provide recommendations for URLs that haven’t been indexed yet.

Solution: When the recommendation is requested, add an entry for the URL, flagging it needing to be scraped. Flag the feed as being not-recommendable. If we don’t have a domain feed for the URL, add a domain feed for the entry and specify it as the feed for the entry. Queue the feed for approval by site admins.

This brings up the issue of being able to narrow the scope of the space into which recommendations are made. Depending on the context, we want to consider different sets of items to recommend. For example, in folksemantic, for personal recommendations we let users add feeds they produce, but we don’t necessarily want to include their stuff in recommendations that we make to other people.

Problem: Narrow the scope of the space that we recommend items from.

Solution: Define recommendation tasks by specifying the aggregation of feeds that we are recommending from and the aggregation of feeds that we are recommending into. Store those ids in the recommendation table.

http://mydomain.com:8080/lms/index.jsp

Some firewalls block port 8080 so I wanted my site to be available on port 80 so that it uses urls like:

http://mydomain.com/lms/index.jsp.

One option was to modify the Tomcat configuration to listen on port 80. However, I already have Apache installed and listening on port 80 (to serve other content) so I couldn’t that. Instead I configured Apache to route requests for my web application to Tomcat. I’ve been through this process a number of times before, and it never seems to go smoothly, so I document it here. I am running Apache 2.2 and Tomcat 6 on 32 bit Ubuntu Linux.

Overview

The steps are:

1. Download the Apache jk connector module (mod_jk.so).
2. Create Apache module configuration files for the jk connector (jk.load and jk.conf) and enable the module.
3. Create a worker.properties file to configure the Tomcat worker for the connector.
4. Define an AJP connector in your Tomcat configuration file (server.xml)
5. Assign urls to Tomcat in your Apache virtual hosts file.

Download the Apache jk connector module (mod_jk.so)

Apache uses the jk connector module to talk to Tomcat. I downloaded it from a subdirectory of http://www.apache.org/dist/tomcat/tomcat-connectors/jk/binaries/. I wasn’t sure which OS I was running and whether or not I was running a 32 bit version (i586 directory) or a 64 bit version (x86_64). To find this out, I ran:

file /usr/bin/file
/usr/bin/file: ELF 32-bit LSB executable, Intel 80386, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.15, stripped

So I downloaded: http://www.apache.org/dist/tomcat/tomcat-connectors/jk/binaries/linux/jk-1.2.28/i586/mod_jk-1.2.28-httpd-2.2.X.so

I chose that version because I am running Apache 2.2. There are different versions for different versions of Apache.

I put the file in my Apache modules directory (/usr/lib/apache2/modules/) and renamed it to mod_jk.so.

Create Apache module configuration files for the jk connector (jk.load and jk.conf) and enable the module

In order to get Apache to load and configure the jk connector module, I created jk.load and jk.conf files (in /etc/apache2/mods-available/) and then enabled them. jk.load just tells Apache where to find the module:

LoadModule jk_module /usr/lib/apache2/modules/mod_jk.so

jk.conf ties everything together by configuring the jk connector module:

# Where to find workers.properties
# Update this path to match your conf directory location (put workers.properties next to httpd.conf)
JkWorkersFile /etc/apache2/workers.properties

# Where to put jk shared memory
# Update this path to match your local state directory or logs directory
JkShmFile     /var/log/apache2/mod_jk.shm

# Where to put jk logs
# Update this path to match your logs directory location (put mod_jk.log next to access_log)
JkLogFile     /var/log/apache2/mod_jk.log

# Set the jk log level [debug/error/info]
JkLogLevel    info

# Select the timestamp log format
JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "

For more information about the Apache jk connector module configuration, see the Tomcat Connector – Apache Webserver HowTo.

Initially I set the JkLogLevel to debug, so I could see any error messages, but then changed it to info once I had everything working.

After creating the files, I enabled the module using:

sudo a2enmod jk

That creates symlinks to the jk.load and jk.conf files in the mods-enabled directory where my Apache is configured to look for modules to load.

Define an AJP connector in your Tomcat configuration file (server.xml)

AJP is an efficient protocol that Apache and Tomcat can be configured to use to talk to each other. I set up an AJP connector in my Tomcat configuration file (/etc/tomcat6/server.xml). The default configuration file has the connector defined but commented out, so I uncommented it:

<!-- Define an AJP 1.3 Connector on port 8009 -->
<Connector port="8009" protocol="AJP/1.3" redirectPort="8443" />

For details see the Tomcat AJP Connector documentation.

Create a worker.properties file to configure the Tomcat ajp worker for the connector

“A Tomcat worker is a Tomcat instance that is waiting to execute servlets or any other content on behalf of some web server”.

Note: this quote from the documentation is a bit curious, because, nowhere in the Tomcat configuration files do I tell Tomcat about the worker. I think that the worker is actually a process that the jk connector spawns.

I configured a worker to listen to Apache requests by creating a worker.properties file in the same directory as the Apache configuration file (/etc/apache2/workers.properties).

# Define 1 real worker using ajp13
worker.list=worker1

# Set properties for worker1 (ajp13)
worker.worker1.type=ajp13
worker.worker1.host=localhost
worker.worker1.port=8009

The jk connector knows how to talk to this worker, because the file name is specified in the Apache jk connector configuration file (/etc/apache2/mods-available/jk.conf). For more information see the Tomcat Connector Quick Start or the Tomcat Connector Reference Guide.

Assign urls to Tomcat in your Apache virtual hosts file

After I configured a Tomcat worker to listen to AJP requests and configured Apache to use the jk connector module to talk to that worker, the last thing that was needed was to configure my web site’s virtual host (/etc/apache2/sites-available/default) to route urls to Tomcat:

<VirtualHost *:80>
...
        JkMount /lms/* worker1
...
</VirtualHost>

Note that worker1 is the name I gave to the worker I set up in the workers.properties file. Note that by using a * mask, I routed all requests (including static files) through Tomcat. Alternatively I could have configured only jsp requests to be routed to Tomcat, using:

<VirtualHost *:80>
...
        JkMount /lms/*.jsp worker1
...
</VirtualHost>

Restart Tomcat and Apache

sudo /etc/init.d/tomcat6 restart
sudo /etc/init.d/apache2 restart

Lifecycle

To the best of my understanding, the relevant lifecycle is:

1. When Tomcat starts up, it begins listening for AJP requests on port 8009 (because the connector is defined in /etc/tomcat6/server.xml).
2. When Apache starts up, it loads the jk connector module (because it is defined in /etc/apache2/mods-enabled/jk.load).
3. When Apache loads the jk connector, its configuration file (/etc/apache2/mods-enabled/jk.conf) tells it to send requests to the specified Tomcat worker and to use shared memory to do that.
4. It is not clear to me whether or not the Tomcat worker gets spawned when Apache starts up or on each request. I don’t see how it could get spawned when Tomcat starts up since Tomcat has no way of knowing about it.
5. Apache receives a request for a url that is mapped to Tomcat (in the virtual host file – /etc/apache2/sites-enabled/default).
6. Apache uses the jk connector module (mod_jk.so) to generate a request to send to Tomcat via a Tomcat worker.
7. The Tomcat worker communicates with Tomcat using the protocol (AJP) and port (8009) defined in the workers configuration file (/etc/apache2/workers.properties).
8. Tomcat processes the request and returns the response back through the worker and connector to Apache which returns it to the client.

Kind of complicated, huh? And of course, this is just my best guess.

Questions and problems

1. Dug up my old copy of Visual Studio 6 CDs and installed the command line utilities. Apparently, not just any version will do; you have to be in sync with the version used to compile ruby.
2. Added VS6’s bin directories to my Windows path. Default install locations are:
   C:\Program Files\Microsoft Visual Studio\VC98\Bin;C:\Program Files\Microsoft Visual Studio\Common\MSDev98\Bin;

   Alternatively, you can get a command prompt in  C:\Program Files\Microsoft Visual Studio\VC98\Bin and run VCVARS32.BAT to add those directories to your path.

3. Added some typedefs and a function to C:\Program Files\Microsoft Visual Studio\VC98\Include\SYS\TYPES.H that are not available on Windows:

   #ifndef _UINT_T_DEFINED
   typedef unsigned char  u_int8_t;
   typedef unsigned short u_int16_t;
   typedef unsigned int   u_int32_t;
   typedef unsigned __int64 u_int64_t;
   #define _UINT_T_DEFINED
   #endif

   #ifndef snprintf
   #define snprintf _snprintf
   #endif

Now I can do:

gem install bcrypt-ruby

I get the “Building native extensions. This could take a while…” message and it works!

I’m happy once again. I’m satisfied. And I didn’t even have to change religions.

While attending the 2008 Mountain West Ruby Conference and needing something to hack on I decided to take a crack at the project. I already hacked Google Translate for Send2Wiki so I figured it would be a snap to do for this project. My plan was to take the search text, run it through the translator for each of the languages to search, then pass the translated queries off to the Google search sites for each of the languages and then pass those pages through Google translate to get English versions of the pages. I soon found that Google has already done most of the work for me with their cross-language search.

The only thing cross-language search doesn’t do for me is collate all of the language results into a single results page. You can only search for results in a single targeted language. Anyway, between sessions (a coder has always got to brag about how fast he can work right I threw together a Multilingual Google Search Mashup that does the job. As I put it together, a couple of things almost immediately stood out:

• Wikipedia owns the top hit slot for many searches. Because their pages are essentially equivalent in the different languages, listing that entry for each of the languages isn’t especially useful.
• Interleaving search results is difficult. Rather than try to figure out an intelligent way to order in real-time the search results from the various languages, I just give the first two from each language and provide a language for getting more. I’ve got ideas for interleaving results, but none of them are too easy. Notice also that I haven’t included English in the search, which is probably where the most relevant pages will actually come from.

These issues makes me wonder if a different approach would be preferrable. Perhaps Google could annotate search results with relevant pages in different languages. This also makes me think about Google’s search result ordering. Google search results appear to be determinative (if you execute the same search twice, the same item will show up at the top of the list). While this may be what we have come to expect, my experience with writing OER Recommender makes me believe that it isn’t necessarily the best or the fairest thing to do. When ranking pages it is often the case that the scores of the top two or even 10 pages are statistically indistinguishable. So why should the one that happens to have a .00000001% higher score always show up first. My approach with was to identify a strata of rankings for those “highest ranked pages” that are virtually indistinguishable, I randomize the order. This seems fairer since it is quite natural for users to click on the first item in on a search results page, thus biasing it to become more and more popular.

With slicehost I get a VPS for $20 / mo. I’ve never used a VPS before but have run my own web servers, so it seems worth a shot. In the process of getting it set up I’ve learned more about DNS, iptables, fastcgi, etc than before, so that is fun, but also a time drain. Anyway, now I’ve got the access I need to set up Send2Wiki and run rails apps.

After hearing Justin rave about WPMU, I thought I would give it a shot. WP normally installs easy, so I expected the same. It would have except I wanted to install WPMU in a subdirectory. My Googling gave warnings against doing so, but I shrugged them off and forged ahead and tried to hack it to get it to work. Bad idea. It may be possible to do, and I got it to almost work, but certainly it doesn’t fall within the scope of famous 5 minutes install. So I’ve backed off and returned to the single user version.

• Added Collections. The index now contains more than 90,000 records from over 100 collections and 26 languages.
• Added Support for Harvesting via SQI/WSDL. Support for harvesting SQI/WSDL using Axis was added in order to harvest MERLOT via Araidne.
• Catalog Links. When providing recommendations if we have catalog links and direct links for resouces such as in the case of OER Commons and MERLOT both are provided.
• OAI Set Discovery – In order to get the names of collections from OER Commons the ability to discover the OAI Sets (collections) was added to the harvester.
• Lucene. The home brewed search and recommendation system was swapped out with Lucene. This makes for faster and better searching as well as faster indexing. The full range of query syntax supported by Lucene is now supported.
• Multi-Lingual. With Lucene in place OER Recommender can now support language-specific search and recommendation.
• Additional Metadata. Additional metadata was added to search and recommendation results: Descriptions, Authors, Date (metadata), Date (relevance was calculated).
• Home Page Cleanup. The home page was simplified by moving the Greasemonkey script and example resources and to a separate page.
• Search Results. Search results were modified to look similar to Google search results. In cases where the index contains both links to catalog pages and direct links for a resource, a Metadata link next to the title takes you to the catalog page. A “Related Resources” link is also provided next to each item in search results. This makes it easy to see recommendations.
• More Recommendations Page. The geek friendly page was replaced with a page that looks essentially like the search results page. A link to the original page containing details such as is included near the top of the page.
• Incremental Updates Support. The recommender was modified to support incrementally updating the indexes and recommendations without losing user data. It now runs every night, harvesting the collections, indexing and creating recommendations for new records. Once a week it re-runs all recommendations so that recommendations could be created that point at new records.
• Time on Page. Average time on page tracking was added and used to adapt the recommendations algorithm.
• Localized Interface. The main web pages were translated into Spanish, French, German, Japanese, Dutch, Russian, and Chinese using Google Translate (if you speak one of those languages and want to help the translation, feel free to send me fixes). Localization is supported via the swell Simple Localization rails plugin. The web app also auto-detects the language set in your web browser and sets that as the default search interface.

More on implementation later…

Other times problems occur because we broke something and we didn’t catch it in our testing. This is aggravated by browser incompatibilities. Because old hardware and software tend to hang around schools longer than other places, strange things show up. Right now I’m trying to track down some of those types of issues.

A few years ago when I was more actively developing the NLVM I used to keep old machines around so I could test old browsers. One of the reasons multiple machines were needed was that I couldn’t find an easy way to run multiple versions of Internet Explorer on the same machine. Yesterday I googled to see if there is anything new out there to help with this issue. I was pleasantly surprised to find a utility by tredosoft that allows you to install multiple versions of IE on your PC. Thanks tredosoft! Unfortunately after installing the multiple browsers, I’m still not able to see the reported problem even when running on the same browser.

Yesterday we released the OER Recommender system that I have worked on. There are still many things that could be added or tweaked, but it does something useful already so out the door it goes! I’m concerned that we are calling it a recommender as the “recommendations” it currently provides are not specific to a user, they are related resources generated via a content-based approach. The proposal and plans are to make it a recommender based on user profiles. See my previous post for details about where it is intended to go.

The Problem. The William and Flora Hewlett Foundation, the National Science Digital Library, and other large organizations have made large investments in the development of electronic resources that can be used for teaching, learning, and research. They are keen on finding ways to increase the impact of their investment. One way to do that is to create tools that make it easier for poeple to find resources that are useful to them. One approach to solving this problem are search and browsing tools that help people find good resources such as NSDL, OCW Finder, and of course Google. Recommender systems approach the problem by helping bringing resources to peoples’ attention without having them to go look for them specifically Google’s Adsense technology and Amazon’s recommendations are two of the most common examples. Some of the challenges with general search tools such as Google and even more focused ones such as NSDL’s main search is that they cast too wide a net and you get back resources do not match. On the other hand, while most collections of open courseware and digital libraries provide search, it is nice to be able to search across repositories at the same time to find the best resources wherever they reside.

The Vision. In creating the OER Recommender we set out to create a service that would help people find relevant open education resources. The first step in this was to create an automated process for clustering related resources. The hope is that presenting links to related resources to users when looking at a resource could help people stumble upon resources that are relevant to them even though they might have not been actively looking for them. We could tune this service by monitoring what resources people visit, share, rate, tag, and otherwise pay attention to. We could use this attention metadata to create profiles of what people’s interest are. The profiles could be used to push recommendations to people even when they are not browsing perhaps via email or other means.

Exploration. Since learning about it I have felt that the recommender would be one of the more interesting folksemantic tools to work on since I initially thought to pursue a latent semantic analysis approach. I had heard about LSA from my exposure to Andy Walker’s and Art Graesser’s work. I have also had a number of interesting discussions about clustering with Adele Cutler about her work on Random Forests. I got a hold of the Handbook of Latent Semantic Analysis and explored the LSA online resources I could find as well as the lsa module for R. After spending significant time reading and playing I came to the conclusion that it would likely more complex to implement and computationally expensive than I wanted. I also concluded that everything that I would need to do to implement a standard Term Vector Model approach could be used to later implement an LSA approach. During my exploration, I came across a number of useful online resources and books including Gerard Salton’s work.

An Explanation for My Mother. I’ve already been asked by Shelly to explain the implementation of the recommender in language that my mother would understand, so I’ll give that explanation first. We use the folksemantic feed harvester to gather information (metadata) about OERs into databases. For each pair of resources that are related enough, the recommender uses the titles, description, and tags to calculate a score indicating how related they are. Recommended resources are the ones scored to be the most similar. The similarity of two resources is based on an automated analysis of the words in their metadata. Got that Mom?

We use four phases to arrive at recommendations: (1) parse metadata, (2) calculate local term weights, (3) calculate global term weights, (4) calculate similarity scores.

Parse Metadata. Using a string tokenizer we break metadata text into terms. We throw away stop words (common words such as ‘and’, ‘is’, ‘of’ that do not add meaning). Next we convert terms into their stems using a common algorithm; for example ‘running’, ‘ran’, and ‘runs’ all get collapsed into ‘run’.

Calculate Local Term Weights. A local term weight is a measure of how important a word is for describing a document. The more frequently a word appears in a document, the more important it is… up to a point. Of course, where a word appears in a document is probably a good indicator of how important the word is as well. For example if a word appears in a title, it is probably more important than if it appears in the body. One more important factor to consider is document length (total number of terms in the document); all other things being equal, the longer a document is, the more times a term will appear in it. So we normalize term frequencies using the document length. One last issue relates to filler words.

To calculate local term weights OER Recommender uses a function that looks like this.

The x-axis represents term frequency and the y-axis represents the local term weight. As you can see, the more time a term appears in a document, the more weight it is given. However after reaching a threshold, it plateaus. I think this is especially important in situations where document creators might be trying to game the system. The formula used is:

ltw_i = 1 / (1 + (e^(.0044)dlen) .7^{(fti – 1)})

ltw_i is the local term weight for a given term in a document.

dlen is the total number of terms in the document.

ft_i is the number of times a given term appears in a document.

Note that the constants .0044 and .7 in the equation determine the shape of the curve. I chose those values based on their use in the MeSH system. As explained there, the values should be tuned to your data set using an empirical approach. I played with the parameters some and the values they used seemed fine, so I adopted them.

Calculate Global Term Weights. Global term weights are a measure of how important a word is for distinguishing documents within a collection. The more documents a word appears in, the less value it has for characterizing documents. For example, the term ‘USU’ appears in every metadata record for resources in USU’s OpenCourseware. As a result, it has no value for characterizing clustering resources. The term ‘USU’ becomes in a sense, a stop word, similar to those thrown away during the parse phase. To calculate the global term weight for a term, the number of documents that it appears in are counted as well as the total number of documents.

To calculate global term weights OER Recommender uses a function that looks like.

The x-axis represents the number of documents that a term appears in. The y-axis represents the global weight assigned to the term. The formula used is:

gtw_i = log (D/df_i)

gtw_i is the global term weight for term.

D is the total number of documents.

df_i is the number of documents that the term appears in.

See Mi lslita’s explanation of Term Vector Theory for details.

Calculate Similarity Scores. Once local term weights and global term weights are calculated, we are ready to create recommendations. To create recommendations for a document we first find all documents that have any of the same terms in it. It turns out that this can be a very large number of documents (e.g. 40,000 in our system). For each pairing of the document being considered and a document with matching terms we calculate a similarity score. Because calculating 40,000 scores can take a long time (about 15 seconds in our current system), we shorten the list to consider to 200. We do this by sorting the pairs according to the number of overlapping terms. To calculate the similarity score for a pair of documents, we sum over all of the terms that they share in common. The contribution from each term is a combination of the local term weight in the first document, the local term weight in the second document and the global term weight. Because in OER recommender, each feed can be considered a separate collection, we calculate global term weights for each of the feeds and use them in calculating the similarity score.

To calculate the contribution of an individual term to the similarity score, OER Recommender uses:

sst_i = (gtw_1i)(ltw_1i)(gtw_2i)(ltw_2i)

sst_i Is the contribution to the similarity score for a given term.

gtw_1i is the global term weight from the feed that the first document is in.

ltw_1i is the local term weight from the first document.

gtw_2i is the global term weight from the feed that the second document is in.

ltw_2i is the local term weight from the second document.

See NCBI’s explanation of how MeSH calculates Related Articles for details. Once OER Recommender has calculated similarity scores for the 200 documents, it sorts the documents by those scores and stores the top 10 in a database.

Displaying Recommendations. Now that the recommendations are stored in a database, displaying them to the user is straightforward. We have created a Greasmonkey script that requests recommendations for each web page that a user browses. If OER Recommender returns any, the script inserts HTML for the recommendations into the web page. The eduCommons team is building a plone tool so that anyone running eduCommons can turn on recommendations. That way users won’t have to have install the Greasemonkey script in order to see recommendations on the eduCommons website. The OER Recommender site publishes the XML format it returns so others could add recommendations into their websites.

Getting Resources into the Recommender. Currently OER Recommender only provides recommendations for URLs that it has metadata for, so in order to get recommendations, you need to register a metadata feed. If you have an OCW site that provides an RSS feed, you can do that at OCW Finder. If you have an OAI feed or an RSS feed for learning objects or other OERs, you can add your metadata by sending a feed title, URL, and display URL (home page of the repository) to oerrecommender AT cosl DOT usu DOT edu.

Future Directions. There are many things left to be done: (1) adapt recommendations by monitoring which recommended resources people click on and how long they stay at recommended resources, (2) provide links to more recommendations (so users can see more than the default 5), (3) provide a way for people to indicate when a recommendation doesn’t work, (4) provide a way for people to register and login, so they can receive personalized recommendations (perhaps via email), (5) create a process whereby recommendations can be updated without going through all the documents in the database, (6) add functionality for retrieving recommendations for arbitrary web pages (recommending on demand) perhaps via a bookmarklet, (7) create whatever Greasemonkey-like add on is supported on IE. We also plan to integrate the recommender with the aggregator and feed reader to provide recommended news items based on the attention people have previously paid to news articles.

Wow! I realize that this is a long post, but it is not a simple topic. Hopefully the explanation is useful.

The database. I’ve checked the database and found that we have nearly 2 million rss entries and about 350 thousand unique tags. We’ve indexed the entries so that we can access recent ones quickly. We haven’t indexed the entries for fast searching by tag.

Search was dog slow. Ozmozr’s search which is visible from most pages, takes entered terms and treats them as tags with which to search entries. Joining through the massive tag table to the even more massive entries table was taking forever (on the order of 45-60 seconds per query). We temporarily disabled entry searching until I poked around and found that we hadn’t created an index on tag names. Just putting an index on tag names reduced the query time to around 8 seconds. Much shorter, but still too long. So I reduced the complexity of the query. It may not be as powerful as it used to be, but now executes in under two seconds.

Aggregator’s 20 Java threads talking to postgres was stealing the CPU. We use a Java aggregator daemon that I wrote to harvest RSS feeds. We haven’t harvested in a while because of the problems we have experienced. When I fired it up today I found that it brought the site to its knees when it started doing its work. By monitoring the CPU I saw that the aggregator daemon was stealing all of the CPU leaving none for Rails to use to serve pages. By default the aggregator checks feeds every hour. When doing this it can use up to 20 threads. Each thread creates a connection to postgres. It was actually postgres that was stealing all of the CPU, but only because of how the Java daemon was talking to it. My quick fix is to limit the aggregator to 1 thread. I initially designed the aggregator to use many threads because it turns out that most of the time spent in harvesting is used up in waiting for web servers to respond. I guess I need to figure out another approach. I’m going to look to see if we can throttle how much CPU the Java threads (and corresponding Postgres processes) use.

Shrinking tag clouds are a fun idea but a pig to implement. Similar to how OCW Finder allows you to filter your browsing by selecting tags, we initially implemented the same idea in oz via a shrinking tag cloud. The idea was that as soon as you clicked on a tag in a cloud, in addition to the items being filtered, the tags in the resulting tag cloud would be filtered as well (shrinking the cloud). It turns out that this kind of query is horrendously expensive, at least the way we implemented it. It also turns out that users don’t seem to understand what is going on. As a result of this, we ripped the shrinking tag clouds out… almost. I just found an instance where it was left in. They are gone now.

There are more ugly queries to look at but I’ve gotten most queries down to under 4 seconds, which is still a long time, but at least the website doesn’t die. I’ll write more as I find it out.

As a note, the way I was able to easily identify the nasty queries was by going into the postgres config file (var/lib/pgsql/data/postgresql.conf) and setting the option log_min_duration_statement = 3000 (milliseconds). With this option set, every query that takes longer than three seconds is written to the postgres log file.

Summary

• Searching and Publishing – People come to the MTDL to find and share resources
• Overcoming Isolation – People come to the MTDL to help overcome isolation
• Discuss Development – Talk with others about the development of math software

Personas

• Teacher Developer
• Professional Developer
• Educational Researcher
• Inexperienced Developer
• Hobbyist Developer

Metaphor and Principles

• Workshop metaphor
• Design for multiple roles
• Design for multiple levels of expertise
• Provide activity indicators

I like the workshop metaphor, though I think that perhaps there are better. I can’t really discern the implication of designing for different roles and different levels of expertise. I lilke the idea of activity indicators. I realize that this is an area of recent interest throughout the field.