I signed up for a new AWS account and ran their sample LogAnalyzer Job Flow using the AWS console. That was easy enough. Next I attempted to run the same sample from the command line using the Amazon Elastic MapReduce Ruby Client.

Note: The Ruby Client README turns out to be very helpful.

Next I downloaded the source and looked at. Seems simple enough. I notice that this sample uses a library called Cascading, which appears to be a way to simplify common job flow tasks.

After adding the elastic-mapreduce app to my path and setting up my credentials file, I ran:

elastic-mapreduce –create –jar  s3n://elasticmapreduce/samples/cloudfront/logprocessor.jar –args  “-input,s3n://elasticmapreduce/samples/cloudfront/input,-output,s3n://folksemantic.com/cloudfront/log-reports,-start,any,-end,2010-09-07-21,-timeBucket,300,-overallVolumeReport”

It produced:

INFO Exception Retriable invalid response returned from RunJobFlow: {”Error”=>{”Details”=>#<SocketError: getaddrinfo: nodename nor servname provided, or not known>, “Code”=>”InternalFailure”, “Type”=>”Sender”}} while calling RunJobFlow on Amazon::Coral::ElasticMapReduceClient, retrying in 3.0 seconds.

After some poking around, I realized that I specified “west-1″ as my region when it should have been “us-west-1″. This resulted in the client trying to contact a non-existent server I’m guessing.

So now, my jobs started, but failed immediately. I logged into the AWS console and clicked on one of the failed job flows to see the reason for the failure (Last State Change Reason):

The given SSH key name was invalid

Googling found: http://developer.amazonwebservices.com/connect/message.jspa?messageID=166768

Which at first confused me, then I went ahead and followed the link (while logged in) and did what it said to. (Amazing how that works sometimes ) It prompted me to create a new key and to assign it a name.

After I had generated the key and put its name in the credentials.json, things worked like a charm. It turns out that if you run a job from scratch, it has to fire up an EC2 instance in order to run the job, and that can take a while. To avoid that start up time, you can run:

elastic-mapreduce –create –alive –log-uri s3://my-example-bucket/logs

As mentioned in the README.TXT

My next steps are to:

1. Modify the job flow and run that job flow.
2. Run the job flow locally.
3. Debug the MapReduce portion of the job flow.

• Duplicate entries. Search and recommendation results that list multiple entries for the same resource.
• Catalog pages. Search and recommendation results that link to catalog pages for resources (people would rather go directly to the resource, but the metadata providers want people to go to their catalog entry for the resource).
• Dead links. Results that link to resources that no longer exist.
• Urls without metadata. When someone shares a resource or inserts the recommender widget in a page for which we don’t have metadata, we need to be able to generate metadata.

Duplicate entries show up because:

• Two feeds specify entries with the same permalink.
• The same feed gets added twice (maybe different formats for the same feed, eg. RSS, Atom)
• Multiple catalogs provide metadata for the same resource.

Dealing With Duplicate Feeds

Problem: In folksemantic a user can enter the url of their blog and we will detect the feeds from the page and add them. We use the feeds to generate personal recommendations. The problem is, a blog typically has 3 or more feeds all of which contain the same content, just provided in different formats (e.g. RSS, Atom etc). So we really don’t want all of the feeds to be generated.

Solution 1: One approach to solving this is to try to detect the duplicate feed the first time we harvest it, don’t add its entries to the index, and then flag the feed as “duplicate” so that we don’t harvest it again. Store in the feed the id of the feed it duplicates. One potential problem with this is that if someone registers a feed that has just the entries tagged a certain way (e.g. all of the entries tagged apple on the gizmodo feed), then if the main feed is already registered, all of the entries on the filtered feeds duplicate the entries in the main feed, so the entries are duplicate, but the feed is not. If we want to use the feed as a basis for making recommendations to the user, we don’t want to use the main feed.

Solution 2: Another approach to the problem is to just add the feed, and harvest it, but then flag the entries as duplicates. Our thought about doing this is to store in each entry a list all of the feeds that the entry belongs to. We need to verify that this won’t slow down our Lucene queries.

It seems that Solution 2 may be best and make it up to the app to avoid adding duplicate feeds (like the 4 feeds for the same blog that Folksemantic does).

Dealing with Catalog Entries

A number of NSDL and other projects such as OER Commons  have created large catalogs of online resources. Sometimes their metadata is harvested directly from the resource websites. Sometimes they enhance that metadata with new information. Sometimes they create metadata for resources that don’t provide their own metadata. The catalog websites often provide services such as rating, discussion, and other valuable services and so they want people to come to their websites and use them. While, these services are nice, when people are searching for resources, they likely want to look at the resource first and make their own judgement if that is possible, and then read more about it if they are interested. I think this is because the cost of looking at an online resource is minimal (as compared to buying something or attending a course, for example). So the catalog issue leads to two problems:

Problem: When people see search results, they likely want to go directly to the resource instead of to a catalog page.

Solution: When a catalog page is the only entry for a resource, that entry is flagged “primary”. As soon as we create an entry that goes directly to the resource, we flag that new entry primary, and the catalog entry as not primary; we also store the id of the catalog entry in the list of duplicate entries that we store in the new entry. When searching, by default return only primary entries unless the application explicitly requests all entries. Return a flag indicating that an entry has catalog entries. Provide an API for requesting catalog entries for a specific entry.

Problem: In most cases, catalog metadata does not provide the url of the resource it is cataloging.

Solution: Initially flag the entry as “primary” so it will show up in search results. Later, asynchronously crawl the catalog pages to find the url of the catalogued resource. Once the direct url is known, create a new entry for the resource and store the id of the catalog entry in the list of “related entries” that we store for the new entry. Flag the catalog entry as not primary and the new entry as primary. Copy the metadata from the catalog entry into the new entry. Use the resource domain as the key for the feed to add the new entry to. If the feed does not already exist, create one for it.

Problem: If there are multiple entries (catalog etc for a resource), which metadata should we use to calculate the recommendation for the resource?

Solution: Options might be: (a) the metadata provided by the resource, (b) metadata generated by a crawl of the resource – I think this is bad because frequently metadata is more descriptive than the page itself, (c) the first catalog entry found for the resource, (d) the largest set of metadata for the resource. My thought it to always use the largest set of metadata for the resource unless there is no catalog entries (like in the case of where we crawl a website), in which case we must use the metadata generated by the crawl. In order to facilitate this approach, we: (1) for entries, we store whether or not the metadata came from that resource itself or not, (2) whenever we detect a new catalog entry for a resource that already has an entry, we look to see if the metadata in the existing entry was copied from a catalog entry; if it was, compare the size of the metadata from the two entries and update the metadata with the new catalog entry metadata if it is larger. For the purpose of calculating recommendations it might make sense to use all of the metadata from all of the sources.

Problem: When a website requests recommendations for a url, normally we want to return non-catalog entries, but when a catalog requests recommendations for one of its urls, they likely want their own catalog entries back if they exist.

Solution: When generating recommendations, for recommended entries that have catalog entries, check those and recommend those catalog entries instead.

Detecting and Handling Feed Entry Deletions

Problem: OAI has a way to tell you that an entry has been deleted, but RSS does not. How can you detect when an entry has been deleted, and what should you do when it is deleted?

Solution: My thought is that this is just part of what our dead link handler does. It finds entries with dead links and flags them deleted or actually deletes them. When we re-index we remove items from the index that have been flagged deleted.

Dealing with Dead Links

Problem: Many times the resources in our indexes get taken down or moved without notification (the source of the metadata doesn’t get updated or it doesn’t get updated for a while). What should we do in that situation?

Solution: We will write a bot that will flag entries dead. Once entries are dead they won’t show up in search or recommendation results. Should they still be used as the basis for recommendations? Probably not. Maybe we create another process that looks for the new location of the dead entries?

Generating Metadata for a URL

Problem: When someone adds an entry but doesn’t provide metadata, we need to be able to generate metadata for the entry. We also need to know which feed to put it into.

Solution: The application should give us a feed id, or a display url for the feed along with the entry URL. If it does not send a feed ID, we will look for feed using the host portion of the entry permalink. If one does not exist, it will create one and specify the host the display url for the feed, that way future entries for that feed will always go into that feed.

Generating Recommendations for Web Pages We Haven’t Indexed Yet

This is similar to the previous issue, we want people to be able to add the OER Recommender widget to their pages and have them just start working, removing the requirement that they add their resources to our index before we can provide recommendations. We can analyze and provide recommendations in real time, but that tends to bury our server if it gets a bunch of requests for real time recommendations  all at once.

Problem: Provide recommendations for URLs that haven’t been indexed yet.

Solution: When the recommendation is requested, add an entry for the URL, flagging it needing to be scraped. Flag the feed as being not-recommendable. If we don’t have a domain feed for the URL, add a domain feed for the entry and specify it as the feed for the entry. Queue the feed for approval by site admins.

This brings up the issue of being able to narrow the scope of the space into which recommendations are made. Depending on the context, we want to consider different sets of items to recommend. For example, in folksemantic, for personal recommendations we let users add feeds they produce, but we don’t necessarily want to include their stuff in recommendations that we make to other people.

Problem: Narrow the scope of the space that we recommend items from.

Solution: Define recommendation tasks by specifying the aggregation of feeds that we are recommending from and the aggregation of feeds that we are recommending into. Store those ids in the recommendation table.

• Personal recommendations page – For each user, provide a personal recommendations page. Visually separate recommendations that they have already clicked on.
• Personal recommendations tool – Include a personal recommendations button on our folksemantic bar that when clicked on will display their recommendations in the right panel. Linking to the recommendations from that panel will refresh the content in the iframe (not do a full page refresh).
• Personal recommendations action link – Include a link to the user’s recommendations page on their dashboard.
• Inject recommendations into activity feeds – Whenever we generate new personal recommendations, inject them into a user’s activity feed that is displayed on their dashboard.
• Email personal recommendations – Email personal recommendations to users as often as they would like (controllable in their account settings).

• Identity feeds – RSS feeds that users register that represent their interests. For example, their blog or their delicious account.
• Clicks – The articles that the user clicks on.
• Shares – The articles that the user shares to others.
• Comments – Articles that the user comments on.
• Time on page – Amount of time that a user spends on an article before moving on.
• Searches – Searches the user executes.

Recommendation Assumptions

Some of our assumptions are:

• Semantic relatedness – The more semantically similar an article is to articles that a user has paid attention to, the more interesting to the user.
• Attention types – Different types of attention should be given different weights. For example, following a link to an article should not give it as much weight as writing the article.
• Attention details – The particulars of a given type of attention might make it more important than another attention of the same type. For example, if a person shares an article with 100 people, it might be reasonable to infer that it is more important than an article that they share with one person.
• Entry recency – The more recently an article has been added to the system, the more interesting to the user (they probably haven’t seen it before).
• Attention recency – The more recently a user has showed attention to an article, the more weight that should be given to it.
• Attention frequency – The more frequently a user has showed attention to an article, the more weight that should be given to it.

Stating these assumptions reminds me of the difference between relevance and certainty. So while an item that a user clicks on may be more relevant than an blog article they have written, it is harder to be certain of that. Our approach is to give the click less weight than the article.

Recommendation Score

Right now, we score articles using the formula:

(relevance)(attention type)(attention details)(attention recency)(article recency)

For all articles that a user has paid attention to, we score the 20 “related articles” using this algorithm; rank the scores and cache the top 20 (that the user hasn’t already clicked on) to recommend to the user. There are obvious weaknesses to this approach, but we are going to start there and see where to go next.

Possible Extensions / Improvements

We are considering:

• Collaborative Filtering, Bipartite Graph, and Discriminative Weight Algorithms – Putting into production, the algorithm that combines discriminative weights with a novel sparse matrix clustering method we conducted research on.
• Modeling user interests separately – Modeling users interests using multiple term vectors (one for each interest) by extracting vectors from closely related (clustered documents) that users have paid attention to.