1. Dug up my old copy of Visual Studio 6 CDs and installed the command line utilities. Apparently, not just any version will do; you have to be in sync with the version used to compile ruby.
2. Added VS6’s bin directories to my Windows path. Default install locations are:
   C:\Program Files\Microsoft Visual Studio\VC98\Bin;C:\Program Files\Microsoft Visual Studio\Common\MSDev98\Bin;

   Alternatively, you can get a command prompt in  C:\Program Files\Microsoft Visual Studio\VC98\Bin and run VCVARS32.BAT to add those directories to your path.

3. Added some typedefs and a function to C:\Program Files\Microsoft Visual Studio\VC98\Include\SYS\TYPES.H that are not available on Windows:

   #ifndef _UINT_T_DEFINED
   typedef unsigned char  u_int8_t;
   typedef unsigned short u_int16_t;
   typedef unsigned int   u_int32_t;
   typedef unsigned __int64 u_int64_t;
   #define _UINT_T_DEFINED
   #endif

   #ifndef snprintf
   #define snprintf _snprintf
   #endif

Now I can do:

gem install bcrypt-ruby

I get the “Building native extensions. This could take a while…” message and it works!

I’m happy once again. I’m satisfied. And I didn’t even have to change religions.

While attending the 2008 Mountain West Ruby Conference and needing something to hack on I decided to take a crack at the project. I already hacked Google Translate for Send2Wiki so I figured it would be a snap to do for this project. My plan was to take the search text, run it through the translator for each of the languages to search, then pass the translated queries off to the Google search sites for each of the languages and then pass those pages through Google translate to get English versions of the pages. I soon found that Google has already done most of the work for me with their cross-language search.

The only thing cross-language search doesn’t do for me is collate all of the language results into a single results page. You can only search for results in a single targeted language. Anyway, between sessions (a coder has always got to brag about how fast he can work right I threw together a Multilingual Google Search Mashup that does the job. As I put it together, a couple of things almost immediately stood out:

• Wikipedia owns the top hit slot for many searches. Because their pages are essentially equivalent in the different languages, listing that entry for each of the languages isn’t especially useful.
• Interleaving search results is difficult. Rather than try to figure out an intelligent way to order in real-time the search results from the various languages, I just give the first two from each language and provide a language for getting more. I’ve got ideas for interleaving results, but none of them are too easy. Notice also that I haven’t included English in the search, which is probably where the most relevant pages will actually come from.

These issues makes me wonder if a different approach would be preferrable. Perhaps Google could annotate search results with relevant pages in different languages. This also makes me think about Google’s search result ordering. Google search results appear to be determinative (if you execute the same search twice, the same item will show up at the top of the list). While this may be what we have come to expect, my experience with writing OER Recommender makes me believe that it isn’t necessarily the best or the fairest thing to do. When ranking pages it is often the case that the scores of the top two or even 10 pages are statistically indistinguishable. So why should the one that happens to have a .00000001% higher score always show up first. My approach with was to identify a strata of rankings for those “highest ranked pages” that are virtually indistinguishable, I randomize the order. This seems fairer since it is quite natural for users to click on the first item in on a search results page, thus biasing it to become more and more popular.

With slicehost I get a VPS for $20 / mo. I’ve never used a VPS before but have run my own web servers, so it seems worth a shot. In the process of getting it set up I’ve learned more about DNS, iptables, fastcgi, etc than before, so that is fun, but also a time drain. Anyway, now I’ve got the access I need to set up Send2Wiki and run rails apps.

After hearing Justin rave about WPMU, I thought I would give it a shot. WP normally installs easy, so I expected the same. It would have except I wanted to install WPMU in a subdirectory. My Googling gave warnings against doing so, but I shrugged them off and forged ahead and tried to hack it to get it to work. Bad idea. It may be possible to do, and I got it to almost work, but certainly it doesn’t fall within the scope of famous 5 minutes install. So I’ve backed off and returned to the single user version.

• Added Collections. The index now contains more than 90,000 records from over 100 collections and 26 languages.
• Added Support for Harvesting via SQI/WSDL. Support for harvesting SQI/WSDL using Axis was added in order to harvest MERLOT via Araidne.
• Catalog Links. When providing recommendations if we have catalog links and direct links for resouces such as in the case of OER Commons and MERLOT both are provided.
• OAI Set Discovery – In order to get the names of collections from OER Commons the ability to discover the OAI Sets (collections) was added to the harvester.
• Lucene. The home brewed search and recommendation system was swapped out with Lucene. This makes for faster and better searching as well as faster indexing. The full range of query syntax supported by Lucene is now supported.
• Multi-Lingual. With Lucene in place OER Recommender can now support language-specific search and recommendation.
• Additional Metadata. Additional metadata was added to search and recommendation results: Descriptions, Authors, Date (metadata), Date (relevance was calculated).
• Home Page Cleanup. The home page was simplified by moving the Greasemonkey script and example resources and to a separate page.
• Search Results. Search results were modified to look similar to Google search results. In cases where the index contains both links to catalog pages and direct links for a resource, a Metadata link next to the title takes you to the catalog page. A “Related Resources” link is also provided next to each item in search results. This makes it easy to see recommendations.
• More Recommendations Page. The geek friendly page was replaced with a page that looks essentially like the search results page. A link to the original page containing details such as is included near the top of the page.
• Incremental Updates Support. The recommender was modified to support incrementally updating the indexes and recommendations without losing user data. It now runs every night, harvesting the collections, indexing and creating recommendations for new records. Once a week it re-runs all recommendations so that recommendations could be created that point at new records.
• Time on Page. Average time on page tracking was added and used to adapt the recommendations algorithm.
• Localized Interface. The main web pages were translated into Spanish, French, German, Japanese, Dutch, Russian, and Chinese using Google Translate (if you speak one of those languages and want to help the translation, feel free to send me fixes). Localization is supported via the swell Simple Localization rails plugin. The web app also auto-detects the language set in your web browser and sets that as the default search interface.

More on implementation later…

Other times problems occur because we broke something and we didn’t catch it in our testing. This is aggravated by browser incompatibilities. Because old hardware and software tend to hang around schools longer than other places, strange things show up. Right now I’m trying to track down some of those types of issues.

A few years ago when I was more actively developing the NLVM I used to keep old machines around so I could test old browsers. One of the reasons multiple machines were needed was that I couldn’t find an easy way to run multiple versions of Internet Explorer on the same machine. Yesterday I googled to see if there is anything new out there to help with this issue. I was pleasantly surprised to find a utility by tredosoft that allows you to install multiple versions of IE on your PC. Thanks tredosoft! Unfortunately after installing the multiple browsers, I’m still not able to see the reported problem even when running on the same browser.

Yesterday we released the OER Recommender system that I have worked on. There are still many things that could be added or tweaked, but it does something useful already so out the door it goes! I’m concerned that we are calling it a recommender as the “recommendations” it currently provides are not specific to a user, they are related resources generated via a content-based approach. The proposal and plans are to make it a recommender based on user profiles. See my previous post for details about where it is intended to go.

The Problem. The William and Flora Hewlett Foundation, the National Science Digital Library, and other large organizations have made large investments in the development of electronic resources that can be used for teaching, learning, and research. They are keen on finding ways to increase the impact of their investment. One way to do that is to create tools that make it easier for poeple to find resources that are useful to them. One approach to solving this problem are search and browsing tools that help people find good resources such as NSDL, OCW Finder, and of course Google. Recommender systems approach the problem by helping bringing resources to peoples’ attention without having them to go look for them specifically Google’s Adsense technology and Amazon’s recommendations are two of the most common examples. Some of the challenges with general search tools such as Google and even more focused ones such as NSDL’s main search is that they cast too wide a net and you get back resources do not match. On the other hand, while most collections of open courseware and digital libraries provide search, it is nice to be able to search across repositories at the same time to find the best resources wherever they reside.

The Vision. In creating the OER Recommender we set out to create a service that would help people find relevant open education resources. The first step in this was to create an automated process for clustering related resources. The hope is that presenting links to related resources to users when looking at a resource could help people stumble upon resources that are relevant to them even though they might have not been actively looking for them. We could tune this service by monitoring what resources people visit, share, rate, tag, and otherwise pay attention to. We could use this attention metadata to create profiles of what people’s interest are. The profiles could be used to push recommendations to people even when they are not browsing perhaps via email or other means.

Exploration. Since learning about it I have felt that the recommender would be one of the more interesting folksemantic tools to work on since I initially thought to pursue a latent semantic analysis approach. I had heard about LSA from my exposure to Andy Walker’s and Art Graesser’s work. I have also had a number of interesting discussions about clustering with Adele Cutler about her work on Random Forests. I got a hold of the Handbook of Latent Semantic Analysis and explored the LSA online resources I could find as well as the lsa module for R. After spending significant time reading and playing I came to the conclusion that it would likely more complex to implement and computationally expensive than I wanted. I also concluded that everything that I would need to do to implement a standard Term Vector Model approach could be used to later implement an LSA approach. During my exploration, I came across a number of useful online resources and books including Gerard Salton’s work.

An Explanation for My Mother. I’ve already been asked by Shelly to explain the implementation of the recommender in language that my mother would understand, so I’ll give that explanation first. We use the folksemantic feed harvester to gather information (metadata) about OERs into databases. For each pair of resources that are related enough, the recommender uses the titles, description, and tags to calculate a score indicating how related they are. Recommended resources are the ones scored to be the most similar. The similarity of two resources is based on an automated analysis of the words in their metadata. Got that Mom?

We use four phases to arrive at recommendations: (1) parse metadata, (2) calculate local term weights, (3) calculate global term weights, (4) calculate similarity scores.

Parse Metadata. Using a string tokenizer we break metadata text into terms. We throw away stop words (common words such as ‘and’, ‘is’, ‘of’ that do not add meaning). Next we convert terms into their stems using a common algorithm; for example ‘running’, ‘ran’, and ‘runs’ all get collapsed into ‘run’.

Calculate Local Term Weights. A local term weight is a measure of how important a word is for describing a document. The more frequently a word appears in a document, the more important it is… up to a point. Of course, where a word appears in a document is probably a good indicator of how important the word is as well. For example if a word appears in a title, it is probably more important than if it appears in the body. One more important factor to consider is document length (total number of terms in the document); all other things being equal, the longer a document is, the more times a term will appear in it. So we normalize term frequencies using the document length. One last issue relates to filler words.

To calculate local term weights OER Recommender uses a function that looks like this.

The x-axis represents term frequency and the y-axis represents the local term weight. As you can see, the more time a term appears in a document, the more weight it is given. However after reaching a threshold, it plateaus. I think this is especially important in situations where document creators might be trying to game the system. The formula used is:

ltw_i = 1 / (1 + (e^(.0044)dlen) .7^{(fti – 1)})

ltw_i is the local term weight for a given term in a document.

dlen is the total number of terms in the document.

ft_i is the number of times a given term appears in a document.

Note that the constants .0044 and .7 in the equation determine the shape of the curve. I chose those values based on their use in the MeSH system. As explained there, the values should be tuned to your data set using an empirical approach. I played with the parameters some and the values they used seemed fine, so I adopted them.

Calculate Global Term Weights. Global term weights are a measure of how important a word is for distinguishing documents within a collection. The more documents a word appears in, the less value it has for characterizing documents. For example, the term ‘USU’ appears in every metadata record for resources in USU’s OpenCourseware. As a result, it has no value for characterizing clustering resources. The term ‘USU’ becomes in a sense, a stop word, similar to those thrown away during the parse phase. To calculate the global term weight for a term, the number of documents that it appears in are counted as well as the total number of documents.

To calculate global term weights OER Recommender uses a function that looks like.

The x-axis represents the number of documents that a term appears in. The y-axis represents the global weight assigned to the term. The formula used is:

gtw_i = log (D/df_i)

gtw_i is the global term weight for term.

D is the total number of documents.

df_i is the number of documents that the term appears in.

See Mi lslita’s explanation of Term Vector Theory for details.

Calculate Similarity Scores. Once local term weights and global term weights are calculated, we are ready to create recommendations. To create recommendations for a document we first find all documents that have any of the same terms in it. It turns out that this can be a very large number of documents (e.g. 40,000 in our system). For each pairing of the document being considered and a document with matching terms we calculate a similarity score. Because calculating 40,000 scores can take a long time (about 15 seconds in our current system), we shorten the list to consider to 200. We do this by sorting the pairs according to the number of overlapping terms. To calculate the similarity score for a pair of documents, we sum over all of the terms that they share in common. The contribution from each term is a combination of the local term weight in the first document, the local term weight in the second document and the global term weight. Because in OER recommender, each feed can be considered a separate collection, we calculate global term weights for each of the feeds and use them in calculating the similarity score.

To calculate the contribution of an individual term to the similarity score, OER Recommender uses:

sst_i = (gtw_1i)(ltw_1i)(gtw_2i)(ltw_2i)

sst_i Is the contribution to the similarity score for a given term.

gtw_1i is the global term weight from the feed that the first document is in.

ltw_1i is the local term weight from the first document.

gtw_2i is the global term weight from the feed that the second document is in.

ltw_2i is the local term weight from the second document.

See NCBI’s explanation of how MeSH calculates Related Articles for details. Once OER Recommender has calculated similarity scores for the 200 documents, it sorts the documents by those scores and stores the top 10 in a database.

Displaying Recommendations. Now that the recommendations are stored in a database, displaying them to the user is straightforward. We have created a Greasmonkey script that requests recommendations for each web page that a user browses. If OER Recommender returns any, the script inserts HTML for the recommendations into the web page. The eduCommons team is building a plone tool so that anyone running eduCommons can turn on recommendations. That way users won’t have to have install the Greasemonkey script in order to see recommendations on the eduCommons website. The OER Recommender site publishes the XML format it returns so others could add recommendations into their websites.

Getting Resources into the Recommender. Currently OER Recommender only provides recommendations for URLs that it has metadata for, so in order to get recommendations, you need to register a metadata feed. If you have an OCW site that provides an RSS feed, you can do that at OCW Finder. If you have an OAI feed or an RSS feed for learning objects or other OERs, you can add your metadata by sending a feed title, URL, and display URL (home page of the repository) to oerrecommender AT cosl DOT usu DOT edu.

Future Directions. There are many things left to be done: (1) adapt recommendations by monitoring which recommended resources people click on and how long they stay at recommended resources, (2) provide links to more recommendations (so users can see more than the default 5), (3) provide a way for people to indicate when a recommendation doesn’t work, (4) provide a way for people to register and login, so they can receive personalized recommendations (perhaps via email), (5) create a process whereby recommendations can be updated without going through all the documents in the database, (6) add functionality for retrieving recommendations for arbitrary web pages (recommending on demand) perhaps via a bookmarklet, (7) create whatever Greasemonkey-like add on is supported on IE. We also plan to integrate the recommender with the aggregator and feed reader to provide recommended news items based on the attention people have previously paid to news articles.

Wow! I realize that this is a long post, but it is not a simple topic. Hopefully the explanation is useful.

The database. I’ve checked the database and found that we have nearly 2 million rss entries and about 350 thousand unique tags. We’ve indexed the entries so that we can access recent ones quickly. We haven’t indexed the entries for fast searching by tag.

Search was dog slow. Ozmozr’s search which is visible from most pages, takes entered terms and treats them as tags with which to search entries. Joining through the massive tag table to the even more massive entries table was taking forever (on the order of 45-60 seconds per query). We temporarily disabled entry searching until I poked around and found that we hadn’t created an index on tag names. Just putting an index on tag names reduced the query time to around 8 seconds. Much shorter, but still too long. So I reduced the complexity of the query. It may not be as powerful as it used to be, but now executes in under two seconds.

Aggregator’s 20 Java threads talking to postgres was stealing the CPU. We use a Java aggregator daemon that I wrote to harvest RSS feeds. We haven’t harvested in a while because of the problems we have experienced. When I fired it up today I found that it brought the site to its knees when it started doing its work. By monitoring the CPU I saw that the aggregator daemon was stealing all of the CPU leaving none for Rails to use to serve pages. By default the aggregator checks feeds every hour. When doing this it can use up to 20 threads. Each thread creates a connection to postgres. It was actually postgres that was stealing all of the CPU, but only because of how the Java daemon was talking to it. My quick fix is to limit the aggregator to 1 thread. I initially designed the aggregator to use many threads because it turns out that most of the time spent in harvesting is used up in waiting for web servers to respond. I guess I need to figure out another approach. I’m going to look to see if we can throttle how much CPU the Java threads (and corresponding Postgres processes) use.

Shrinking tag clouds are a fun idea but a pig to implement. Similar to how OCW Finder allows you to filter your browsing by selecting tags, we initially implemented the same idea in oz via a shrinking tag cloud. The idea was that as soon as you clicked on a tag in a cloud, in addition to the items being filtered, the tags in the resulting tag cloud would be filtered as well (shrinking the cloud). It turns out that this kind of query is horrendously expensive, at least the way we implemented it. It also turns out that users don’t seem to understand what is going on. As a result of this, we ripped the shrinking tag clouds out… almost. I just found an instance where it was left in. They are gone now.

There are more ugly queries to look at but I’ve gotten most queries down to under 4 seconds, which is still a long time, but at least the website doesn’t die. I’ll write more as I find it out.

As a note, the way I was able to easily identify the nasty queries was by going into the postgres config file (var/lib/pgsql/data/postgresql.conf) and setting the option log_min_duration_statement = 3000 (milliseconds). With this option set, every query that takes longer than three seconds is written to the postgres log file.

Summary

• Searching and Publishing – People come to the MTDL to find and share resources
• Overcoming Isolation – People come to the MTDL to help overcome isolation
• Discuss Development – Talk with others about the development of math software

Personas

• Teacher Developer
• Professional Developer
• Educational Researcher
• Inexperienced Developer
• Hobbyist Developer

Metaphor and Principles

• Workshop metaphor
• Design for multiple roles
• Design for multiple levels of expertise
• Provide activity indicators

I like the workshop metaphor, though I think that perhaps there are better. I can’t really discern the implication of designing for different roles and different levels of expertise. I lilke the idea of activity indicators. I realize that this is an area of recent interest throughout the field.