For more info and cool images check out the WISE project website.

Some of Technology for WISE was developed at Space Dynamics Laboratory here in Cache Valley where some of my neighbors work. Another fun connection for me is that the rocket that carried WISE into orbit was launched from Vandenberg Air Force base where my Dad was serving when I was born.

Here is some press that covered the award:

• Utah State Today – USU Researchers Awarded Governor’s Medals for Science and Technology
• USU College of Science – Three USU Scientists Receive Governor’s Science Medal
• Salt Lake Tribune – Herbert to honor recipients for contributions to science
• UtahPolicy.com – State Advisory Council For Science And Technology Announces Governor’s Science Medals For 2009

Here are some previous articles about the team as well:

• USU – Developed Virtual Math Tutor Aiding Students Worldwide
• USU – Winter Insights 2006

(1..100).each {|page| system("curl -o #{page}.html
   http://dmlcompetition.net/pligg/index.php?page=#{page}")}

Ruby rocks!

http://code.google.com/p/android/issues/detail?id=4784 (comment 35)

1) create a new gmail account on the computer.
2) call your cell phone with a different phone.
3) answer your cell phone then hit the back button and it will take you to the home
screen.
4) turn on wifi so it can do data and voice at the same time (remember that the phone
is still connected)
5) go to Settings -> Location & Security and disable lock pattern (you’ll need to
enter to correct pattern previously set)
6) go to Settings -> Accounts & Sync and click “Add Account” and add your newly
created Gmail account.
7) hang up the phone.
turn the phone back on, at the lockout screen, enter your new Gmail account info
and it should let you back in
9) once the phone is unlocked, you can go in setting and remove the newly added Gmail
account and keep the old one.

Google giveth, and Google taketh away, and then Google giveth again.

• Personal recommendations page – For each user, provide a personal recommendations page. Visually separate recommendations that they have already clicked on.
• Personal recommendations tool – Include a personal recommendations button on our folksemantic bar that when clicked on will display their recommendations in the right panel. Linking to the recommendations from that panel will refresh the content in the iframe (not do a full page refresh).
• Personal recommendations action link – Include a link to the user’s recommendations page on their dashboard.
• Inject recommendations into activity feeds – Whenever we generate new personal recommendations, inject them into a user’s activity feed that is displayed on their dashboard.
• Email personal recommendations – Email personal recommendations to users as often as they would like (controllable in their account settings).

• Identity feeds – RSS feeds that users register that represent their interests. For example, their blog or their delicious account.
• Clicks – The articles that the user clicks on.
• Shares – The articles that the user shares to others.
• Comments – Articles that the user comments on.
• Time on page – Amount of time that a user spends on an article before moving on.
• Searches – Searches the user executes.

Recommendation Assumptions

Some of our assumptions are:

• Semantic relatedness – The more semantically similar an article is to articles that a user has paid attention to, the more interesting to the user.
• Attention types – Different types of attention should be given different weights. For example, following a link to an article should not give it as much weight as writing the article.
• Attention details – The particulars of a given type of attention might make it more important than another attention of the same type. For example, if a person shares an article with 100 people, it might be reasonable to infer that it is more important than an article that they share with one person.
• Entry recency – The more recently an article has been added to the system, the more interesting to the user (they probably haven’t seen it before).
• Attention recency – The more recently a user has showed attention to an article, the more weight that should be given to it.
• Attention frequency – The more frequently a user has showed attention to an article, the more weight that should be given to it.

Stating these assumptions reminds me of the difference between relevance and certainty. So while an item that a user clicks on may be more relevant than an blog article they have written, it is harder to be certain of that. Our approach is to give the click less weight than the article.

Recommendation Score

Right now, we score articles using the formula:

(relevance)(attention type)(attention details)(attention recency)(article recency)

For all articles that a user has paid attention to, we score the 20 “related articles” using this algorithm; rank the scores and cache the top 20 (that the user hasn’t already clicked on) to recommend to the user. There are obvious weaknesses to this approach, but we are going to start there and see where to go next.

Possible Extensions / Improvements

We are considering:

• Collaborative Filtering, Bipartite Graph, and Discriminative Weight Algorithms – Putting into production, the algorithm that combines discriminative weights with a novel sparse matrix clustering method we conducted research on.
• Modeling user interests separately – Modeling users interests using multiple term vectors (one for each interest) by extracting vectors from closely related (clustered documents) that users have paid attention to.

David shared Evidence Based Design (not sure if this is what he was referring to) as a model. I shared Conditions of Learning – the idea that different types of learning outcomes should be taught differently, and Jim Cangelosi’s (forgive the flashing text) work on designing mathematics instruction for different types of learning outcomes. Interestingly he has advocated the idea of mini experiments as an approach for teachers to learn about and evolve learning.

ATE Central is an example of how a project in NSDL can influence other NSF programs. It brings all of the ATE resources in to one searchable portal. It builds the ATE brand and helps disseminate the projects. ATE is different than NSDL in that they focus on content development, industry connections, and the improvement of training and teaching for workforce development. ATE offers smaller grants and larger center grants. Example national, regional, and resource centers of excellence are geoTech, CARCAM, AgrowKnow. ATE Central has been funded for 1 year. They focus more on events than in other portals. This is partially because ATE focuses a lot on workshops including virtual. They create resource areas on ATE Central for each projects and centers. This has been a big deal to their projects to help them collaborate. ATE has a center that is funded just for evaluation. They send out a monthly update and are creating success stories. She showed videos of people that have found success of students that have benefited from ATE.

Linea Fletcher and Rachel is interested in the life of NSDL projects that continue beyond funding (are sustainable). They want to capture and share these stories. Another focus is on how to capture of evidence of impact across projects. They currently track 320 projects and aggregate and share it in interesting ways.

This work reminds me some of what Justin did with 51weeks – a platform for supporting communication around a conference and the other 51 weeks of the year.

Modeling is important because it is what scientists do. It is also a valuable way to interest and help students learn. You can do problems that are too hard or impossible to visualize, that you just don’t see if you don’t have the computational resources. “The difference between physics and math is that I have fun demos. In physics as opposed to math. We do have answer. It is what I measured”. The Falling Slinky Model: What happens to the bottom when it begins to fall? Physically, it is hard to see. The bottom doesn’t know it has dropped until it gets a message from the top of the slinky. Colliding Galaxy model. There is no way for someone to do this on paper.

Adoptable, adaptable, and understandable. They have to be modular, adaptable, visual, interactive, internationalization, quality control, easy to get to, vetted by other teachers, descriptions of how it has been used. Francisco Esquembre – creator of Easy Java Simulations. He says the most important characteristics are adoptable, adaptable, and understandable. This gives power to the people. This was my theory and excitement in my dissertation study. Give the teachers the tools, and they will create. My findings can be summarized by the 90-9-1 rule. 1% of teachers have the time, interest, and skills to create. Of course more powerful tools make it more possible for people to participate.

He showed an Inferior Ptolemaic Model of what it would be like if the earth was the center of the earth and Venus went around the earth. How could you tell which model is right? People can conjecture and explore.

He showed the Falling Loop Model. A loop that is falling through a magnetic field. Faraday’s law. It creates a current that opposes the falling. Other physicists can open (download) the model and modify it with minimal programming. His students can do this as well. This is similar (but more advanced) to my work in the eNLVM that allows users to configure and serialize applets. They connect EJS to the library to make it easy for people to find, discuss, and share their models. Their library allows users to rate, collect, relate, comment, sort, annotate, and share. Again, the themes of our Mellon work and the ODLMS shows its head. One of the contributors to Open Source Physics has written a text book for which their File Cabinet contains the resources aligned with the sections of the text book. They have build into EJS the ability to browse repositories of EJS models. There is a professor in Taiwan who has over EJS 150 models. Doug Brown, Wolfgang, and Lyle Barbato are some of the key people. A project called Tracker that generates video from EJS simulations. They have also done a Data Tool.

Their stats (Open Source Physics?) are 350 resources, 1700 users per week, 31,000 downloads per month. By comparison, the NLVM has about 110 applets and gets 40,000 users per day.

Jim Jenkins asks about how attribution and copyright works when you copy resources from one system to another. In Send2Wiki we address this by embedding it in the database. In eNLVM we track the source and attribute it in a status bar along the bottom.

nlvm^.5 nsdl^.3 math^.7 learners^.2 education^.8 “grand challenges”^.7

The advantage of identifying and modeling interests are that instead of modeling the average of a persons interests, you model specific interests. Then when you match people or documents, you match them with specific interests instead of the average of their interests. You can see the benefit of this when you consider what the average of math and bicycling is? Alex and I have discussed this issue, but have not yet implemented it. I’m interested to see any research that compares the two approaches.

So some of the questions related to modeling a learner using term vectors are:

• How many interests you should model?
• How do you discover the interests?
• How many terms should you include in each term vector?
• How do you do modeling and recommendations efficiently?

In Ed Fox’s presentation we are listening to Peter Brusilovsky talk via Second Life about the kinds of data that they use to model users including: search, tagging, comments, resources they have created including paths (Walden’s Paths), collaborative recommendations, and social navigation (guides users to most active resources). They uses these models to provide recommendation of content and other users. CUMULATE is their group modeling framework. They are tracking of browsing and problem solving. Steve demonstrated a bunch of tools they have in Second Life for Ensemble.

We are addressing similar approaches in Folksemantic. “Systems support personalization by adapting functionality to individuals, by allowing users to customize the system, and by supporting human interaction inside of the system.” We allow people to register their bookmark, blog and other feeds. We also will be using click, search, comment, and share data to model users and provide recommendations.

Dr. De Strulle is talking about the report Fostering Learning in the Networked World: The Cyberlearning Opportunity and Challenge.

Highlights of the report:

• Crossdisciplinary approach – use
• Platform perspective – adopt & integrate with existing platforms – support
• Learning across the lifespan – achieve decentralization while retaining what we know about learning and education
• Promote OER
• Sustain innovation – build on previous work so that it has life beyond its own project

Cyberlearning – learning mediated by technology (formal ed, informal ed, brought together by networked computing and ICT). Think about life long learning.

We need to think about redistributing learning over time. Cyberlearning is not restricted by the constraints of schools. Not limited by time, location, formal-informal boundaries.

Arlene comes from a Virtual Reality background from the military and illustrates cyberlearning through examples. She’s using examples of virtual worlds, earth systems sensors providing real time data. One example shows a person explore a 3D model of climate change including historical data.

Example: interacting with a dynamic model of a virus as one million atoms interacting with each other.

Haptic table a tool that lets people who can’t see, you can feel the images. Assistive technologies allow people with learning challenges to participate. Virtual laboratory allow people to practice surgery using remote surgery tools. This allows people to practice procedures repeatedly without the consequencies of real mistakes. The future workforce will require that people have skills to use technology. University of Pennsylvania. The GLORIAD network is an optical network that supports cooperation between scientists, educators, and students. It is an example of cross-institutional networks of faculty and students through the use of collaboratories. This helps faculty members focus on educational issues. Science museum virtual reality allows students to become emerged. This is essential to create cognitive presence in the learning environment. Game-based learning is being used to train our troops. It affords tremendous transfer of knowledge and individualization. Research is increasingly showing the power of effectively designed game based instruction. Game based learning increases public engagement in science and policy. Next to LaGuardia airport there is a plot of land that is set aside for a park. They have created a virtual version of Second Life and have invited the community to help design the park in that environment. Part of the design decision is the discussion and learning about the science of water and land related to land use. New mobile small devices have been created that allow data to be gathered in remote situations and provide access to STEM content. Mobile technologies are a democratic device because they allow everyone to access to all kinds of knowledge and to participate in the discussion. It allows people to learn information about their very location, for example the tree that they are looking at. It also allows remote communities to easily communicate (e.g. classrooms around the world). Adler Planetarium lets students look through the best telescopes in the world in real time and report them to the scientists. Technology based learning allows learning and teaching to be tracked and evaluated. The use of avatars in virtual worlds is an area of major research. She pointed us to additional reports: Cyberinfrastructure for Education Workshop Series, NSF Cyberinfrastructure Vision for the 21st Century.

What surprises did you encounter implementing cyberlearning?

Cathy Manduca – Pedagogy in action. Implementation in the Geoscientists. Implementation as a library. The goal is to help teachers and faculty to make better choices about education. If teachers and faculty had more information and examples about methods in subjects they teach, they would make better decisions about which methods to use and the barriers to adoption would be lowered. They created a catalog of teaching methods connected to examples. That was widely used by lots of people, so they created a service structured as a library. People can draw from the library to embed them in their websites and contribute content. Tracking demonstrates people are using the resources. Interviews give evidence that the site has given them new methods and confidence to try them. The resources speaks directly to teachers and supports adoption of the technology. Pedagogies and actions. Their project highlights cyberlearning comment such as teaching with data, models, visualizations, and games. You need to treat it as learning (remember all of the standard instructional design issues). The design takes advantage of teacher motivation PBL where the problem is what I’m going to teach today and collaborative allowing. The innovation have been able to allow teachers to author web pages. Cloning allows you to copy content into your website

David Porcello (SMILE Pathway) targets informal education of science and math. Learning Sciences and Informal Environments, people, places and. It talks about 6 strands of learning: learners experiencing, exploring, participating, reflecting about science. Recommendations include: learning objects should have learning goals, be interactive, have multiple ways to approach and use iterative design. Front line staff need support to actively discuss questions for diverse cultural views. SMILE is a central place for hands on science activities. NSF is building a lot of tools that include lots of personal content, mobile access, and widgets. It sounds a lot like what we are proposing for the ODLMS. SMILE is creating tools and services to support these in informal learning. They want to encourage people to use the SMILE tools. Focus on the educator and their needs.

Kim Lightle – Middle School 2 portal patheways. Teachers, students, and policy that affects teaching at the middle level. At Ohio State, Serena Nair is working with students to design learning environments for students. Mary Henton is looking at the policy of how to change what happens in middle schools. At the MS2 project they have built a portal for teachers. They are using a Ning based platform to connect content important to for teachers to know about in a way that it is all based in the technology. As teachers are learning about systems knowledge they are in wikis, in forums, and blogging. They are becoming part of a virtual learning environment for teachers. Surprises: Kim is not convinced that the “Digital Native” actually exists. Kids can buy shoes. Example: many 21 year old teachers didn’t know . Architecture of participation. If you build it, they might not come. They are putting content in lots of places to people will find it: Curriki, Indigo. “We are “. Object centered sociality – when you build a social network, you have to have content to build it around. The more complex the object, the more gravity it has and the more discuss. They have 570 registered members. Objects that have attracted attention are about integrating technology. It is interesting what people are looking for. The average age of registered members has 15 years of teaching experience. We should focus on second stage teachers. That is where we should focus (teachers who have figured out how to manage and are saying, I need something more). People found about the portal through friends primarily. One size doesn’t fit all. CBANS – Concerns based adoption model helps you understand what the needs of teachers are at a given stage. A social network allows differentiation by allowing people at each level to congregate and help each other. Book: Daniel Pink a whole new mind. He talks about symphony and about stem. We should focus on creative aspects of stem, not recal. Design for the future, not the past.

Bruce Mason, Director of Compadre. It is a service organization for teachers and learners in physics and astronomy. They try to bring people together to learn and find new methods. One of the biggest challenges of running Compadre. He wears 3 hats: Faculty member – don’t like change, showing them really cool stuff they can do is cool, but it will take a lot of work to get them into their classroom. Developer – he helps develop resources, faculty members evidence that approaches work and the materials and plans to easily fit new resources into what they are doing. We need to bring the results of past educational research into the world of cyberlearning. A community of educators that focus on the learning is powerful. The idea of a platform is a big challenge. Just last week 2 publishers came with learning platforms, but they already have Desire2Learn at their college. NSDL member – we really need to provide these services, not just a library and content. More important is that we show others how to develop content and use the tools. This should part of our sustainability model.

Questions: The cost of higher ed has multiplied. New technologies are being laid on top of them all and we are being charged for them all. How will cyberlearning address this? The open opportunities to do new things rather than replacing or substituting. It is far more expensive to not teach well. How do you evaluate student learning while technology is changing so rapidly? Use stable measures of learning consistently measure each technology. Teacher confidence is impacted heavily by reading websites. What does participation mean? What are they getting out of the experience?

Magnetic Sculpture kit from Collin Cunningham on Vimeo.

Or at least watch other people do, and think how cool it would be to do

Today I came across the STEM Challenges website, a new site created by STEM Net in the UK:

A new initiative to inspire the next generation of British scientists and engineers was launched today (Monday 12 October 2009) on the Olympic Park in London. All secondary schools in the UK are being urged to take part in a series of ten STEM Challenges which encourage pupils to explore the challenges involved in delivering the London 2012 Olympic and Paralympic Games and tackle them using science, technology, engineering and maths (STEM) skills.

This sounds really cool and aligned with my idea for stem challenges. Centering challenges around something as interesting as the Olympics, creating a public forum for sharing solutions, and rewarding responses.

Each term there will be a regional winner of the Challenge, chosen by a panel of STEM Ambassadors and online voting by all of the participating schools. Each of the ten regional finalist teams will then go on to compete at national level for a change to win an ‘experience’ prize, related to the Olympics.

I’m interested to see what comes of this. I think similar competitions could be centered around the grand challenges.

1. Dug up my old copy of Visual Studio 6 CDs and installed the command line utilities. Apparently, not just any version will do; you have to be in sync with the version used to compile ruby.
2. Added VS6’s bin directories to my Windows path. Default install locations are:
   C:\Program Files\Microsoft Visual Studio\VC98\Bin;C:\Program Files\Microsoft Visual Studio\Common\MSDev98\Bin;

   Alternatively, you can get a command prompt in  C:\Program Files\Microsoft Visual Studio\VC98\Bin and run VCVARS32.BAT to add those directories to your path.

3. Added some typedefs and a function to C:\Program Files\Microsoft Visual Studio\VC98\Include\SYS\TYPES.H that are not available on Windows:

   #ifndef _UINT_T_DEFINED
   typedef unsigned char  u_int8_t;
   typedef unsigned short u_int16_t;
   typedef unsigned int   u_int32_t;
   typedef unsigned __int64 u_int64_t;
   #define _UINT_T_DEFINED
   #endif

   #ifndef snprintf
   #define snprintf _snprintf
   #endif

Now I can do:

gem install bcrypt-ruby

I get the “Building native extensions. This could take a while…” message and it works!

I’m happy once again. I’m satisfied. And I didn’t even have to change religions.

In my part of our presentation I plan to talk about Teacher Training and Curriculum development. Here are some links I plan to use / reference:

• Overview of the eNLVM
• eNLVM eModules
• Curriculum development / teacher training case study
• How to rite good applets
• eModule research materials
• Transformation of Functions (example observations)
• Probability and Relative Frequency (example of simulations)
• What’s My Line (example of reinforcing group discussion with hands on experience)
• Patterns, Relations, and Functions (example of adapting by adding questions)
• Digging Dirt (example of open-ended, manipulative supported, grounded instruction)
• THEORY FOR AUTHORING TOOLS THAT SUPPORT TEACHER ADAPTATION OF MATHLETS
• Presentation on the eNLVM
• When Teachers Reuse and Remix Interactive Online Resources
• Interesting Interactive Math Websites

Teacher Training. A process I have used with teachers:

1. Demo a lesson
2. Discuss / critique the lesson together based on a rubric
3. Have teachers observe students while the lesson is taught
4. Debrief the experience
5. Revise the lesson
6. Reteach

Training Goals. My goals are for teachers to learn:

• Content knowledge
• Instructional strategies
• Familiarity with using high quality materials
• Where to find high quality materials
• Material evaluation / selection
• Help them enter a community
• Implementation issues
• Barriers / workarounds
• How to adapt existing resources
• How to evaluate student learning
• Processes for continuous improvement

Developing Interactive Math Curriculum

When developing curriculum, one of the first thing designed are the elements and structures of design and the processes of design, whether they be ad-hoc or structured. Curriculum design structures include time, learning outcomes, content models, instructional strategies, messages, representations, and media elements such as pages, displays and controls. Important design activities include:

• analyzing content,
• identifying desired learning outcomes,
• recognizing common student errors and difficulties,
• developing an overarching design,
• using design tools that allow for quick inexpensive prototyping and iteration,
• putting off expensive development as long as possible,
• using modular approaches,
• expert review,
• testing with real users,
• iteration

Considerations for Designing Interactive Math Curriculum

The design of interactive math curriculum should take into account the affordances that the medium can offer. These include efficient rich, dynamic, and linked representations, exploration, simulation of physically in-accessible situations and events, guided practice, immediate feedback, easy revision, recording sharing and replay, collaboration at a distance, linkages to real-time data, data sampling, and complex computation.

Contexts of Use

• Group presentation – projection, activity, discussion, worksheet
• Classroom station
• Computer lab (self-paced, pairs)
• Home work (self-paced)

Three Levels of Adaptation

• Activity instructions (via web or worksheet)
• Problem sequence (via web or instructions)
• Virtual manipulative (via configuration or code)

Multi-Disciplinary Teams

• Mathematician – Make sure the concepts / content / terminology / representations are correct
• Instructional Designer – Translate ideas into concrete technology designs
• Programmer – Develop virtual manipulatives
• Educator – Instructional sequences, strategies and types of activities
• Classroom Teacher – Access to students, anticipate student responses, guide implementation

Some Observations

• When an interactive model is on the screen, students ignore the text
• Text must be short and to the point
• Questions that require responses can help focus attention
• Transitions (going to lab, setting up equipment, getting people started, etc) waste time and need to be taken into account when designing experiences
• Always test out exact usage before going live and check again in the morning
• Developing virtual manipulatives is expensive
• Leverage existing manipulatives
• Choose areas where the impact will be greatest
• Teachers are much more likely to adapt than create activities

After morning hikes we swam in the lake which was surprisingly pleasant.

The older kids and I explored polygamy cave.

My Dad and I backpacked into White Pine lake where we ran into beaver. One beaver (I’m assuming the mother) made quite a show repeatedly slapping her tail on the water.

Seth made a compass course for those of us who are directionally challenged.

Good times. Now get back to work!

This vision of providing tools for teachers to create online content is similar to what I envisioned for my dissertation work which led to the eNLVM. My eyes were soon opened to the fact that most teachers do not have the time or skill to create online content, especially from scratch. I suggested to Turadg that if he wanted to encourage better and more problem submissions that they could provide example problems from which teachers could base similar problems. I also pointed out that there is already a massive supply of math problems in textbooks that could be tapped. He and others present mentioned concerns about copyright. To me, this is not a problem. By looking at a math problem you can extract the essence of the problem or it’s “problem type” and use that to easily generate many more of the same type of problem with different cover stories and values. Of course, until you solve a problem it can be difficult to know that the problem has similar solution structure as another. This is the basis for a project I would like to do some day: a library of math problem generators coupled with math test generators that leverage the problem generators and their alignments with standards and textbooks.

Ido presented a study that measured metacognition, specifically help seeking behavior. He began by flaming simple recall as a learning outcome, showing the example of the YouTube video of the child who can point to the names of the countries that her parents name. He did this probably because a PostDoc sitting in the presentation focuses on fact learning (Chinese). Ido’s study compared a new measure to the “assistment” measure used by Carnegie Learning’s tutors as predictors of learning. It seems to me that they pretty much measured the same things, and both are somewhat good predictors of learning.

This is an interesting area. Information seeking is a metacognitive skill: knowing when you know enough to proceed and when you don’t. Having the will to not take the lazy out when you know enough. Knowing where to go to find information you need. The picture is actually much more complex than this. When you are first learning something, or solving a novel problem, it is expected that you would need more information. Better problem solvers and learners recognize this and seek the needed information effectively. As you learn more in an area, you don’t need as much help and so you should stop relying on it. In a situation where making a wrong decision could cause someone to die, the good problem solver relies on additional sources to verify that what they think is a good decision is actually one .

Measuring information seeking behavior is an important way to measure problem solving ability. Unfortunately, school, and even worse, school testing situations, are very unnatural problem solving situations where information seeking behavior is called cheating

The Masters in Entertainment Technology (MET) program brings together people from multiple disciplines to work together on intensive entertainment technology projects. He said that this is the head fake; the MET program is designed to help people learn to communicate together and work as a team. It only made me wish I could be back in school doing their program! Now if I can just convince Julianne

Drew explained that they Randy likes research and had pretty much moved Alice and his group back to the CMU campus for some time even before the last lecture. While Alice is a good tool for introducing programming to novices, it is not the tool of choice for developing production quality 3D games. The ETC now uses Panda3D heavily with Jesse Schell one of the primary contributors on staff. Panda3D is the open-source, python programmable, game engine used by Disney to develop games such as Toontown and Pirates of the Caribbean.

Very cool! I could have kicked myself when standing outside waiting for the bus I realized that I failed to take out my camera during my tour of the ETC.

I resonate with David Yaron’s concern that we teach the wrong things in High School and introductory College courses, focusing on teaching students to follow mathematical procedures rather gain a conceptual understanding of the content. I shared my theory a cause:

1. We emphasize in our teaching what we test
2. We test what is easy to test
3. Testing simple recall and procedure following is easy
4. We emphasize simple recall and procedure following in our tests
   
5. We emphasize simple recall and procedure following in our teaching
   

The remedy is to develop automated measures of higher level thinking: conceptual understanding, problem solving, design. He agreed in part but challenged that we don’t know or agree what problem solving is and have an even harder time measuring it. I agree in part, but think we do know something and can begin heading in the direction of trying to measure problem solving and higher level thinking.

Problem solving is what we do when we don’t know what to do.

Problem solving involves recognizing and defining a problem, searching for relevant information, forming appropriate subgoals, selecting appropriate strategies for accomplishing subgoals, executing procedures, monitoring progress and redirecting efforts when appropriate, recognizing when satisfactory solution has been arrived at, and interpreting the results of problem solving efforts. Interestingly this relates to the conversations I had at the PSLC later in my trip.

Yaron, who sits on the AP Chemistry board, also indicates that even if we had good automated measures of higher level thinking it would take a long time for them to be widely adopted and that a revision approach is more likely to succeed than a revolution approach.

JCDL 2008 trip continued: I talked with Bryan Chapman about aligning content with standards (see his paper Exploring Educational Standard Alignment: In Search of ‘Relevance’). He pointed me to the CNLP’s Curriculum Assignment Tool and to the Teacher’s Domain cross walking service as potential sources of tools and providers of standards alignment. I have the idea that if we could create a backbone set of standards that was as superset of all of the state standards and then align content with that set of standards, then it would make it dramatically easier to provide answers to a teacher’s query for resources relevant to what they are teaching.

Bryan believes that it is nearly impossible to develop effective crosswalks between the standards. Different standards focus on different levels of detail and address different levels of outcome. They use the same words to mean different things and some standards assume the context of their location in an hierarchy rather than restating it. This still seems like an interesting problem to try to solve, maybe even something that recommender technology could be applied to.

Tuesday morning I got in a run along the Allegheny and saw up close a few of the 466 bridges of Pittsburgh as well as the inclines.

After years of keeping them in boxes, I recently got out my University Physics and Math books and put them on my shelves. Just getting them out inspired me . This morning while testing the Open Yale Courses feed that I had added to OER Recommender, I ran across the Fundamentals of Physics course. This led me to R. Shankar’s home page, where I noticed an old friend, Shankar’s book that I used in my quantum mechanics class at the University of Utah. I also smiled to see him list his

Most important contribution to physics

“Discovered a small parameter that justifies most calculations performed in physics: 1/ego, where ego is the author’s ego.”

Very clever. It is cool to see a self-deprecating physicist. The description of his other book shown there: Basic Training in Mathematics rang true to my experience. My love and interest while studying at the UofU was always Physics, but I ended up taking so much math that I decided to go ahead and major in Math as well. While doing so, I found that almost everything I learned in my Math classes I had previously learned in my Physics classes .

Problem solving is what you do when you don’t know what to do. Problem solving requires recognizing and defining the problem, selecting an approach, breaking the problem down into sub-problems, selecting procedures for solving those sub-problems, executing those procedures, evaluating and diagnosing progress, recognizing when a solution is satisfactory, and interpreting results. Note, if practice makes perfect, we better give students opportunities practice in all of these aspects of problem solving, not just simple recall and algorithmic procedures.

Common wisdom says that we should wait until people have developed the basics before we ask them to solve problems and do higher level thinking. I reject that notion. Higher level thinking may not so much be “higher level” as it is “different level”. Kids at the youngest ages can and need to be given opportunities to engage in real problem solving. Maybe, part of why kids learn to hate math is because we spend so much time focusing on “repeat what I just said” and “do what I just did”, to the exclusion of authentic problem solving.

“Theories are like toothbrushes, everyone has a theory, but no one wants to use someone else’s theory”

This is true within a given field such as artificial intelligence / cognitive science in which the PSLC work is based. It is even more true if you look across multiple fields such as education (teacher preparation), cognitive science, instructional design, and math education. Each of these fields have something to say about learning and teaching math, but the languages of their literatures are as different as English, Urdu, Chinese, and Russian.

I recently presented on the PSLC and this framework at a recent USU Math & Stat Journal Club meeting (see my PSLC Theoretical Framework Trailfire Trail). In our discussion it was brought up that sometimes teachers object to the type of instruction that is typical of the PSLC model tracing intelligent tutors because it is not open-ended or exploratory. My reactions to this criticism is that the tutors are not meant to replace all instruction. This type of instruction is very effective at teaching procedural (algorithmic) skills consisting of a sequence of steps. It turns out that a lot of the math what we expect middle school and high school students to learn (and demonstrate on standardized tests) is of this nature.

1. In primary grades the major emphasis is on recall of basic math facts (e.g. 3 + 5).
2. Many kids aren’t wired for simple recall (they aren’t good at memorizing). I’m not.
3. Commonly used instructional approaches don’t give the immediate feedback essential to developing simple recall (e.g. do this worksheet which will be graded later today or tomorrow).
4. Because of the heavy emphasis on simple recall kids perceive that it represents all of math.
5. This Simon-Says approach to math instruction often continues throughout a student’s public education.
6. By the time students are given the chance to engage in other types of mathematical thinking, they have developed a negative self-perception that pre-disposes them against trying or learning math.

While attending the 2008 Mountain West Ruby Conference and needing something to hack on I decided to take a crack at the project. I already hacked Google Translate for Send2Wiki so I figured it would be a snap to do for this project. My plan was to take the search text, run it through the translator for each of the languages to search, then pass the translated queries off to the Google search sites for each of the languages and then pass those pages through Google translate to get English versions of the pages. I soon found that Google has already done most of the work for me with their cross-language search.

The only thing cross-language search doesn’t do for me is collate all of the language results into a single results page. You can only search for results in a single targeted language. Anyway, between sessions (a coder has always got to brag about how fast he can work right I threw together a Multilingual Google Search Mashup that does the job. As I put it together, a couple of things almost immediately stood out:

• Wikipedia owns the top hit slot for many searches. Because their pages are essentially equivalent in the different languages, listing that entry for each of the languages isn’t especially useful.
• Interleaving search results is difficult. Rather than try to figure out an intelligent way to order in real-time the search results from the various languages, I just give the first two from each language and provide a language for getting more. I’ve got ideas for interleaving results, but none of them are too easy. Notice also that I haven’t included English in the search, which is probably where the most relevant pages will actually come from.

These issues makes me wonder if a different approach would be preferrable. Perhaps Google could annotate search results with relevant pages in different languages. This also makes me think about Google’s search result ordering. Google search results appear to be determinative (if you execute the same search twice, the same item will show up at the top of the list). While this may be what we have come to expect, my experience with writing OER Recommender makes me believe that it isn’t necessarily the best or the fairest thing to do. When ranking pages it is often the case that the scores of the top two or even 10 pages are statistically indistinguishable. So why should the one that happens to have a .00000001% higher score always show up first. My approach with was to identify a strata of rankings for those “highest ranked pages” that are virtually indistinguishable, I randomize the order. This seems fairer since it is quite natural for users to click on the first item in on a search results page, thus biasing it to become more and more popular.

With slicehost I get a VPS for $20 / mo. I’ve never used a VPS before but have run my own web servers, so it seems worth a shot. In the process of getting it set up I’ve learned more about DNS, iptables, fastcgi, etc than before, so that is fun, but also a time drain. Anyway, now I’ve got the access I need to set up Send2Wiki and run rails apps.

After hearing Justin rave about WPMU, I thought I would give it a shot. WP normally installs easy, so I expected the same. It would have except I wanted to install WPMU in a subdirectory. My Googling gave warnings against doing so, but I shrugged them off and forged ahead and tried to hack it to get it to work. Bad idea. It may be possible to do, and I got it to almost work, but certainly it doesn’t fall within the scope of famous 5 minutes install. So I’ve backed off and returned to the single user version.

On a related note, Patricia Moyer-Packenham , the former MEC Director recently accepted a position in the USU College of Education. I’m excited that she will be close by and hope to work with her.

This past week I began updating the NLVM Application to support switching between multiple languages. On the screen that displays applets I added a JComboBox below the applet containing a list of languages that a user can select. Everything seemed to work fine until I noticed that once I clicked on the list box the applets stopped updating. After much pain and suffering and digging through the Swing source code I figured out the problem:

When JComboBox gets focus it (via javax.swing.plaf.basic.BasicPopupMenuUI.grabContainer) calls addMouseListener, addMouseMotionListener, and addMouseWheelListener on the applet, which are implemented in java.awt.Component. This is where the problems begin. All of those methods contain an innocuous line:

newEventsOnly = true;

There is the evil. The effect of this flag being set bears fruit in java.awt.Component.dispatchEventImpl where there is a set of nested if statements that begins:

if (newEventsOnly) {
if (eventEnabled(e)) {
processEvent(e);
}
}
...

The result is that none of the old AWT events (that the applets depend on) are sent after one of those listeners is added and the flag is set. It turns out, there are more methods that set the flag as well: addComponentListener, addFocusListener, addHierarchyListener, addHierarchyBoundsListener, addKeyListener, and addInputMethodListener, but the JComboBox never calls those methods.

My simple hack was to override those addXXXListener methods in my base class to prevent the calls from reaching down into Component where the flag gets set. I did this by writing a new J14Applet base class which extends java.applet.Applet and extending my former J10Applet base class from J14Applet instead of Applet. I’m sure this may have side effects but it seems to work fine for now.

As to why JComboBox was adding those listeners in the first place? I think it is so it can handle mouse messages when it has the focus but the mouse is not over the component, so I have defeated that behavior, but I can live with it. It is not as clean a hack as I would like because I want to be able to compile the applets for running on machines that don’t have a 1.4+ JVM. It would be nice to be able to have my Swing app dynamically add those methods like you can do in Ruby, but I don’t quickly see how to do that. For now, I will just change what J10Applet extends depending on what target I’m compiling for.

• Added Collections. The index now contains more than 90,000 records from over 100 collections and 26 languages.
• Added Support for Harvesting via SQI/WSDL. Support for harvesting SQI/WSDL using Axis was added in order to harvest MERLOT via Araidne.
• Catalog Links. When providing recommendations if we have catalog links and direct links for resouces such as in the case of OER Commons and MERLOT both are provided.
• OAI Set Discovery – In order to get the names of collections from OER Commons the ability to discover the OAI Sets (collections) was added to the harvester.
• Lucene. The home brewed search and recommendation system was swapped out with Lucene. This makes for faster and better searching as well as faster indexing. The full range of query syntax supported by Lucene is now supported.
• Multi-Lingual. With Lucene in place OER Recommender can now support language-specific search and recommendation.
• Additional Metadata. Additional metadata was added to search and recommendation results: Descriptions, Authors, Date (metadata), Date (relevance was calculated).
• Home Page Cleanup. The home page was simplified by moving the Greasemonkey script and example resources and to a separate page.
• Search Results. Search results were modified to look similar to Google search results. In cases where the index contains both links to catalog pages and direct links for a resource, a Metadata link next to the title takes you to the catalog page. A “Related Resources” link is also provided next to each item in search results. This makes it easy to see recommendations.
• More Recommendations Page. The geek friendly page was replaced with a page that looks essentially like the search results page. A link to the original page containing details such as is included near the top of the page.
• Incremental Updates Support. The recommender was modified to support incrementally updating the indexes and recommendations without losing user data. It now runs every night, harvesting the collections, indexing and creating recommendations for new records. Once a week it re-runs all recommendations so that recommendations could be created that point at new records.
• Time on Page. Average time on page tracking was added and used to adapt the recommendations algorithm.
• Localized Interface. The main web pages were translated into Spanish, French, German, Japanese, Dutch, Russian, and Chinese using Google Translate (if you speak one of those languages and want to help the translation, feel free to send me fixes). Localization is supported via the swell Simple Localization rails plugin. The web app also auto-detects the language set in your web browser and sets that as the default search interface.

More on implementation later…

Other times problems occur because we broke something and we didn’t catch it in our testing. This is aggravated by browser incompatibilities. Because old hardware and software tend to hang around schools longer than other places, strange things show up. Right now I’m trying to track down some of those types of issues.

A few years ago when I was more actively developing the NLVM I used to keep old machines around so I could test old browsers. One of the reasons multiple machines were needed was that I couldn’t find an easy way to run multiple versions of Internet Explorer on the same machine. Yesterday I googled to see if there is anything new out there to help with this issue. I was pleasantly surprised to find a utility by tredosoft that allows you to install multiple versions of IE on your PC. Thanks tredosoft! Unfortunately after installing the multiple browsers, I’m still not able to see the reported problem even when running on the same browser.

Yesterday we released the OER Recommender system that I have worked on. There are still many things that could be added or tweaked, but it does something useful already so out the door it goes! I’m concerned that we are calling it a recommender as the “recommendations” it currently provides are not specific to a user, they are related resources generated via a content-based approach. The proposal and plans are to make it a recommender based on user profiles. See my previous post for details about where it is intended to go.

The Problem. The William and Flora Hewlett Foundation, the National Science Digital Library, and other large organizations have made large investments in the development of electronic resources that can be used for teaching, learning, and research. They are keen on finding ways to increase the impact of their investment. One way to do that is to create tools that make it easier for poeple to find resources that are useful to them. One approach to solving this problem are search and browsing tools that help people find good resources such as NSDL, OCW Finder, and of course Google. Recommender systems approach the problem by helping bringing resources to peoples’ attention without having them to go look for them specifically Google’s Adsense technology and Amazon’s recommendations are two of the most common examples. Some of the challenges with general search tools such as Google and even more focused ones such as NSDL’s main search is that they cast too wide a net and you get back resources do not match. On the other hand, while most collections of open courseware and digital libraries provide search, it is nice to be able to search across repositories at the same time to find the best resources wherever they reside.

The Vision. In creating the OER Recommender we set out to create a service that would help people find relevant open education resources. The first step in this was to create an automated process for clustering related resources. The hope is that presenting links to related resources to users when looking at a resource could help people stumble upon resources that are relevant to them even though they might have not been actively looking for them. We could tune this service by monitoring what resources people visit, share, rate, tag, and otherwise pay attention to. We could use this attention metadata to create profiles of what people’s interest are. The profiles could be used to push recommendations to people even when they are not browsing perhaps via email or other means.

Exploration. Since learning about it I have felt that the recommender would be one of the more interesting folksemantic tools to work on since I initially thought to pursue a latent semantic analysis approach. I had heard about LSA from my exposure to Andy Walker’s and Art Graesser’s work. I have also had a number of interesting discussions about clustering with Adele Cutler about her work on Random Forests. I got a hold of the Handbook of Latent Semantic Analysis and explored the LSA online resources I could find as well as the lsa module for R. After spending significant time reading and playing I came to the conclusion that it would likely more complex to implement and computationally expensive than I wanted. I also concluded that everything that I would need to do to implement a standard Term Vector Model approach could be used to later implement an LSA approach. During my exploration, I came across a number of useful online resources and books including Gerard Salton’s work.

An Explanation for My Mother. I’ve already been asked by Shelly to explain the implementation of the recommender in language that my mother would understand, so I’ll give that explanation first. We use the folksemantic feed harvester to gather information (metadata) about OERs into databases. For each pair of resources that are related enough, the recommender uses the titles, description, and tags to calculate a score indicating how related they are. Recommended resources are the ones scored to be the most similar. The similarity of two resources is based on an automated analysis of the words in their metadata. Got that Mom?

We use four phases to arrive at recommendations: (1) parse metadata, (2) calculate local term weights, (3) calculate global term weights, (4) calculate similarity scores.

Parse Metadata. Using a string tokenizer we break metadata text into terms. We throw away stop words (common words such as ‘and’, ‘is’, ‘of’ that do not add meaning). Next we convert terms into their stems using a common algorithm; for example ‘running’, ‘ran’, and ‘runs’ all get collapsed into ‘run’.

Calculate Local Term Weights. A local term weight is a measure of how important a word is for describing a document. The more frequently a word appears in a document, the more important it is… up to a point. Of course, where a word appears in a document is probably a good indicator of how important the word is as well. For example if a word appears in a title, it is probably more important than if it appears in the body. One more important factor to consider is document length (total number of terms in the document); all other things being equal, the longer a document is, the more times a term will appear in it. So we normalize term frequencies using the document length. One last issue relates to filler words.

To calculate local term weights OER Recommender uses a function that looks like this.

The x-axis represents term frequency and the y-axis represents the local term weight. As you can see, the more time a term appears in a document, the more weight it is given. However after reaching a threshold, it plateaus. I think this is especially important in situations where document creators might be trying to game the system. The formula used is:

ltw_i = 1 / (1 + (e^(.0044)dlen) .7^{(fti – 1)})

ltw_i is the local term weight for a given term in a document.

dlen is the total number of terms in the document.

ft_i is the number of times a given term appears in a document.

Note that the constants .0044 and .7 in the equation determine the shape of the curve. I chose those values based on their use in the MeSH system. As explained there, the values should be tuned to your data set using an empirical approach. I played with the parameters some and the values they used seemed fine, so I adopted them.

Calculate Global Term Weights. Global term weights are a measure of how important a word is for distinguishing documents within a collection. The more documents a word appears in, the less value it has for characterizing documents. For example, the term ‘USU’ appears in every metadata record for resources in USU’s OpenCourseware. As a result, it has no value for characterizing clustering resources. The term ‘USU’ becomes in a sense, a stop word, similar to those thrown away during the parse phase. To calculate the global term weight for a term, the number of documents that it appears in are counted as well as the total number of documents.

To calculate global term weights OER Recommender uses a function that looks like.

The x-axis represents the number of documents that a term appears in. The y-axis represents the global weight assigned to the term. The formula used is:

gtw_i = log (D/df_i)

gtw_i is the global term weight for term.

D is the total number of documents.

df_i is the number of documents that the term appears in.

See Mi lslita’s explanation of Term Vector Theory for details.

Calculate Similarity Scores. Once local term weights and global term weights are calculated, we are ready to create recommendations. To create recommendations for a document we first find all documents that have any of the same terms in it. It turns out that this can be a very large number of documents (e.g. 40,000 in our system). For each pairing of the document being considered and a document with matching terms we calculate a similarity score. Because calculating 40,000 scores can take a long time (about 15 seconds in our current system), we shorten the list to consider to 200. We do this by sorting the pairs according to the number of overlapping terms. To calculate the similarity score for a pair of documents, we sum over all of the terms that they share in common. The contribution from each term is a combination of the local term weight in the first document, the local term weight in the second document and the global term weight. Because in OER recommender, each feed can be considered a separate collection, we calculate global term weights for each of the feeds and use them in calculating the similarity score.

To calculate the contribution of an individual term to the similarity score, OER Recommender uses:

sst_i = (gtw_1i)(ltw_1i)(gtw_2i)(ltw_2i)

sst_i Is the contribution to the similarity score for a given term.

gtw_1i is the global term weight from the feed that the first document is in.

ltw_1i is the local term weight from the first document.

gtw_2i is the global term weight from the feed that the second document is in.

ltw_2i is the local term weight from the second document.

See NCBI’s explanation of how MeSH calculates Related Articles for details. Once OER Recommender has calculated similarity scores for the 200 documents, it sorts the documents by those scores and stores the top 10 in a database.

Displaying Recommendations. Now that the recommendations are stored in a database, displaying them to the user is straightforward. We have created a Greasmonkey script that requests recommendations for each web page that a user browses. If OER Recommender returns any, the script inserts HTML for the recommendations into the web page. The eduCommons team is building a plone tool so that anyone running eduCommons can turn on recommendations. That way users won’t have to have install the Greasemonkey script in order to see recommendations on the eduCommons website. The OER Recommender site publishes the XML format it returns so others could add recommendations into their websites.

Getting Resources into the Recommender. Currently OER Recommender only provides recommendations for URLs that it has metadata for, so in order to get recommendations, you need to register a metadata feed. If you have an OCW site that provides an RSS feed, you can do that at OCW Finder. If you have an OAI feed or an RSS feed for learning objects or other OERs, you can add your metadata by sending a feed title, URL, and display URL (home page of the repository) to oerrecommender AT cosl DOT usu DOT edu.

Future Directions. There are many things left to be done: (1) adapt recommendations by monitoring which recommended resources people click on and how long they stay at recommended resources, (2) provide links to more recommendations (so users can see more than the default 5), (3) provide a way for people to indicate when a recommendation doesn’t work, (4) provide a way for people to register and login, so they can receive personalized recommendations (perhaps via email), (5) create a process whereby recommendations can be updated without going through all the documents in the database, (6) add functionality for retrieving recommendations for arbitrary web pages (recommending on demand) perhaps via a bookmarklet, (7) create whatever Greasemonkey-like add on is supported on IE. We also plan to integrate the recommender with the aggregator and feed reader to provide recommended news items based on the attention people have previously paid to news articles.

Wow! I realize that this is a long post, but it is not a simple topic. Hopefully the explanation is useful.

The database. I’ve checked the database and found that we have nearly 2 million rss entries and about 350 thousand unique tags. We’ve indexed the entries so that we can access recent ones quickly. We haven’t indexed the entries for fast searching by tag.

Search was dog slow. Ozmozr’s search which is visible from most pages, takes entered terms and treats them as tags with which to search entries. Joining through the massive tag table to the even more massive entries table was taking forever (on the order of 45-60 seconds per query). We temporarily disabled entry searching until I poked around and found that we hadn’t created an index on tag names. Just putting an index on tag names reduced the query time to around 8 seconds. Much shorter, but still too long. So I reduced the complexity of the query. It may not be as powerful as it used to be, but now executes in under two seconds.

Aggregator’s 20 Java threads talking to postgres was stealing the CPU. We use a Java aggregator daemon that I wrote to harvest RSS feeds. We haven’t harvested in a while because of the problems we have experienced. When I fired it up today I found that it brought the site to its knees when it started doing its work. By monitoring the CPU I saw that the aggregator daemon was stealing all of the CPU leaving none for Rails to use to serve pages. By default the aggregator checks feeds every hour. When doing this it can use up to 20 threads. Each thread creates a connection to postgres. It was actually postgres that was stealing all of the CPU, but only because of how the Java daemon was talking to it. My quick fix is to limit the aggregator to 1 thread. I initially designed the aggregator to use many threads because it turns out that most of the time spent in harvesting is used up in waiting for web servers to respond. I guess I need to figure out another approach. I’m going to look to see if we can throttle how much CPU the Java threads (and corresponding Postgres processes) use.

Shrinking tag clouds are a fun idea but a pig to implement. Similar to how OCW Finder allows you to filter your browsing by selecting tags, we initially implemented the same idea in oz via a shrinking tag cloud. The idea was that as soon as you clicked on a tag in a cloud, in addition to the items being filtered, the tags in the resulting tag cloud would be filtered as well (shrinking the cloud). It turns out that this kind of query is horrendously expensive, at least the way we implemented it. It also turns out that users don’t seem to understand what is going on. As a result of this, we ripped the shrinking tag clouds out… almost. I just found an instance where it was left in. They are gone now.

There are more ugly queries to look at but I’ve gotten most queries down to under 4 seconds, which is still a long time, but at least the website doesn’t die. I’ll write more as I find it out.

As a note, the way I was able to easily identify the nasty queries was by going into the postgres config file (var/lib/pgsql/data/postgresql.conf) and setting the option log_min_duration_statement = 3000 (milliseconds). With this option set, every query that takes longer than three seconds is written to the postgres log file.

How does one begin to participate in a community?

If you accept my characterization of a community, you begin participating in the community when you begin caring about the same problems that the community cares about. If you wanted to create an entry path for people to join a community, you could create a sequence of problems to solve and resources to begin solving them including records of past attempted solutions.

• Playing with something is a better way to begin learning about it than to be told all about it.
• When people begin creating things they begin learning the language of design.
• It is OK that most of what people create is no good, at least they are learning the language of creation.
• The tools you create shape what they can and do create.

After returning from the hotel last night we sat and brainstormed about what we are and can create based on what we are doing. A lot of the discussion centered around defining the problem. Here are some possible problems:

• What are future educational scenarios and how can we help shape / facilitate them?
• How do you form successful online communities (ones that have lots of members who regularly contribute and gain value from the community)?

I woke up thinking about these questions.

Can we create a space where people can pose and solve problems?

Can we glean principles from Ze Frank, MySpace, and the MathForum’s problems of the week that could help make that space successful. Why do people like MySpace?

• (Power) They can create what they like
• (Show Me) They can show it to their friends
• (Feedback) People can comment on their pages

I’m personally interested in creating such a space and online community for math, science, and technology.

I envision a place where people can pose and solve problems using interactive models. Connect this to current news involving math and to the history of math discoveries. Provide tool sets that let people quickly create, play with, and adapt with interactive models. Provide podcasts that engage learners in problems, invite them to work on those problems, see how others work on those problems, and pose their own.

COSL’s interests are of course much broader than math, science, and technology. They are all topics. So what is generalizable to all topics. I think a space people can pose and solve problems. Here is an idea that came out of thinking about that general problem.

What if you had a catalog of every problem that you had ever worked on?

What if that catalog additionally included links to web resources you accessed when trying to solve it, and any notes you made about your efforts and potential solutions? What if you could share that catalog with your friends and with communities that cared about the same problems?

It seems like that would be valuable, but how would such a thing get created. What if you had an agent that monitored (and stored for your private use only) a record of all of all of your searching and browsing activity. The agent analyzes those activities and clusters them based on concerns. It allows you to annotate those clusters with tags, notes, and explanations so that they can be more useful to you and others. It organizes and ranks web resources based on how long you spent at those pages, how often you came back and so on. It lets you contribute those problems and answers to relevant communities. It recommends relevant web resources, communities and users.

“It was too easy to make a mess in Smalltalk” Ward Cunningham (Dynamic languages allow you to be undisciplined.)

Fundamental dilemma. Bad code will fundamentally slow me down. But when faced with a deadline, we go ahead and write bad code anyway. Showed productivity vs time curve. As a project wears on you become less and less productive. The only solution is to clean up the code.

Possibilities:

• Grand redesign in the sky?
  • It will take forever
• Incremental improvement
  • Always check code in better than you checked it out.
  • Never let the sun set on bad code.
  • Test First! – Provides the necessary flexibility.

RSpec & TestUnit – unit testing frameworks. Write your tests first. Target 100% coverage.

Showed an example of building out a command line argument parser. Starting with handling boolean arguments, then other data types.

Discusses the problem with if-else statements:

• They tend to duplicate themselves. If you want to change the code you need to duplicate it in lots of places.
• The original design pattern did not scale.
• Violates the DRY principle and the Open Close principles. Applications should be open for changes but closed for modification. Design so that you can add functionality without changing existing code.

Refactor when you smell a mess (festering pile) not when you have a mess (as early as possible). So there is this fine balance be over-engineering and keeping your code clean. So clean it when it starts to smell and not before.

Similar to writing essays, we should re-write code periodically.

Incrementalism

Don’t make massive changes in the name of improvement.

• TDD: Keep the system running at all times! I am allowed to make a change that breaks the system. Every tiny change I make must keep the system working.
• Running tests helps point out all of the places that changing the code impacts.

Get into the feel of running the tests continually. Autotest is a product that allows you do this.

Continuing his demo he showed how by creating marshallers he was able to move most of the data type specific code into one place.

Software is about separation of concerns. Segment code that changes according to how often they change. Put things that change a lot in a different place than those things that don’t.

Was it worth it? (refactoring his demo code) If you are worried about function and function alone you should not be in this business. If you are not a craftsman you shouldn’t be here.

The Green Band (A wrist band he wears)

• Professionals write tests first
• Clean their code
• The only way to go fast is to go well

Making a mess is not faster.

I’m here at the First Day with JRuby on Rails tutorial with Charlie Nutter and Tom Enebo. We ate lunch with them at the Oasis at the Mtn West Ruby Conference in SLC. Cool guys.

I just downloaded JRuby. This tutorial is supposed to get me started with JRuby.

To get going we include the Java library and then fire away.

First problem. This is a new laptop and while I have Eclipse loaded on it, I didn’t have a JDK installed. Ok, here goes trying to download 53 MB of JDK across the conference’s wireless. Trent predicted that the wireless would be brought down within the first 30 minutes of the conference. I guess I’m contributing to this eventuality. We will see.

Topics Charlie and Tom covered:

• Installing JRub
• Why JRuby?
• Unicode support
• Threads

I’m still back on installing Ruby thanks to no JDK installed. Even if I had that, they spent all of about 10 seconds on Installing Ruby. Hopefully they will come back to that topic later and I will have the JDK downloaded.

They are spending a lot of time talking about threads. They motivate the topic by saying “have you ever wanted to launch a long running process from ruby?” Actually yes, I have and I do. For Ozmozr and OCWFinder we run various daemons to harvest RSS and work on the database. When we first started looking at this I spent some time looking at the various open source tools available for doing RSS on Python, Ruby, and Java. I benchmarked a number of them on all three platforms. Java won hands down by about a factor of 3. I’m sure it wasn’t super scientific, but it was enough to convince me insomuch that I’ve been developing Java for about 5 years. Anyway, we wrote our daemons and harvesters in Java and it seems to be a great combination of Java and Rails (our apps are written in Rails) to solve the problems we need to.

9:02 JDK download is 23% complete

Class path. Instead of having to list full class paths you can just throw the jar files in the directory:

~/work/jruby/lib

Charlie claims that classpath is one of the most painful things about getting things up and running in Java. Funny how many times I’ve run into that. The fact that you can’t tell java to load all of the jar files in a directory is truly amusing and unexpected.

9:12 JDK download is 49% complete

Charlie mentioned that implementing migration support for JDBC was a bit challenging and required DB specific code.

Charlie points out that Java start up time is abysmal. This can be a problem if you are running lots of command line rails scripts. To solve this problem they are looking at running a Java server that you send requests to so that you don’t have to start up Java every time you want to run a script.

9:23 JDK download is 92% complete. My connection keeps flaking out. Please don’t die, please don’t die.

Set up and Deployment. Is the single aspect of Rails development right now.

9:25 Finished JDK download. Wahoo!

GoldSpike! A Rails plugin for building WAR files. A WAR file is a single file containing all of a libraries and dependencies for a web application.

They love GlassFlish (Sun’s application server). glassfish.dev.java.net. Swell. I’ll have to tell trent about this.

Mephisto is a web publishing system that contains blogging and simple CMS support. I will have to check it out.

Glitches. I had to define JRUBY_HOME to point to the root of the installed JRUBY and then add the JRUBY_HOME\bin to my path. Tom says I shouldn’t have to define JRUBY_HOME. Anyway I get all of this set up and go to my favorite rails app (OCWFinder) and try jruby script/server. Output:

Cannot find gem for Rails ~>1.2.3.0:
Install the missing gem with ‘gem install -v=1.2.3 rails’, or
change environment.rb to define RAILS_GEM_VERSION with your desired version.

Hmmm….

K, Duhh. Charlie helped me realize that I just needed to follow the slides that we blew through quickly at the beginning of the session. I needed to install rails from within JRuby:

jruby -S gem install rails

Ok, now I’m starting to see the same error messages that I’ve seen before when I didn’t have my postgres driver set up correctly. I’ll continue to follow up when I get back from lunch.

I posted my problem to #railsconf and got no help. I asked if anyone has gotten a railsapp under jruby and one person came forward but then said that it was a while ago.

I’ve seen stuff like this before, but am scratching my head on where to go right now.

NetBeans. Supposedly the best Java IDE available. Now has Ruby support.

Someone is snoring really loudly. How rude

NetBeans provides for ruby: code completion, jump to declaration, and refactoring support. All things I’ve come to expect from using Eclipse. I wonder when they will be implemented there.

Someone from freenode has come onto #railsconf asking for an official contact from the conference. I guess everyone is connecting on the same IP address.

I’m currently thinking about the problem in the context of ozmozr and the NSDL. To tackle the problem, I’ve broken it down into the following questions:

• Who – Who are we recommending things to? (people or groups)
• What – What are we recommending? (groups, feeds, stories/web pages, tags)
• When – When do we offer recommendations? (when searching, when visiting home page)
• When not – When do we refrain from making recommendations? (when we don’t reach a certain threshold of certainty)
• Criteria – What factors should we include when considering what recommendations to make? What weight should be given to those criteria.
• Algorithm – What algorithm should we use to implement the recommender?
• Implementation – How do we implement recommender? (use R to do the analysis and store the results in the DB, present it via rails)
• Priority – What is the priority of implementation? Where will we get the biggest payoff for our efforts?

The factors considered in the algorithm will depend on the what is being recommended and who it is being recommended to.

Recommending Stories to Users

Our approach will combine content filtering (co), collaborative filtering (cl), and rational analysis (ra) (rules that make sense).

Factors to consider:

• Titles, tags, and contents of stories they have read, tagged, shared, voted, or externally linked

• Feeds they subscribe to
• Groups they belong to
• Stories that people like them have read, tagged, or shared, and how recently they read, tagged or shared them
• Whether or not they have read the story before
• How recent the story was published
• How often the story was viewed

Algorithm:

1. (Rational) Get the set of stories published within the specified recency.
2. (Rational) Eliminate stories the user has read before.
3. (Rational) Eliminate stories from the feeds that the user subscribes to?
4. (Rational) Eliminate stories from the feeds that groups subscribe to that the user belongs?
5. (Content) Get the set of stories similar to ones that the user has read, tagged, or shared before.
6. (Collaborative) Get the set of stories that similar users have read, tagged, or shared.
7. Store in a “user_story_recommendation” table (user_id, story_id, content_score, collaborative_score)
8. Display recommendations by querying the r-table for stories except 2-4, ranked according to a combination of the content and collaborative scores and recency.

Proposed formula for composite score: rank = r ( A(cn) + B(cf) ). Where cn = content score, cf = collaborative score, r = recency score, and A and B are arbitrary weights to allow us to tune the relative contribution of the content and collaborative filtering scores.

Should the recency factor decay linearly, logarithmically, or exponentially?

I attended a discussion group facilitated by Damon Bahr of BYU that focused on finding the balance between teaching for procedural and conceptual knowledge. I also attended a presentation by Blake Peterson of BYU that reported research comparing the presence in US and Japanese preservice training of discussion related to classroom management.

Summary

• Searching and Publishing – People come to the MTDL to find and share resources
• Overcoming Isolation – People come to the MTDL to help overcome isolation
• Discuss Development – Talk with others about the development of math software

Personas

• Teacher Developer
• Professional Developer
• Educational Researcher
• Inexperienced Developer
• Hobbyist Developer

Metaphor and Principles

• Workshop metaphor
• Design for multiple roles
• Design for multiple levels of expertise
• Provide activity indicators

I like the workshop metaphor, though I think that perhaps there are better. I can’t really discern the implication of designing for different roles and different levels of expertise. I lilke the idea of activity indicators. I realize that this is an area of recent interest throughout the field.

He took his first Cog Sci class (on problem solving) from Jim Greeno who he called his mentor. He has been studying problem solving for most of his career.

He took a job at UCSB out of grad school and has been there for the past 28 years. It has been a good environment for doing his research.

The lineage of his research is:

1. Advance organizers
2. How illustrations (pictures) can portray analogies and metaphors which can help facilitate problem solving
3. Animations

While there are many guidelines out there for how to design instruction, most guidelines are not based on evidence or theory. He is attempting to discover multimedia design principles that are theory and research based.

He showed a diagram depicting the information processing model of cognition. Perception, sense making, and integration.

Me: This makes me think about what a theory behavior and cognition is. At a minimum, it is an explanation for how people function. It seems like useful theories can be used to make testable predictions.

A technology-centered approach contrasts heavily with a learner-centered approach. We should first try to understand how people learn before we try to figure out how to design instruction for them.

He makes a distinction between two main kinds of learning outcomes:
-remembering (recall)
-understanding (transfer)

Me: Of course, what constitutes new situations? What constitutes transfer? It is easy to differentiate between recall and application, but not so easy to determine the degree or distance of transfer? That assumes that there is some way to quantify or measure the elements of knowledge.

The level of cognitive activity determines level of meaningful learning.

Me: What about phsychomotor learning?

Activities that do not involve interaction do not support learning.

What makes an activity meaningful?

A lack of explanations and opportunities to question why may be one cause.

Does adding pictures to text-based instruction increase learning?

It can.

Example: Lightning lesson

• retention test: write down all you remember
• transfer tests:
  • redisgn
  • troubleshoot
  • predict

Two versions (a) paper based, (b)computer based (with animation and voice over)

Text plus pictures was better than text alone. Text plus animation was better than text alone.

Redesign: How could you make it more effective?
Troubleshooting: If you did x, and it didn’t work, what might have gone wrong?
Explanation: Why does x?

Has also worked with instructional games and simulations

Assumptions of cognitive theory:

1. Dual channels – humans have separate channels for processing visual and auditory information
2. Limited capacity – William James – humans can only process so much information at one time
3. Active processing – meaningful learning occurs when people are actively involved: (a) selecting/attending – paying attention, (b) organizing – into a coherent representation that makes sense to you, (c) integrating – relate the new information to your existing knowledge

How can you foster integration given that people have limited working memory?

What about tool use and using the environment as cues to remembering and solving?

Research questions:

Multimedia Effect
-Does adding a graphic (to text) help? – Text alone vs. together Yes: great improvement in transfer test (multimedia effect) – graphic voice over was better than text

Contiquity Effect
-What makes a good graphic? separated vs. contiguous A: Integrated with text works better – locate text close to relative portions of images
(spatial or temporal contiguity effect)

Coherence Effect

Sometimes it is better to present less information than more. Seductive details – interesting but not related information (does it motive them more and make them more interested)
-Adding those details has a big negative effect on people’s transfer ability
-Adding background music depresses
-Reading summaries produced more learning than reading whole lessons
-Extraneous information gets in the way of making connections

Modality Effect
-People do much better when the words are spoken rather than printed
-When you present them in textual form it overloads the visual system. Presenting them auditorily offloads the processing to the auditory system
-Graphics and narration and text results in less learning than just graphics and

Personalization effect – conversational results in more learning than formal presentation

Pacing effect -

Signaling effect

Individual differences – low knowledge learners need better

Theory and research based

Clark & Mayer (2003) E-learning and the science of instruction

Mayer (2003) Learning and instruction.

Analysis of graphics in CA text books. Over 50% of the space was devoted to graphics. About of the 85% of the graphics were not instructional

Barbara White – Learning Physics in a game environment – learned better when asked to reflect afterwards

Games
Design a plant – that will live on another planet (for a environment
Dr. Fizz – Electromagnet
-modality and interactivity effects observed

-Motivational? Does adding a game aspect to instruction help learners persist longer?

Progression of mental models

Me: What about the component fluency hypothesis? He agreed that some things need to be automated to free up cognitive resources for conceptual processing

Future research interests:
-games and simulations (because of how motivating they appear to be to children)
-individual differences – cognitive style (verbal, auditory)
-online pedagogical agents (because they are forecasted to be the guide on the side in the future)
-What kinds of questions do people ask a tutor (when learning SPSS)?
-What kinds of answers do tutors give?
-Theory of tutoring?

Pavio & Battely – dual channel effect

What would a theory based on cognitive principles be like for:
-Tool use?
-Tool use instruction?
-Tool design?

Of course, like life is, I’m not any less busy now that I’ve defended. The only difference is that I now have 8 projects all more or less equally competing for my time instead of having one obvious first priority.

Bring to the foreground that there were two main purposes for my dissertation:

• Evaluate and refine TATSTAM – Study the process of how middle school math teachers reuse and adapt mathlets
• Evaluate and improve TADRIOLA – Develop a practical tool that teachers can use to author instruction

Add quotes and prompts for me to tell stories. My stories should include the turning points of my study. Also those things which were unexpected.

Similarly he suggested adding an Outcomes chapter to my dissertation. He thought the Discussion chapter should more be a reflection on my research methods and implications for further research.

He said that my committee will likely know everything I have to say and be mostly interested in hearing stories about what happened. Those stories can help convince them that the research is real, that it is original, and important.

He said that in qualitative, the researcher is the instrument. In reporting my findings I need to convince the audience that I am a good instrument for gathering data: (a) I listened and observed with fidelity, (b) I made good decisions going into the research, (c) I opportunistically shifted the focus of the research to follow up on new aspects of the research that opened up and to firm up my understanding of themes that emerged, (d) I modified instruments as needed to achieve the goals of my research.

His interpretation of what my research is all about went something like this. The context of my study is an ecology that includes middle school math teachers, classrooms, and students. Teachers lie along a continuum of wanting completely pre-packaged self-contained instruction that they can just instantiate and being Michael Angelo’s of math instruction wanting only the parts from which they will masterfully create their own instruction. I have created TADRIOLA, a tool that can be scaled and used by teachers at any point along that continuum. My study was to let this new organism TADRIOLA loose into the ecology and to study the reaction of the ecology to the organism. In which parts of the ecology was it embraced or not?

I told him that I hadn’t yet done any member checking. He said that in fact I had, by virtue of the iterative design of my research. He thought that was one of the strengths of the research design that I used. By repeatedly going back to the same type of people with preliminary findings is a type of member checking. He said I might as well send a high level view of my findings to participants to formally complete member checking. He agreed with me that more than likely they will not respond.

He indicated that the flow of my dissertation defense should be roughly as follows:

1. Quickly move through all of the intro leading up to the analysis in order to convince them that I’m organized and thorough.
2. Digress and take time telling the story of the research. Share quotes and stories. Talk about the decisions that I made. This is where I convince them that I am a good researcher and a good research instrument.
3. Wrap up with the outcomes and discussion.

I asked him about how to best include quotes in my findings. He said that what I shouldn’t do is include multiple quotes saying the same thing. I have the data and they can look at it if they want to. Present one quote saying one thing, then another perhaps contrasting comment, and then wrap up saying that approximately x% agreed with the comment.

When boiling down data to numbers, things like member checking, inter-rater reliability, and triangulation assume more importance when I am trying to make claims that strongly oppose existing theories. How attractive are competing alternative hypothesis? He said the strongest types of statements I want to say are that the majority of respondents said x and the rest were all over the map. The purpose of it is to convince them that I am a quality researcher.

What are your thoughts as a teacher? Which would rather have, a prepared lesson (not a description of a lesson, but the actual materials to use such as black line masters, manipulatives etc), or a catalogue of pieces (e.g. CDs full of clip art, handouts, and worksheets) from which you can assemble lessons?

Does your answer change if we are talking about web based lessons that can be easily adapted?

I think a careful look at how technology could change the role of the teacher would alleviate these fears and in fact excite teachers. I have teachers responsd to Saxon and similar heavily scripted curriculums by saying that it turns them into a robot. Some teachers react to to scripted online instruction in similar ways.

Technology contrasts heavily with this. I believe that a major opportunity that technology offers is to allow us to deliver mastery learning (teach students where they are at and don’t move on until they do) in a way that also encourages understanding as opposed to rote learning of facts and procedures.

My perception is that direct instruction approaches to mastery learning try to narrow the gaps between learners, but often results in rote learning (and higher test scores by the way). Side note: I take it for granted that understanding is much harder to measure than recall and for that reason we assess recall and results based learning (NCLB) values instructional methods that produce it.

I believe technology offers an alternative to trying to homogenizing all learners, it can help teachers manage the logistical chaos of trying to let large numbers of learners go at their own pace. The role change I see of teachers is from dispenser of information to diagnoser of student understanding and customizer of instruction. To me this does not denigrate or demean the teacher, rather it seems to be immensely freeing. It allows you to be a fellow sojourner for understanding. I’d be interested to hear the opinions of lots of teachers on this? I’m sure one reaction, is yeah, that should great but it is pie in the sky.

• User interface design � What are the principles and processes can be followed when designing user interfaces to increase their usability?
• User centered design / action research / design research � What processes can be followed to put developers in contact with users to efficiently gather data to inform their designs?
• Cross platform issues � What are the common problems that developers encounter when trying to develop mathlets for multiple platforms? What are the ways that developers often approach these problems?
• Authoring tools for teachers � What are the functionalities and characteristics of authoring tools that are accessible by teachers?
• Free and open software � What are the different options available for licensing software to and from others? What are the business models that the various licensing support?
• Software reuse � What is out there that can be reused? What do mathlet developers want to be able to reuse? Where can I find software that I can reuse? If I want to provide software that can be reused, what issues should I consider?
• Strengths of instructional software � What are the strengths of instructional software in general and for math specifically that can be taken advantage of? (dynamic representations, linked representations, learner modeling, instructional feedback, problem sequencing, automated testing, collaboration over distances)

Then there is Elliot Solaway’s statement I’ve heard him make on one more than occasion “I tried so long to figure out how to make computers smarter, then one day I realized, its not the computers that we should try to make smarter, its the students, so that is what I’ve tried to focus on since.” Both of these resonate with the theme of the Social Life of Information by JSB that I’m currently reading.
I ran across Quantum Simulations intelligent tutoring systems again today. I’ve looked at it briefly before and will look at it again. I have been a student of the ITS literature and attended the CIRCLE Summer School a couple of years ago. I taught a semester of College Algebra here at USU using Carnegie Learning’s Quantitative Literacy Tutor. My masters project involved in the creation of an ITS for teaching solving systems of simultaneous linear equations (I’ll post a link to it tomorrow). I also created an expert system as part of some research on feedback that I conducted with Dr. Gibbons. One of the five dissertation proposals I put together focused on problem sequencing. To prepare for that project I conducted an extensive review of the literature on problem sequencing including ITS research (I’ll post that too).

So what does ITS have to offer? From my reading:

• Methods for analyzing human performance that are perhaps more rigorous than those typically offered by the field of ID.
• User interfaces that allow performances to be carried out in non-linear sequences as opposed to the strictly sequential sequences typically found in most CBT.
• Ways to interpret more complex types of student input than typical CBT.
• Methods for dynamically sequencing instruction or problems.

• Repetition
• Visualization
• Procedures for generating the answers
• Counting by __ (for multiplication)
• Using reference values

If anyone has other strategies or pointers to online resources that are good examples of these strategies to teach basic math facts, please share.

The most common strategy it seems is recall by drilling and killing like in my boy’s school. One effective way to do this is by using flash cards. The Surfnetkids homework help newsletter lead me to their math facts flashcards page. The problem with most flash cards is that they don’t teach any alternative besides recall. It seems like flash cards could be created that taught alternate strategies.

• ESCOT (dev, JavaDocs)
• Instructional Architect
• CreateStudio
• Create@Brown
• CourseWare
• E-Slate

I welcome pointers to additional projects like these. When I get a chance I will write reviews of each of these projects. I’m hoping to find synergy with projects like these.

If you were given a small amount of resources, what would you do?

Who should be targeted? Those that are flunking out, the average student, the high achiever? What age level?

A number of opinions have been expressed:

• Technology can help, but in itself it is not the solution.
• Integrated curriculum can help motivate students, but in itself it is not the answer.
• Highly structured, programmed instruction is needed to help students gain basic skills, but if it is used in isolation it results in students loosing interest and not learning higher order skills. It should be used in conjunction with other exploratory and discovery methods.
• We need to start early with students. If we wait until middle school or high school it is probably too late. A contrasting opinion is that we can in fact help change the culture of middle school and high school classrooms, which results in improved student perspectives of mathematics (though possibly not improved math scores).
• A professor of secondary ed stated that he thought what made the biggest difference (in student learning and attitude?) is the interaction pattern used by teachers. He deplores the IRE (teacher initiates a question, student responds, and teacher evaluates the students response). He believes that this results in students feeling like everything they say will be evaluated, and so they stop saying anything.
• We need to support individualized instruction (a la Mastery Learning). Teach each student where they are at, and don’t move on until they learn it.
• In order to support individualized instruction, a detailed model of what each student knows should be developed and follow students through their educational career. This model should be used by teachers to help individualize instruction.

To me, the challenge seems to be to develop a model, materials, and support system that teachers will buy into, that allows them to individualize instruction. Does anyone out there have examples of how this has successfully been accomplished? I am also interested to hear opinions on the points I have listed.

Here is an approach I have dreamed up. I think it is a type of cluster analysis. Anyway, the first step is to code all textual data according to issues and propositions. Then create a metric to measure the distance between each pair of participants. The metric would work like this:

• If both participants made statements that supported a given proposition then don’t add anything to the distance score.
• If one participant made a statement that supported a given proposition and the other did not, then the add 1 (or some other magic numberto the distance score.
• If one participant made a statement that supported a given proposition and the other made a statment that opposed the proposition, then add 2 (or some other magic number) to the distance score.

I would weight each proposition according to how much of a factor I felt it was in determining the overall distance between two learners. Using a similar rubric, I could include some of my survey and background data in the calculation of difference scores.

This would result in distance scores for each pair of teachers. The hope is that clusters could be identified. If no clusters were observable, I could tweak the weights until I found some clusters (ha, ha). Really it seems like this is the process that one goes through when analyzing qualitative data. It is interesting that it seems more natural for qualitative researchers to try to come up with profiles or categories of learners than to try to describe average participant responses to a given issue. That is probably not so surprising, since it is often taken for granted that the sample is likely not representative. I think I will try to do both.

Reigeluth, C.M., & Frick, T.W. (1999). Formative research: A methodology for improving design theories. In C.M. Reigeluth (Ed.), Instructional-Design Theories and Models: A New Paradigm of Instructional Theory. (Volume II). Hillsdale, NJ: Lawrence Erlbaum Assoc.

Miles, M. B., & Huberman, A. M. (1984). Analyzing qualitative data: A source book for new methods. Beverly Hills, CA: Sage Publications.

Yin, R. K. (1984). Case study research design and methods. Beverly Hills, CA: Sage Publications.

Wes’s Presentation
Wes gave a helpful overview of the main services of the MathForum. While I have visited there periodically it is so big that it is hard to get my mind wrapped around all that is there. Perhaps the most interesting things to me are the Problem of the Week (POW) online mentoring.

One of the challenges they are currently facing is sustainablity. They have mentors who respond to student responses to problems of the week. Because they recieve 300+ responses each week, they need a way to reduce the cost of doing so. Approaches to this that they are considering include: (1) Enlist University Professors and others to respond on a voluntary basis, (2) Start charging for the service. Data from a recent survey they conducted indicates that ~30% of teachers and students even care about the moderating. Another option I discussed with him is to try to automate some of the work. I proposed the following process:

1. Analyze past student responses to see what the degree of clustering of response types is.
2. If there is sufficient clustering, perhaps the work could be divided into two tasks, first sorting student responses, and then writing a response for each of class of responses.

Anderson and Koedinger’s research on feedback is that diagnosing errors in detail and giving detailed customized remediation based on the type of error is not nearly as important as getting learners back on the right track. Of course a lot of this research comes from studies of immediate feedback which, it doesn’t appear that PoW mentoring is. Something to consider anyway.

He presented briefly on the Math Tools DL. He gave the vision that they hoped to utilize what they have learned from the somewhat naturalistic evolution of the Math Forum to apply to the development of this new resource.

I also discussed with Wes the possibility of analyzing some of the NLVM log files to contribute to a characterization of different types of online communities. Here are some hypotheses I would like to test in regard to that:

• Sessions that last ~30-45 minutes are more likely to be from classrooms then sessions of different lengths.
• Multiple IP addresses from the same domain are likely from a given class.
• If students in a class access many different resources in the NLVM, they are likely “playing”. The teacher’s use of the NLVM is not for the purpose of supporting specific learning outcomes.
• In contrast, students all accessing the same small set of resources, is likely targeted use.
• Repeat usage can be gauged by looking at repeat visits of same domain over time.

My guess is that there is a lot of playing going on. We would like to increase the amount of purposeful use. This will likely increase when objective focused sets of activities are made available. Currently the library is what I like to call mathlet centered as opposed to objective centered. So far, using TADRIOLA I have created a few lessons that are objective centered.

One interesting hypothesis that Wes shared is that being busy is part of what it means to be a teacher. It is part of their identity. It is also a way of being comfortably secure. When a teacher is busy, life is hectic, but at least they can deal with it. When you ask them to do something new, and they respond that they cannot because they are busy, that may be a way of saying, I’m comfortable doing what I’m doing, I don’t want to try something new. That is not to denigrate the fact that teachers are in fact busy. There is lots of data to show that. Wes pointed out that despite their business, never has a teacher turned him down when he asked them if they would spend an hour with him to do an interview. My experience has been similar. In general, teachers have been very gracious to spend time with me.

I talked with Wes about analyzing the qualitative data that I gathered. He indicated that in a similar analysis he conducted a few years ago, he found teacher types that characterized teachers according to their skill and comfortability level using technology. He found that technical skill was highly correlated with pedagogical expertise. I’m guessing that my data doesn’t exactly bear that out.

I told him of my guess that while many of the teachers that I worked with indicated interest, I doubted that many would follow up. I hope that I am wrong. He responded, that with the sample size I had (~80), he expected that there would be only 1 or 2 among them who would take the initiative to lead out in innovating with the technology.

Sanne’s Presentation (PPT)

The main thing I got out of Sanne’s presentation related to problem representation. He talked about how a problem is represented, or how a person represents to themselves has a major effect on how effectively and efficiently they can solve it. This ties in heavily to what we have been doing on the NLVM. Providing visual, dyamic, and interactive representations of problems that make them easier to conceptualize and solve. Good representations obscure details that are not needed to make decisions and accentuate details that are important for solving the problem.

Dr. Merrill’s Presentation

As usual, Dr. Merrill’s presentation was passionate and thought provoking. He pushed for theory building and empirical validation. I am all for that, however, there are questions other than will students learn from this? that are important, such as will teachers use this? will students use this?

CC licenses are being developed as an alternative to copyright. In the case of copyright items, if you want to copy, use publically, create derivative works, or otherwise use the item, you must contact and obtain permission from the copyright holder (with the exception of fair use exceptions).

The idea with CC is to create alternative CC symbols that have recognizeable meaning to them so people who want to use the work can do so in the ways the associated licenses allow without having to get explicit permission. Of course this does not preclude people from contacting the creator to obtain permission to use the work in ways not specified by the CC license.

If this understanding is correct, I agree with Wiley’s latest proposal, that CC-EDU be a single symbol and license that has a single meaning, so all that see the symbol could know exactly what it means without further investigation. I also agree with Wiley’s the license options that Wiley has suggested.

Perhaps this has already been discussed, but it seems that the other option is one that is currently practiced by many websites, which is to place a statement of allowed use that describes in detail how the works hosted on the website can be reused. I guess the advantage of the the CC license is that people won’t have to try to read and decypher those statements.

In TADRIOLA, teachers can share classes, lessons, or activities that they create or assemble. I give them the options provided by CC, plus some additional ones (screen capture). Then when teachers go to borrow resources, they are notified of the conditions. The point I think of the previous discussion, is it may be better to simply define one set of use conditions that all must agree to if they are to play in my sandbox, thus avoiding potential confusion. It might be interesting to gather some data.

I know that well constructed multiple choice questions can do a good job of assessing many things, but my suspicion is that we can do more and differently using mathlets. Anyway, these ideas seem to have a place in what Popham calls work samples and performance-task test items.

• Look for big findings first.
• See if you can detect archetypes teachers from the data.
• The quantitative data I gathered is likely to be very noisy.
• I should work from a theory of learning and instruction, not technology.
• Finishing my dissertation needs to be a primary concern.

Look for big findings first
Clearly this only makes sense, though I probably would have not come up with that. Why give each bit of data and topic mentioned equal weight. He suggested that I just take each packet of information and sort them into four piles, not even being specific about what the piles are. I’m in the process of enterring all of the data into the computer and so I probably won’t go through the physical steps of what he described. I will do the equivalent electronically though. This leads to the next comment…

See if you can detect archetypes teachers from the data
My initial plan was to consider each of the tasks, barriers, and guidelines separately and look at the data I had gathered seeing what it had to say about each. In retrospect, this is probably a very quantitative approach. I would probably be able to report statistics about the percent of participant statements that gave evidence or support for each of the elements. The approach he suggests contrasts heavily with this. His approach is to look for archetypes into which I could group my participants. Each archetype description would obviously not completely describe each participant I placed in that category. However it is perhaps a good way to think about it.

Example archetypes could be: (a) teacher sees this and immediately envisions how it could be used, (b) teacher likes what they see, but needs some hand holding, (c) teacher wants something handed to them that they could use out of the box, (d) teacher would never touch this stuff.

The pragmatic usefulness of identifying archetypes might be to help give direction to where developers (I) should spend future development time and money. For example, I could focus on creating additional teacher support and integration with existing standards and textbooks, or I could just focus on developing newe mathlets.

After I have identified archetypes I could then move to analyzing parts of the archetypes.

I recently read a writeup of a closely related study. In doing that I was bit concerned about the lack of reporting of numerical data along side of provided anecdotal quotes. Because I’m pretty new to qualitative research, I was not clear as to how you should present your data. He indicated that he uses the following approach:

1. Present a statement of a theme
2. Give representative quotes to back it up
3. Indicate some percentages about the number of participants or statements that concur or disagree

Another method I have seen mentioned in the literature is to present and subsequently discount alternative explanations of the data. One thing that Dr. Lancy recommended is to make sure that any claims I make can be supported the data. I guess this is obvious advice, but from some of my readings, not always followed.

The quantitative data I gathered is likely to be very noisy
I had a difficult time getting teachers to comment on issues like: In what ways would you like to reuse and adapt interactive online resources? and What features would you like in an authoring tool? I’m guessing that this is because they have had relatively little experience in these areas. So what I chose to do was present them with lists and descriptions and asked them to rate items. This resulted in likert scale and ranking survey data. Lancy thinks this data will likely be very noisy if I try to look at distributions across all participants. I agree, it will be interesting to see if looking at the data by archetype makes it less noisy.

I should work from a theory of learning and instruction, not technology
Dr. Lancy’s impression is that many of us Instructional Technology types tend to focus more on the technology than on the learning. He pointed out that there is already significant literature on teacher adoption of innovation and that we should not expect their approach to adopting this innovation to vary greatly from other types of innovation. He suggests grounding theory and interpretation of data in the context of that literature.

Finishing my dissertation needs to be a primary concern
A number of times during our discussion, I started talking about how I could gather some additional data or do this or that. Each time, Dr. Lancy brought me back to reality, reminding me that I need to finish. His most memorable statement was “Joel, we don’t want you to retire on a graduate student salary.” I think my wife would resonate to that statement

Surely he is being both sarcastic and realistic. I guess when writing a blog you have to decide who you are writing it for. Surely when you start like I have just done, you have to be doing it for yourself or you will likely be deeply dissapointed.

I’ve tried to look around and see who in my world of interest is blogging and I don’t see a lot. Areas of interest for me are:

• Math and science education
• Public school teacher reuse and adaptation of online resources
• Authoring tools
• Using simulations in instruction
• Intelligent tutoring systems
• User interface design
• Sequencing problems for instruction
• Instructional feedback

Clearly I don’t have a good idea of how to look and haven’t reviewed a small fraction of what is out there. If you have pointers to blogs highly relevant to these topics, please share.

Many thanks to my Ph.D. chair David Wiley for getting me started.

My primary interest is in completing my dissertation study titled Design Theory for Authoring Tools that Support Teacher Adaptation of Mathlets (TATSTAM).

While my wider interest is the reuse and adaptation of all kinds of interactive online resources, for the purpose of my study I have narrowed my focus to middle school math teacher reuse of mathlets.

The three areas of investigation of my dissertation study are:

• In what ways do middle school math teachers want to reuse and adapt interactive online resources?
• What barriers do middle school math teachers encounter when trying to reuse and adapt interactive online resources?
• What design guidelines support the development of authoring tools that support teacher reuse and adapt interactive online resources?

As part of the study I’m developing a web-based authoring tool called TADRIOLA. Following a slightly adultrated formative research methodology (Reigeluth & Frick, 1999) TADRIOLA is an instance of the initial version of TATSTAM that I created by synthesizing the literature and personal experience. My main deviance from “orthodox” formative research methodology was to use it to develop software design theory as opposed to instructional design theory.

A simple explanation of what I have done is:

1. Review the literature to identify ways that teacher want to reuse and adapt interactive online learning resources a well as barriers that they encounter.
2. Propose a design theory for authoring tools that support teacher reuse and adaptation of interactive online learning resources (TATSTAM).
3. Design a web-based authoring tool (TADRIOLA) that is an instance of the design theory.
4. Conduct focus groups with middle school math teachers to gain empirical evidence about teacher wants, barriers, and the “goodness” of TADRIOLA and TATSTAM.
5. Analyze data and revise TADRIOLA and TATSTAM based on the analysis of the data.
6. Repeat.

I’ve utilized a case study methodology (Yin, 1984) and analyzed data using qualitative coding methods (Miles & Huberman, 1984).

So far I’ve conducted focus groups with middle school math teachers at 6 different locations. At the first location I worked 1-on-1 with 7 teachers using questionnaires, structured interviews, and user tests. At the other groups I have worked with groups using surveys, questionnaires, group discussions, and user tests. In all I have worked with 59 middle school teachers and some 20 other types including technology specialists and college professors who teach students preparing to be math teachers.

Reigeluth, C.M., & Frick, T.W. (1999). Formative research: A methodology for improving design theories. In C.M. Reigeluth (Ed.), Instructional-Design Theories and Models: A New Paradigm of Instructional Theory. (Volume II). Hillsdale, NJ: Lawrence Erlbaum Assoc. [PDF]

Miles, M. B., & Huberman, A. M. (1984). Analyzing qualitative data: A source book for new methods. Beverly Hills, CA: Sage Publications.

Yin, R. K. (1984). Case study research design and methods. Beverly Hills, CA: Sage Publications.