He took his first Cog Sci class (on problem solving) from Jim Greeno who he called his mentor. He has been studying problem solving for most of his career.

He took a job at UCSB out of grad school and has been there for the past 28 years. It has been a good environment for doing his research.

The lineage of his research is:

1. Advance organizers
2. How illustrations (pictures) can portray analogies and metaphors which can help facilitate problem solving
3. Animations

While there are many guidelines out there for how to design instruction, most guidelines are not based on evidence or theory. He is attempting to discover multimedia design principles that are theory and research based.

He showed a diagram depicting the information processing model of cognition. Perception, sense making, and integration.

Me: This makes me think about what a theory behavior and cognition is. At a minimum, it is an explanation for how people function. It seems like useful theories can be used to make testable predictions.

A technology-centered approach contrasts heavily with a learner-centered approach. We should first try to understand how people learn before we try to figure out how to design instruction for them.

He makes a distinction between two main kinds of learning outcomes:
-remembering (recall)
-understanding (transfer)

Me: Of course, what constitutes new situations? What constitutes transfer? It is easy to differentiate between recall and application, but not so easy to determine the degree or distance of transfer? That assumes that there is some way to quantify or measure the elements of knowledge.

The level of cognitive activity determines level of meaningful learning.

Me: What about phsychomotor learning?

Activities that do not involve interaction do not support learning.

What makes an activity meaningful?

A lack of explanations and opportunities to question why may be one cause.

Does adding pictures to text-based instruction increase learning?

It can.

Example: Lightning lesson

• retention test: write down all you remember
• transfer tests:
  • redisgn
  • troubleshoot
  • predict

Two versions (a) paper based, (b)computer based (with animation and voice over)

Text plus pictures was better than text alone. Text plus animation was better than text alone.

Redesign: How could you make it more effective?
Troubleshooting: If you did x, and it didn’t work, what might have gone wrong?
Explanation: Why does x?

Has also worked with instructional games and simulations

Assumptions of cognitive theory:

1. Dual channels – humans have separate channels for processing visual and auditory information
2. Limited capacity – William James – humans can only process so much information at one time
3. Active processing – meaningful learning occurs when people are actively involved: (a) selecting/attending – paying attention, (b) organizing – into a coherent representation that makes sense to you, (c) integrating – relate the new information to your existing knowledge

How can you foster integration given that people have limited working memory?

What about tool use and using the environment as cues to remembering and solving?

Research questions:

Multimedia Effect
-Does adding a graphic (to text) help? – Text alone vs. together Yes: great improvement in transfer test (multimedia effect) – graphic voice over was better than text

Contiquity Effect
-What makes a good graphic? separated vs. contiguous A: Integrated with text works better – locate text close to relative portions of images
(spatial or temporal contiguity effect)

Coherence Effect

Sometimes it is better to present less information than more. Seductive details – interesting but not related information (does it motive them more and make them more interested)
-Adding those details has a big negative effect on people’s transfer ability
-Adding background music depresses
-Reading summaries produced more learning than reading whole lessons
-Extraneous information gets in the way of making connections

Modality Effect
-People do much better when the words are spoken rather than printed
-When you present them in textual form it overloads the visual system. Presenting them auditorily offloads the processing to the auditory system
-Graphics and narration and text results in less learning than just graphics and

Personalization effect – conversational results in more learning than formal presentation

Pacing effect -

Signaling effect

Individual differences – low knowledge learners need better

Theory and research based

Clark & Mayer (2003) E-learning and the science of instruction

Mayer (2003) Learning and instruction.

Analysis of graphics in CA text books. Over 50% of the space was devoted to graphics. About of the 85% of the graphics were not instructional

Barbara White – Learning Physics in a game environment – learned better when asked to reflect afterwards

Games
Design a plant – that will live on another planet (for a environment
Dr. Fizz – Electromagnet
-modality and interactivity effects observed

-Motivational? Does adding a game aspect to instruction help learners persist longer?

Progression of mental models

Me: What about the component fluency hypothesis? He agreed that some things need to be automated to free up cognitive resources for conceptual processing

Future research interests:
-games and simulations (because of how motivating they appear to be to children)
-individual differences – cognitive style (verbal, auditory)
-online pedagogical agents (because they are forecasted to be the guide on the side in the future)
-What kinds of questions do people ask a tutor (when learning SPSS)?
-What kinds of answers do tutors give?
-Theory of tutoring?

Pavio & Battely – dual channel effect

What would a theory based on cognitive principles be like for:
-Tool use?
-Tool use instruction?
-Tool design?

Of course, like life is, I’m not any less busy now that I’ve defended. The only difference is that I now have 8 projects all more or less equally competing for my time instead of having one obvious first priority.

Bring to the foreground that there were two main purposes for my dissertation:

• Evaluate and refine TATSTAM – Study the process of how middle school math teachers reuse and adapt mathlets
• Evaluate and improve TADRIOLA – Develop a practical tool that teachers can use to author instruction

Add quotes and prompts for me to tell stories. My stories should include the turning points of my study. Also those things which were unexpected.

Similarly he suggested adding an Outcomes chapter to my dissertation. He thought the Discussion chapter should more be a reflection on my research methods and implications for further research.

He said that my committee will likely know everything I have to say and be mostly interested in hearing stories about what happened. Those stories can help convince them that the research is real, that it is original, and important.

He said that in qualitative, the researcher is the instrument. In reporting my findings I need to convince the audience that I am a good instrument for gathering data: (a) I listened and observed with fidelity, (b) I made good decisions going into the research, (c) I opportunistically shifted the focus of the research to follow up on new aspects of the research that opened up and to firm up my understanding of themes that emerged, (d) I modified instruments as needed to achieve the goals of my research.

His interpretation of what my research is all about went something like this. The context of my study is an ecology that includes middle school math teachers, classrooms, and students. Teachers lie along a continuum of wanting completely pre-packaged self-contained instruction that they can just instantiate and being Michael Angelo’s of math instruction wanting only the parts from which they will masterfully create their own instruction. I have created TADRIOLA, a tool that can be scaled and used by teachers at any point along that continuum. My study was to let this new organism TADRIOLA loose into the ecology and to study the reaction of the ecology to the organism. In which parts of the ecology was it embraced or not?

I told him that I hadn’t yet done any member checking. He said that in fact I had, by virtue of the iterative design of my research. He thought that was one of the strengths of the research design that I used. By repeatedly going back to the same type of people with preliminary findings is a type of member checking. He said I might as well send a high level view of my findings to participants to formally complete member checking. He agreed with me that more than likely they will not respond.

He indicated that the flow of my dissertation defense should be roughly as follows:

1. Quickly move through all of the intro leading up to the analysis in order to convince them that I’m organized and thorough.
2. Digress and take time telling the story of the research. Share quotes and stories. Talk about the decisions that I made. This is where I convince them that I am a good researcher and a good research instrument.
3. Wrap up with the outcomes and discussion.

I asked him about how to best include quotes in my findings. He said that what I shouldn’t do is include multiple quotes saying the same thing. I have the data and they can look at it if they want to. Present one quote saying one thing, then another perhaps contrasting comment, and then wrap up saying that approximately x% agreed with the comment.

When boiling down data to numbers, things like member checking, inter-rater reliability, and triangulation assume more importance when I am trying to make claims that strongly oppose existing theories. How attractive are competing alternative hypothesis? He said the strongest types of statements I want to say are that the majority of respondents said x and the rest were all over the map. The purpose of it is to convince them that I am a quality researcher.

Here is an approach I have dreamed up. I think it is a type of cluster analysis. Anyway, the first step is to code all textual data according to issues and propositions. Then create a metric to measure the distance between each pair of participants. The metric would work like this:

• If both participants made statements that supported a given proposition then don’t add anything to the distance score.
• If one participant made a statement that supported a given proposition and the other did not, then the add 1 (or some other magic numberto the distance score.
• If one participant made a statement that supported a given proposition and the other made a statment that opposed the proposition, then add 2 (or some other magic number) to the distance score.

I would weight each proposition according to how much of a factor I felt it was in determining the overall distance between two learners. Using a similar rubric, I could include some of my survey and background data in the calculation of difference scores.

This would result in distance scores for each pair of teachers. The hope is that clusters could be identified. If no clusters were observable, I could tweak the weights until I found some clusters (ha, ha). Really it seems like this is the process that one goes through when analyzing qualitative data. It is interesting that it seems more natural for qualitative researchers to try to come up with profiles or categories of learners than to try to describe average participant responses to a given issue. That is probably not so surprising, since it is often taken for granted that the sample is likely not representative. I think I will try to do both.

Reigeluth, C.M., & Frick, T.W. (1999). Formative research: A methodology for improving design theories. In C.M. Reigeluth (Ed.), Instructional-Design Theories and Models: A New Paradigm of Instructional Theory. (Volume II). Hillsdale, NJ: Lawrence Erlbaum Assoc.

Miles, M. B., & Huberman, A. M. (1984). Analyzing qualitative data: A source book for new methods. Beverly Hills, CA: Sage Publications.

Yin, R. K. (1984). Case study research design and methods. Beverly Hills, CA: Sage Publications.

• Look for big findings first.
• See if you can detect archetypes teachers from the data.
• The quantitative data I gathered is likely to be very noisy.
• I should work from a theory of learning and instruction, not technology.
• Finishing my dissertation needs to be a primary concern.

Look for big findings first
Clearly this only makes sense, though I probably would have not come up with that. Why give each bit of data and topic mentioned equal weight. He suggested that I just take each packet of information and sort them into four piles, not even being specific about what the piles are. I’m in the process of enterring all of the data into the computer and so I probably won’t go through the physical steps of what he described. I will do the equivalent electronically though. This leads to the next comment…

See if you can detect archetypes teachers from the data
My initial plan was to consider each of the tasks, barriers, and guidelines separately and look at the data I had gathered seeing what it had to say about each. In retrospect, this is probably a very quantitative approach. I would probably be able to report statistics about the percent of participant statements that gave evidence or support for each of the elements. The approach he suggests contrasts heavily with this. His approach is to look for archetypes into which I could group my participants. Each archetype description would obviously not completely describe each participant I placed in that category. However it is perhaps a good way to think about it.

Example archetypes could be: (a) teacher sees this and immediately envisions how it could be used, (b) teacher likes what they see, but needs some hand holding, (c) teacher wants something handed to them that they could use out of the box, (d) teacher would never touch this stuff.

The pragmatic usefulness of identifying archetypes might be to help give direction to where developers (I) should spend future development time and money. For example, I could focus on creating additional teacher support and integration with existing standards and textbooks, or I could just focus on developing newe mathlets.

After I have identified archetypes I could then move to analyzing parts of the archetypes.

I recently read a writeup of a closely related study. In doing that I was bit concerned about the lack of reporting of numerical data along side of provided anecdotal quotes. Because I’m pretty new to qualitative research, I was not clear as to how you should present your data. He indicated that he uses the following approach:

1. Present a statement of a theme
2. Give representative quotes to back it up
3. Indicate some percentages about the number of participants or statements that concur or disagree

Another method I have seen mentioned in the literature is to present and subsequently discount alternative explanations of the data. One thing that Dr. Lancy recommended is to make sure that any claims I make can be supported the data. I guess this is obvious advice, but from some of my readings, not always followed.

The quantitative data I gathered is likely to be very noisy
I had a difficult time getting teachers to comment on issues like: In what ways would you like to reuse and adapt interactive online resources? and What features would you like in an authoring tool? I’m guessing that this is because they have had relatively little experience in these areas. So what I chose to do was present them with lists and descriptions and asked them to rate items. This resulted in likert scale and ranking survey data. Lancy thinks this data will likely be very noisy if I try to look at distributions across all participants. I agree, it will be interesting to see if looking at the data by archetype makes it less noisy.

I should work from a theory of learning and instruction, not technology
Dr. Lancy’s impression is that many of us Instructional Technology types tend to focus more on the technology than on the learning. He pointed out that there is already significant literature on teacher adoption of innovation and that we should not expect their approach to adopting this innovation to vary greatly from other types of innovation. He suggests grounding theory and interpretation of data in the context of that literature.

Finishing my dissertation needs to be a primary concern
A number of times during our discussion, I started talking about how I could gather some additional data or do this or that. Each time, Dr. Lancy brought me back to reality, reminding me that I need to finish. His most memorable statement was “Joel, we don’t want you to retire on a graduate student salary.” I think my wife would resonate to that statement

Surely he is being both sarcastic and realistic. I guess when writing a blog you have to decide who you are writing it for. Surely when you start like I have just done, you have to be doing it for yourself or you will likely be deeply dissapointed.

I’ve tried to look around and see who in my world of interest is blogging and I don’t see a lot. Areas of interest for me are:

• Math and science education
• Public school teacher reuse and adaptation of online resources
• Authoring tools
• Using simulations in instruction
• Intelligent tutoring systems
• User interface design
• Sequencing problems for instruction
• Instructional feedback

Clearly I don’t have a good idea of how to look and haven’t reviewed a small fraction of what is out there. If you have pointers to blogs highly relevant to these topics, please share.

Many thanks to my Ph.D. chair David Wiley for getting me started.

My primary interest is in completing my dissertation study titled Design Theory for Authoring Tools that Support Teacher Adaptation of Mathlets (TATSTAM).

While my wider interest is the reuse and adaptation of all kinds of interactive online resources, for the purpose of my study I have narrowed my focus to middle school math teacher reuse of mathlets.

The three areas of investigation of my dissertation study are:

• In what ways do middle school math teachers want to reuse and adapt interactive online resources?
• What barriers do middle school math teachers encounter when trying to reuse and adapt interactive online resources?
• What design guidelines support the development of authoring tools that support teacher reuse and adapt interactive online resources?

As part of the study I’m developing a web-based authoring tool called TADRIOLA. Following a slightly adultrated formative research methodology (Reigeluth & Frick, 1999) TADRIOLA is an instance of the initial version of TATSTAM that I created by synthesizing the literature and personal experience. My main deviance from “orthodox” formative research methodology was to use it to develop software design theory as opposed to instructional design theory.

A simple explanation of what I have done is:

1. Review the literature to identify ways that teacher want to reuse and adapt interactive online learning resources a well as barriers that they encounter.
2. Propose a design theory for authoring tools that support teacher reuse and adaptation of interactive online learning resources (TATSTAM).
3. Design a web-based authoring tool (TADRIOLA) that is an instance of the design theory.
4. Conduct focus groups with middle school math teachers to gain empirical evidence about teacher wants, barriers, and the “goodness” of TADRIOLA and TATSTAM.
5. Analyze data and revise TADRIOLA and TATSTAM based on the analysis of the data.
6. Repeat.

I’ve utilized a case study methodology (Yin, 1984) and analyzed data using qualitative coding methods (Miles & Huberman, 1984).

So far I’ve conducted focus groups with middle school math teachers at 6 different locations. At the first location I worked 1-on-1 with 7 teachers using questionnaires, structured interviews, and user tests. At the other groups I have worked with groups using surveys, questionnaires, group discussions, and user tests. In all I have worked with 59 middle school teachers and some 20 other types including technology specialists and college professors who teach students preparing to be math teachers.

Reigeluth, C.M., & Frick, T.W. (1999). Formative research: A methodology for improving design theories. In C.M. Reigeluth (Ed.), Instructional-Design Theories and Models: A New Paradigm of Instructional Theory. (Volume II). Hillsdale, NJ: Lawrence Erlbaum Assoc. [PDF]

Miles, M. B., & Huberman, A. M. (1984). Analyzing qualitative data: A source book for new methods. Beverly Hills, CA: Sage Publications.

Yin, R. K. (1984). Case study research design and methods. Beverly Hills, CA: Sage Publications.