I just came across another sighting of the mythologic numbers of memory retention, this time on the webpage of HRDQ. Take a look: They claim that, "Research shows people remember only 5% of what they hear in a lecture. But they retain 75% when they learn by doing." Bulloney!! If you want to read my full debunking, click here. If you want to see many bogus sightings, click here and scroll down.

In today's New York Times there is a great article, Who's Minding the Mind?, by BENEDICT CAREY that sums up a large number of research studies on human cognition that show that human beings are more reactive than we might think. We tend to believe that we, as human beings, are very proactive and consciously in control of our thoughts and actions; but these studies show that much of what we do and think is due to hard-wired, often unconscious processes. For example, the article cites how sitting near a briefcase (as opposed to a backpack) can make people more competitive. Or as Carey writes:New studies have found that people tidy up more thoroughly when there’s a faint tang of cleaning liquid in the air; they become more competitive if there’s a briefcase in sight, or more cooperative if they glimpse words like "dependable" and "support" — all without being aware of the change, or what prompted it.This basic fact about human behavior is relevant to the learning and performance field, of course. One of the things I've talked about for years is the notion of "spontaneous remembering." If we create learning right, we're more likely to help our learners—when they're on the job at a later time—by helping them spontaneously trigger memories of what they've learned. We can do this best by requiring our learners to utilize realistic cues in the learning context in making real-world decisions and taking real-world actions. This is why simulations are so effective (if they are well designed). When learners process learning objectives or prequestions before encountering learning material, the learners are primed to pay attention to relevant learning material. It's not necessarily a conscious process, but it works. There are many examples available, but here's another point: The learner-centric movement of the 1990's and 2000's has relied too heavily on the notion that the learners always know best, and that they are in conscious control of their learning and we just need to let them make the best decisions. When we realize that our learners are more deterministically driven than the we want to believe (its about free will a little, isn't it?), we have more work to do if we really want to drive maximum performance. Even when our clients consciously want to do something, we may be able to help them reach their goals by setting up learning and performance situations that unconsciously trigger the behavior they want to achieve.

It has been exactly one year since I offered $1,000 to anyone who could demonstrate that utilizing learning styles improved learning outcomes. Click here for the original challenge. So far, no one has even come close. For all the talk about learning styles over the last 15 years, we might expect that I was at risk of quickly losing my money. Let me be clear, my argument is not that people don't have different learning styles, learning preferences, or learning skills. My argument is that for real-world instructional-development situations, learning styles is an ineffective and inefficient waste of resources that is unlikely to produce meaningful results. Let me leave you with the original challenge: "Can an e-learning program that utilizes learning-style information outperform an e-learning program that doesn't utilize such information by 10% or more on a realistic test of learning, even it is allowed to cost up to twice as much to build?" The challenge is still on.

I've just spent 3 wonderful days in San Jose at the eLearning Guild's DevLearn 2007 conference. Here were some of the highlights for me: Hanging out with Ruth Clark a few times during lunches, keynotes, etc. We had a blast discussing research, the state of the profession, and the joy and challenge of doing the research-to-practice thing. Seeing Ruth Clark and Silke Fleischer (of Adobe Systems) present the research AND practice of Richard Mayer's work on multimedia learning. Silke did a nice job of demonstrating e-learning examples in Adobe's Captivate. Ruth did a wonderful job discussing the research, framing it in terms of practical application, and describing its limitations. Adobe deserves a ton of credit for supporting the dissemination of Ruth's work and helping to distribute it to a wide audience. Three cheers for enlightened companies like Adobe and Questionmark who support research dissemination. Having Google Maps change my behavior by including a public transportation option when I did a search in San Jose. I had actually made a reservation to rent a car so I could drive from my hotel to the downtown hotel where the conference was held. When I went to Google Maps to search for the best route, I was offered a public transportation search. I found out I could get downtown in 12 minutes for only $1.75. I cancelled my rental car and happily commuted. Saving me money (Google showed my gas savings), parking fees, aggravation, etc. Awesome!! Technology changes everything. Seeing a great keynote by Paul Saffo who talked about technology innovation and reminded my how many failures are required before success is rewarded. It was one of those rare keynotes that was both well-delivered and superbly relevant for the conference. Way to go Heidi Fisk (of the eLearning Guild) for a great keynote selection. Wonderful food. Yes. At a conference!! Healthy, fresh, tasty. Way to go Fairmont Hotel. The now-famous DevLearn DemoFest where dozens of e-learning developers show off their wares. It's a great way to take a snapshot of the state of the e-learning industry. Playing tennis with a Wii remote. My wife and I are not TV people, so I never pay attention to all the new remotes, Xbox's, etc. At the conference, I played tennis for about 2 minutes and had some fun. The cool thing about the Wii  is that it tracks the remote's movement and simulates that on the screen. Actually, with the Wii my first serve percentage was about 100%, much better than real life. My Wednesday Breakfast Bytes session on the intersection of e-Technology and Informal Learning. We had a great conversation and I learned some things. My Thursday Breakfast Bytes session on Situation-Based Instructional Design. The basic nugget is that people behave by (1) Being in a situation, (2) Evaluating that situation to make sense of it, (3) Deciding what to do, and (4) taking Action. So, we ought to give our learners practice in doing the whole SEDA process (Situation-Evaluation-Decision-Action). And, we can benefit from asking the Magic Question. Yes, there is more to it than that. The bottom line for me is that my clients have found the concept very helpful in helping them design learning that goes beyond the typical topic-based designs. The DevLearn Breakfast Bytes sessions do a great job in getting conversations going. As always, Guild members come to the conference with experience and are ready to share their insights and wisdom. I love Breakfast Bytes. My regular session on Learning Measurement. Another great discussion with—what seemed to me—like lots of light bulbs going off. Fun, even after several days of conferencing. I met a guy at the Demo Fest—John D’Amours—who had actually tried to do a control-group experiment comparing his e-learning design to a traditional design. Yes. Yes. Yes. We need more folks taking this kind of initiative. Learning that Windows Vista NORMALLY runs slow with 2GB of memory. Glad it wasn't just my machine. AND so many other great conversations and sessions. Sorry if I failed to mention you!! Hugs to all. I really learn a lot at eLearning Guild events. And I have to say, I feel that my contribution is especially appreciated. Thanks Guild members and staff !!!!!!!!!!

In what has become an eternal vigil against the myth that "People remember 10% of what they..." I just hit the jackpot with the help of Jay Banks who just sent me an email. The Wikipedia entry for Edgar Dale had two incorrect references to the bogus numbers that I talk about so often (see my blog category Myths and Worse). I fixed it today, I hope for good. Here's what it looked like: And here's what it looked like in Wikipedia: For those who are shocked that information on the internet might be wrong—or that Wikipedia might be wrong—see my previous entries about Wikipedia (1st Most-Recent).

Starting in 2006, Work-Learning Research offered the learning-and-performance community the Neon Elephant award to celebrate and honor an effort of extraordinary importance to our industry. Last year's honor went to Cal Wick of the Fort Hill Company for his work pushing the field toward true training transfer, leading the development of a tool that supports transfer (Friday 5's), and writing a book (The Six Disciplines of Breakthrough Learning...) to highlight these ideas and insights. This year's award will once again be announced on the day of the Winter Solstice, to honor someone whose truly extraordinary work has helped bring light to our field. Stay tuned...

First of a Series This is the first of a long series of blog entries devoted to the topic of learning measurement that I will offer over the next two weeks. This series draws from my recent thinking on learning measurement and from my 2007 publication, Measuring Learning Results… It also introduces the findings from a remarkable research study that I participated in with the eLearning Guild and several other illustrious authors. For the last year, I have spent many weeks devoted to rethinking the topic of learning measurement from the standpoint of the learning research. My research-to-practice report, Measuring learning results: Creating fair and valid assessments by considering findings from fundamental learning research, highlights the flaws in the current methods we use to measure learning results—and offers recommendations for how to improve our measurement practices. This report is available on my catalog. See below. Why does Will Thalheimer Care about Measurement? Why do I—a person who has spent the last 10 years attempting to bring fundamental learning research into focus—want to spend my "research time" on learning measurement? Here’s why: The performance of the learning-and-performance field is severely deficient—often creating learning that is not remembered and/or not utilized on the job. Of the forces that control and influence our industry and the practices we use, measurement is one of the most critical. Currently, our measurement practices provide us with poor and biased feedback about our performance as learning-and-performance professionals. Because we do poor measurement, we don’t get good feedback (nor do our stakeholders), and so we have very little motivation to critically examine our practices—and improve them as valid feedback would suggest. To put it simply, if we don’t measure better, we will continue to underperform—and we’ll continue to underserve our learners and organizations. The eLearning Guild Report The eLearning Guild report, "Measuring Success," is FREE to Guild members and to those who complete the research survey, even if not a member. Also available, at $1,895 ($1,950 if you are not a member), is Direct Data Access (DDA) to the database of research results , including the ability to filter the results based on a variety of factors, including the survey respondents’ experience, industry, country, job title, etc. These Direct Data Access reports will be invaluable for vendors who want to know how well their products are rated on a number of dimensions (more on this later in this series), and valuable to for those who want to benchmark their efforts against other organizations that are similar to theirs. If you want to make a case for improving your measurement practices, you absolutely have to buy direct data access. Disclaimer: I led the surveying and content efforts on the research report and was paid a small stipend for contributing my time, however, I will receive nothing from sales of the report. I recommend the report because it offers unique and valuable information, including wisdom from such stars as Allison Rossett (the Allison Rossett), Sharon Shrock, Bill Coscarelli, (both of Criterion-Referenced Testing fame) James Ong (at Stettler Henke where he leads in efforts of measuring learning results through comprehensive simulations), Roy Pollock (Chief Learning Officer at Fort Hill Company, which is providing innovative software and industry-leading ideas to support training transfer), Maggie Martinez (CEO of The Training Place, specializing in learning assessment and design), Brent Schenkler (a learning-technology guru at the eLearning Guild), and the incomparable Steve Wexler (The eLearning Guild’s Research Director, research-database wizard, publishing magnate, and tireless calico cat herder). How to Get the Reports    1. eLearning Guild Measuring Success (Free to Most Guild Members) If Member (Member+ or Premium): Just Click Here If Associate Member, Take measurement survey, then access report. If Non-member, Become associate member, take measurement survey, then access report.     2. My Report, Measuring Learning Results: Click through to My Catalog The Series Continues Tomorrow...

Let me start out by saying that I don’t know everything about learning measurement. It’s a topic that is too deep and complicated for easy answers. My value-add is that I look at measurement from the standpoint of how learning works. As far as I can tell, this is a perspective that is new to most of our field’s discussions of measurement. This is ironic of course, because it’s learning measurement we’re talking about. So for example, when we know that learning begets forgetting, why is it that we measure learning before forgetting might even have an effect—thereby biasing the results ridiculously in our favor? The second unique perspective I’m adding to the conversation is the importance of predicting future retrieval. I argue that we must validly predict future retrieval to give us good feedback about how well our learning programs are working. We do an absolutely terrible job of this now. Finally, I’d like to think that I am pushing us beyond the conceptual prison engendered by our old models and methods. It’s not that these models and methods are bad. It’s that most of us—me included—have had a seriously difficult time thinking outside the boundaries of the models’ constraints. Let me use the Donald Kirkpatrick model as an example. Others may beat up on it, expand it, or expound on it for pleasure or treasure, but it’s a great model. It helps us make sense of the world by simplifying the confusion. But the model, by itself, doesn’t tell us everything we need to know about learning measurement. It certainly shouldn’t be seen as a prescription for how we measure. It is simply too crude for that. Three Biases in Measuring Learning at Level 2 Measuring Immediately After Learning. A very high percentage of learning measurement is done immediately at the end of our learning events (In the eLearning Guild research, we found about 90% of e-learning measurement was done right at the end of learning). Immediate tests of learning only make sense if you don’t care about the learner’s future ability to retrieve information. When we measure "learning" what we really want to do is measure "retrieval." Moreover, what we really care about is future retrieval. Isn’t that the goal of learning after all? We want our learners to learn something so they can retrieve it and use the information later. I detail this point in much greater depth in the Guild research report and in even more depth in my report, Measuring Learning Results… By the way, the Guild report is free to Guild members and to those who complete the survey. Measuring in the Same Context Where Learning Took Place. A high percentage of training is measured in the learning context (about 92% in the same or similar context in the Guild research). Unfortunately, context influences retrieval, and so when we measure in the learning context we bias our measurement results. Oh, and we bias them in our favor, again. So for example, in the classic research study, Smith, Glenberg, and Bjork (1978) found that when learners were tested in the same room in which they learned, they were able to recall more than 50% more than when they were tested in a different room from where they learned. This amount of bias is well within the bounds of the Barry Bonds level of cheating!! Would you vote yourself into the Hall of Fame? Measuring Using Inauthentic Assessment Items (like Memorization). Most assessment items purporting to measure learning use memorization questions. Asking learners to simply retrieve what they have learned is bad assessment design because memorization is generally unrelated to future retrieval. So, if we test memorization, we know nothing (or very little) regarding whether our learners will be able to retrieve what is truly important. Sharon Shrock and Bill Coscarelli, two of my co-authors in the eLearning Guild Research Report, highlight the problems of memorization in the third edition of their excellent book, Criterion-Referenced Testing… One of the goals of criterion-referenced testing is to determine whether a learner can be "certified" as competent or knowledgeable about a particular topic area. Schrock and Coscarelli argue that only assessments done on (a) real-world tasks, (b) simulations, and (c) scenarios can be validly used for certification decisions, whereas memorization cannot be used. This is a change from the second edition of their book and provides a paradigmatic shift in our field. In future posts in this series, I will highlight my taxonomy of authenticity for assessment questions that follows up on Schrock and Coscarelli’s thoughtful certification ideas. The Series Continues Tomorrow... References: Shrock, S. A., & Coscarelli, W. C. (2007). Criterion-Referenced Test Development: Technical and Legal Guidelines for Corporate Training (Third Edition). San Francisco: Pfeiffer. Smith, S. M., Glenberg, A., & Bjork, R. A. (1978). Environmental context and human memory. Memory & Cognition, 6, 342-353.

To lead into the weekend, let me hot-wax poetic: Measurement is like a magnetically alluring supermodel we might see across the room at a party. We want to stare and absorb every curve of muscle, every glowing inch of skin. Yet, our primordial core forces our eyes away, perhaps ashamed of our own imperfections, perhaps following some failed inner calculus of future possibilities. The apparition seems impossible to grasp, so we turn away. With another opportunity lost, learning measurement keeps its mystery, its danger, and its transcendent ability to lift our practices to their highest potential. Add your comments to analyze the paragraph above. What do you think I'm trying to say about the state of learning measurement in our industry? About our reasons for failing in this regard? What am I missing? What am I saying about supermodels? Add your own purple prose, poetry, etc., using the comments function below. Note: Both men and women can be supermodels. This quote comes directly out of the eLearning Guild Report. Measurement Series Continues on Monday...

What can e-learning add to measurement? What can e-learning add to measurement? Does e-learning have unique capabilities that enable it to improve learning measurement? I think it does. Here’s a short list: E-learning can capture more data more easily than classroom training. E-learning can capture data during the learning program—not just at the end of the learning event—in a manner that the learners feel is a seamless and natural part of the event. E-learning can track incoming proficiency through the use of pretests to determine whether the learning program actually meets a need, or determine who it meets a need for. E-learning can collect data in a manner that can give learners comparison data while they complete an assessment. E-learning can collect data on learner behaviors during the learning (for example, the click journey, time per screen, etc.) E-learning can track pretest to posttest changes. E-learning can randomly assign learners to program versions, making methodological comparisons possible. For example, a program version that uses immediate feedback can be compared to a program version using delayed feedback to determine which method is more effective. E-learning can capture on-the-job performance data, including learners’ self-ratings, manager ratings, direct-report ratings, etc. This capability puts the focus on on-the-job performance, where benefits can accrue from management oversight and coaching, self-initiated development, and peer learning. E-learning, because it can access and track learners at more than one point in time, can measure how well the learning intervention has performed in creating long-term remembering. E-learning can capture data even when learners don’t know the learning program is being assessed. For example, the learning program can capture data when the learners think they are simply getting practice on the learning material. E-learning can track learners as they move from the training event to the workplace. For example, e-learning programs can track learners’ goals to implement what they have learned to see how successful they have been in transferring the learning to the job. With this power, comes responsibility, and a damn fun challenge. You can read my call-to-action later in this series and in the e-learning Guild Research Report as well. The Measurement Series Continues Tomorrow...