There are basically four types of software tools that can be used for developing instruments to measure learning. There are tools that are dedicated measurement-development tools, for example Questionmark's Perception. There are e-learning authoring tools that offer an assessment-development capability, for example Adobe's Captivate. There are learning content management systems, for example, Blackboard's Academic Suite. And finally, there are general purpose software-development tools, for example, Adobe Flash Professional. To put into a list form: Dedicated Assessment-Development Tools E-Learning Authoring Tools Learning Content Management Systems General-Purpose Software-Development Tools When we asked e-learning professionals from the eLearning Guild membership about their use of tools in developing learning-measurement instruments, they told us an interesting story. Specifically, we asked them, "What PRIMARY tool do you use to develop your measurement instruments?" The most popular two answers were (1) we didn't use a tool, and (2) we used a tool developed in-house. See the graph below. When we broke this down by corporate and education audiences, and looked at product market shares, other interesting findings appear. Take a look at the corporate results, excluding all education and government respondents: Adobe's Captivate dominates with over 50% of the market share—that is, over 50% of respondents said they used Captivate to develop their measurement instruments (they may have used other tools). Even more telling is that six of the top seven items are authoring tools or part of authoring tool suites. You read that right. Authoring tools are by far, by far, by far the way people develop assessment items in the corporate e-learning space. Only Questionmark's Perception and Adobe's Flash Professional sneak into the top nine responses before "Other" takes the tenth spot. This makes sense if we assume, like a dismal economist, that people do what is easy to do. Our authoring tools remind us to add questions, so we add questions. It also tells me that maybe our field puts very little value on measuring learning if our behavior is so controlled by our surroundings that we don't look further than our authoring tools. Or, could it be that our authoring tools provide us with all we need? Let's take a look at the education (with some government) results. (Note to those using the eLearning Guild's Direct Data Access capability: I filtered only for students, interns, academics, and practitioners). The results for the education results are interesting as well, especially as compared with the corporate results. Note how many dedicated-assessment tools in the top ten. There are three (Respondus, StudyMate, and Questionmark's Perception). So perhaps educators care a little bit more about testing. Okay, that makes sense too. Still, there are a lot of e-learning authoring tools at the top, with Captivate dominating again. The Leverage Point The clearest conclusion I will draw from this data is that to improve our e-learning assessment practices, we need to do it at the one clear leverage point—at the one point that we seem to think about measurement the most—in our authoring tools. How might this work: Okay, we could just train people to create better measurement instruments with the idea that they'll use that information the next time they boot up their authoring tool. Better would be to train them to create better measurement instruments while they are using their authoring tool. And give them practice as well, with feedback, etc. You learning researchers will be chanting "encoding specificity" and "transfer-appropriate processing" and those of you who have ever had one of my workshops on the learning research will be thinking of "aligning the learning and performance contexts" to "create spontaneous remembering." Better would be to develop job aids indexed to different screen shots of the authoring tool. Better would be for the authoring tools to be seeded with performance-support tools that encouraged people to utilize better measurement practices. Oh crap. The best way to do this is to get the authoring-tool developers to take responsibility for better measurement and better product design. Entrepreneurial minded readers will be thinking about all kinds of business opportunities. Hey Silke, how about giving me a call? SMILE. Not much of this is going to happen anytime soon, is my guess. So, besides engaging someone like me to train your folks in how to create more authentic assessments, you're pretty much on your own. And we know that's not going to happen either. At least that's what the data shows. Hardly anybody brings in outside experts to help with learning measurement. I guess somebody thinks it's just not that important. More on this as the series continues… The data above was generated by a group of folks working through the eLearning Guild. The report we created is available by clicking here. Here's some more detail: The eLearning Guild Report The eLearning Guild report, "Measuring Success," is FREE to Guild members and to those who complete the research survey, even if not a member.  Disclaimer: I led the surveying and content efforts on the research report and was paid a small stipend for contributing my time, however, I will receive nothing from sales of the report. I recommend the report because it offers unique and valuable information, including wisdom from such stars as Allison Rossett (the Allison Rossett), Sharon Shrock, Bill Coscarelli, (both of Criterion-Referenced Testing fame) James Ong (at Stettler Henke where he leads in efforts of measuring learning results through comprehensive simulations), Roy Pollock (Chief Learning Officer at Fort Hill Company, which is providing innovative software and industry-leading ideas to support training transfer), Maggie Martinez (CEO of The Training Place, specializing in learning assessment and design), Brent Schenkler (a learning-technology guru at the eLearning Guild), and the incomparable Steve Wexler (The eLearning Guild's Research Director, research-database wizard, publishing magnate, and tireless calico cat herder). How to Get the Reports    1. eLearning Guild Measuring Success (Free to Most Guild Members) If Member (Member+ or Premium): Just Click Here If Associate Member, Take measurement survey, then access report. If Non-member, Become associate member, take measurement survey, then access report.     2. My Report, Measuring Learning Results: Click through to My Catalog More tomorrow in the Learning Measurement Series...

In the eLearning Guild report I worked on with several other brilliant authors (SMILE), we asked e-learning professionals whether they were happy with the learning measurement they were able to do. Here's what they said (All the data reported in this blog post is for respondents who create e-learning for workers in their own organizations. The Guild's powerful database technology makes it possible to split the data in different ways). In general, are people able to do the learning measurement they want to? See the graph below. Only about 17 percent were happy with their current measurement practices. About 73 percent wanted to be able to do MORE or BETTER measurement. Clearly there is a lot of frustration. In fact, one of the top reasons people say they can't do the measurement they want to is they don't have the knowledge or expertise to do it. It's in a virtual dead heat for the third most important reason given. See the diagram below. The question then becomes, if people don't have the expertise to do measurement they way they want to, do they hire expertise from outside their organizations? A full 88.8% said they did all their measurement themselves. Wow! The graph below could not be more striking. When we asked people what kind of expertise they do utilize—whether it was in house or contracted—they told us the following (I added a color-coded legend at the top): Most of the folks doing learning measurement are instructional designers and developers with no particular expertise in measuring learning. A full 84% of respondents indicated that non-expert instructional designers are doing measurement at their organizations. Only 51.7% of respondents said their organization uses instructional developers with some advanced education. Only 20% of organizations have masters-level degrees in measurement. Only 6.7% utilize doctoral-level experts. Note that when we look only at organizations that claim to be getting "high value" from doing learning measurement versus all the others, the results are intriguing. Respondents Reporting the Level of Value They got from their Measurement Efforts Less Than High Value High Value for Measurement Percentage of Respondents saying they Utilize People with MASTERS DEGREES ON STAFF. 16.4% 31.4% Percentage of Respondents saying they Utilize People with DOCTORATES ON STAFF. 5.6% 10.1% Percentage of Respondents saying they Utilize People with MASTERS DEGREES HIRED FROM OUTSIDE. 3.7% 11.8% Percentage of Respondents saying they Utilize People with DOCTORATES HIRED FROM OUTSIDE. 3.4% 6.1% Wow!! Those folks who think they are getting very high value for their measurement efforts utilize people with masters degrees on staff more than 91% more compared with those reporting less than high value for their measurement efforts. The high-value people also utilize more doctoral degrees on staff compared to the non-high-value people by 80%, more masters degrees hired from outside by 219%, and more doctoral degrees hired from outside by 79%. While this data is correlational and self-report data, it suggests some sort of relationship between the measurement expertise employed and the level of value an organization can get for their learning measurement. Summary To recap, in a survey of over 900 e-learning professionals, many are frustrated because they want to do more/better measurement. A significant portion of their frustration results from not having the expertise to do measurement correctly. They have very few high-level measurement experts on staff, and they hire almost nobody from the outside to help them. What the hell is wrong with this picture? It confirms for me that learning measurement is just not given the importance it deserves. More to come tomorrow in the Learning Measurement series...

I'd like to announce the winners of the 2007 Neon Elephant Award, given this year to Sharon Shrock and Bill Coscarelli for advocating against the use of memorization-level questions in learning measurement and for the use of authentic assessment items, including scenario-based questions, simulations, and real-world skills tests. The Neon Elephant Award is awarded to a person, team, or organization exemplifying enlightenment, integrity, and innovation in the field of workplace learning and performance. Announced on the day of the winter solstice—the day of the year when the northern hemisphere turns away from darkness toward the light and hope of warmer days to come—the Neon Elephant Award honors those who have truly changed the way we think about the practice of learning and performance improvement. Award winners are selected for demonstrated success in pushing the field forward in significant paradigm-altering ways while maintaining the highest standards of ethics and professionalism. See the full announcement by clicking here...

The eLearning Guild has asked me to lead a discussion at their upcoming conference (Guild Annual Gathering) on how we might think about evaluating Learning 2.0 interventions. I'd welcome your examples and insights. For those who don't know what "Learning 2.0" means, I'll forgo my cynical answer, and say that others describe Learning 2.0 as learning that enables learner-creation of information, comparing Learning 2.0 to the stereotypic traditional model where the teacher teaches and the learner absorbs the information (or the e-learning delivers content and the learner absorbs). So, Learning 2.0 is said to include such things as Wikis, Blogs, Learner Portfolios, Media Development and Sharing by Learners, Informal Learning, etc. Here's a few things I'm contemplating: Traditional metrics are certainly appropriate because bottom-line we want to know if Learning 2.0 interventions produce learning, enable on-the-job performance, and produce desirable individual and organizational outcomes. Comparisons to other methods of learning. Especially important to see if Learning 2.0 methods (on the positive side) create more elaborate mental models, produce more satisfaction, etc. and (on the negative side) waste time, create unproductive distractions, communicate incorrect or inappropriate information. We need to measure not only what HAS been learned, but also on what MAY BE LEARNED IN THE FUTURE. It could be, for instance, that Learning 2.0 is inefficient for learning anything specifically, but enables faster future learning in the same area of inquiry. Here's where I can use your help. Let me know if you know of any of the following: Rigorous research studies on Learning 2.0 interventions Anecdotal evidence on Learning 2.0 interventions Better yet, join me at the eLearning Guild's Annual Conference -- Specifically the Learning Management Colloquim and discuss this in real time.

MIT researchers have developed a technology to track people's social interactions, for example, at a conference. Check out this link to learn more. Can we use such a technology for learning? Certainly, we could use the technology to help people learn about their current networking tendencies and to give learners feedback as they attempt to change those tendencies. But, what other applications can we brainstorm? Let me give this a try. Leadership Simulations: Does the technology enable better in-basket simulations, or in-basket simulations that are more economical to deploy (because they don't require the same high numbers of observer/consultants to observe interactions and provide feedback? Note: By in-basket simulations, I mean simulations in which many learner/players each play a different role, each have different in-basket tasks to accomplish, and the way they act in the simulation is by talking with other learner/players. On-the-job Leadership Activity Feedback: Imagine a retail store manager who is tasked (partially) with developing his or her people (those who work in the store). The system could track the number of interactions the store manager had with each employee, and the interactions the employees had with each other. This "intelligence" data could be used by a store manager to learn about the number of learning opportunities (i.e., coaching, providing feedback, observing, encouraging, sharing, etc.) that occur in a given period of time. Such data could be compared with "best-practice" store manager data, and store managers could use this information to change their behavior. Admittedly, quantity doesn't equate to quality, but by tracking such social contact, managers might get a start in thinking about increasing the number of "learning opportunities." Organizational Learning. Organizations (or business units, teams, etc.) could track each other's social networks to find out who the most networked folks are. Such information could be utilized to select for job assignments, project roles, etc., or to actively change the observed dynamics (for example, encouraging some people to spend more time in individual productive work while encouraging others to limit their isolation). Anyway, these are some initial thoughts. I expect some enlightened simulation companies to begin brainstorming ways to use the technology to differentiate their offerings from the competition. In the meantime, can you think of any other learning opportunities inherent in the technology?

Starbucks is shutting all its stores on February 26th for three hours to train or retrain its employees. You can read related articles at the following links:Seattle Times Starbuck Gossip (Blog) Marketplace (Audio, story starts at 19:42 mins:secs) Street InsiderThe question will be, is the training well-designed? Wednesday morning February 27th will give only a partial answer. March, April, and May will be more important. And of course, maybe this has nothing to do with training at all. Maybe it's a store management problem. Maybe the brand is too diluted, no longer special. Maybe competition from Dunkin' Donuts and McDonalds is creating issues, especially as consumers try to save money in these uncertain economic times. Will the training work? Stay tuned.

Elliott Masie came up with a great and very insightful wish list for LMS's. Click here to access it. He even added a few suggestions in the past few days, probably based on feedback from his loyal audience. I really like the richness that Elliott's suggestions might create for a typical LMS. Most LMS implementations are just a list of course offerings. On the other hand, I worry about overly complicating options for users. Most workers just don't have extra time to waste. Maybe the suggestion to let users rate the courses comes into play here. I also worry about user-generated content. It can be great, could be better than what the training folks can create, could engender more engagement, could be bottom line more effective. But we should all recognize that it is a double-edge sword. User generated content could be incorrect, could be a huge waste of time, could cause the organization to leave itself vulnerable to legal liability. Doesn't Fix the Biggest Problem with the LMS Mentality The biggest problem with LMS's can't be fixed with Elliott's suggestions. The biggest problem is that the whole LMS face sends a powerful hidden message that "learning" is about taking courses or accessing other learning events. This "Learning Means Sitting" LMS mentality infiltrates whole organizations. I've seen this recently with one of my clients, a huge retailer, where their LMS has encouraged store managers and other store leaders to focus learning time on taking courses, in lieu of coaching, learning from each other, trying things out and getting feedback, encouraging store employees to take responsibility for particular areas, etc. It's not that they completely ignore these other learning opportunities; it's that the LMS focuses everyones' time and attention on courses, creating a lot of wasted effort. To get the most from an LMS, you ought to throw away your LMS and start over. People can learn something—develop competencies/skills—from courses or from other means. A competency-management system that offers multiple means to develop oneself is ideal, where courses/events are just one option. I still haven't seen a commercial system that does this though...Most are course first designs. Maybe I'm too over-the-top recommending that we get rid of all LMS's. I make the statement to highlight the humongous problems that the LMS mentality is causing.

I recently completed one of the most comprehensive work-learning audits I've ever been asked to do for a major U. S. retailer. The goal of the audit was to find out how their learning programs AND work-learning environment were supporting the stores in being successful. The audit involved (1) structured and unstructured interviewing with all levels of the organization, especially with store personnel, (2) focus groups held across the country with specific groups of store personnel (e.g., clerk, store managers, assistant managers, etc.), (3) task force meetings with senior line managers and representatives throughout the company, (4) learning audits of e-learning courses, (5) learning audits of a mission-critical classroom course, (6) review of company artifacts (CEO messages, publications, databases, intranet, etc.), (7) interviews with learning-and-performance professionals, (8) discussions of business strategy, (9) discussions regarding corporate information and data-gathering capabilities, (10) job shadowing, (11) store observations, etc. Who/What Do Workers Learn From? One of the most intriguing results came out of a relatively simple exercise I did with focus-group participants. The following is a rough approximation of those results. What I did was ask focus-group participants who they learned from. I would hold up a large 6 x 8 index card with a position label on it, for example, "District Manager," "Clerks," or "Corporate." The group would shout out where they thought that card should go on a large diagram I had created on the wall. I would place it on the wall in a particular category based on the verbal responses and then we would negotiate as a group to determine it's final positioning. So for example, participants could say that they learned the following amounts from that person/position, and we often compromised using in-between placement: Learned Most Learned a Lot Learned Some Learned a Little Learned Least Had Little/No Contact with See the diagram below for a rough example. This one is actually a composite based on several focus groups and more than one position. It gives a fair representation for how frontline retail clerks responded. Note that the orange boxes represent fellow employees, while the blue boxes represent other groups of people or things that they learned from. There are several key insights from these results: People learn the most from those who they work closely with. People learn the most from their experience doing the job. People learn the most from their self-initiated efforts at learning. The more contact, the more learning (for the most part), however there are benefits from learning from experts (e.g., store managers, head clerks), though the worker has to have at least some signicant contact with them to create this benefit. You'll notice that district staff have only a little impact and regional and corporate staff have none. E-learning is seen as somewhat facilitative but not a place where workers learn the most. This result may be organization specific as different e-learning designs and implementations might easily move this result higher or lower. Frontline clerks didn't get much from company magazines and the like, but managers (not represented in the results above) did find value in these. Store managers also reported that networking with other store managers was on of the "Learn Most" entries for them. For this company, this network was even more important than learning from their district managers (their direct bosses). This makes sense because their network is more accessible throughout the heat of the daily grind. These results were eye-opening for my client, and they are still wrangling with the implications. For example, district managers and district training staff seemed to produce very little learning benefits. So, should their roles for learning be de-emphasized or re-emphasized? These types of result have to be understood in the larger data-gathering effort of course. Analyzed alone, they suffer from the problem of de-contextualized self-report data. Combined with multiple other data sources, they paint a really robust picture of an organization's learning environment. Informal Learning, Social Networks, etc. Vendors are out and about in our field now selling the benefits of complicated and expensive analysis tools for looking at how people learn through so-called informal on-the-job mechanisms. The example above shows that if you don't have the big bucks, there are simpler ways to get good data as well. 

I'm writing you from the eLearning Guild's annual conference. I went to a session presented by Silke Fleischer and colleagues at Adobe and was blown away by the work Adobe is doing to create products that support learning and related efforts. I then asked a number of industry thought leaders who confirmed my interpretation: Adobe is now a 500-pound Gorilla, likely to continue out-investing their competitors and thus creating better and better products for folks like us to use. If you're considering elearning tools, you owe it to your organization to consider Adobe products. I have no financial relationship with Adobe, by the way. This is not to say that other products aren't worthy and/or do some things better than Adobe products. My thinking is this: Companies who invest in their products are often more likely to be there for you in the years to come. I've seen many clients who started using a particular tool five-to-ten years ago, and they are basically stuck with it because of their large installed base of learning courses. Here are a few of the things that made me wake up and take notice: Adobe's update cycle on Captivate seems to be shrinking, as they are aggressively moving forward in the development of Captivate 4. Captivate is being used for many purposes, including the development of Podcasts, Advertising, etc. You can embed a working Captivate file into Adobe connect and then have webinar or online-learning participants each interact with Captivate objects. PDF files can now include fully-functional interactive images. So documents are not static anymore!! Adobe is working on a new platform called AIR, which will enable the compilation of many types of objects for display and interaction.

Call me crazy, but I think it's important to invest in the research base for our field. I've spent a good chunk of the last year reviewing research from the world's preeminent refereed journals in regard to how to give learners feedback. I've created, I'd like to think, the seminal research review on how to give learners feedback, written in a way that puts feedback in perspective, that goes deep into the fundamentals to give readers clear mental models for how feedback works. It's this kind of in-depth exploration that allows you as a learning professional to use your wisdom to make the difficult design tradeoffs that you have to make. Recipes are for short-order cooks. Research-based wisdom for learning professionals is much more useful in the gritty day-to-day of our learning shops. And now, instead of selling this document, I'm going to try an experiment, and give it away. Call me crazy, but internet users (hey that's us) just don't like to pay. I've been swimming upstream against the movement toward free information, knowing that the information I'm compiling is the best information out there, and that it takes an incredibly exhaustive effort to sift through refereed research, make sense of it, and repackage it in a way that resonates and is practical. But maybe research karma will work. It's worth a try, right? You can help me by reading the report, AND if you think it's good, sending the link to it to everyone you know in your organization, in every learning-development organization you know, to your mom, your kids, your elected officials, to Elliott Masie and Tony Bingham (CEO of ASTD), to the New York Times. Special thanks go out to my friends at Questionmark who agreed in advance of me finishing the report to license it for their clients and learning community. Questionmark is providing a great service by making it possible to dessimate world-class research-based information that is both valid and useful, including their support of the aforementioned research report on feedback. Here are some of the insights from the two-part 88-page research report: The most important thing to remember about feedback is that it is generally beneficial for learners. The second most important thing to remember about feedback is that it should be corrective. Typically, this means that feedback ought to specify what the correct answer is. When learners are still building understanding, however, this could also mean that learners might benefit from additional statements describing the "whys" and "wherefores." The third most important thing to remember about feedback is that it must be paid attention to in a manner that is conducive to learning. Feedback works by correcting errors, whether those errors are detected or hidden. Feedback works through two separate mechanisms: (a) supporting learners in correctly understanding concepts, and (b) supporting learners in retrieval. To help learners build understanding, feedback should diagnose learners’ incorrect mental models and specifically correct those misconceptions, thereby enabling additional correct retrieval practice opportunities. To prepare learners for future long-term retrieval and fluency, learners need practice in retrieving. For this purpose, retrieval practice is generally more important than feedback. Elaborative feedback may be more beneficial as learners build understanding, whereas brief feedback may be more beneficial as learners practice retrieval. Immediate feedback prevents subsequent confusion and limits the likelihood for continued inappropriate retrieval practice. Delayed feedback creates a beneficial spacing effect. When in doubt about the timing of feedback, you can (a) give immediate feedback and then a subsequent delayed retrieval opportunity, (b) delay feedback slightly, and/or (c) just be sure to give some kind of feedback. Feedback should usually be provided before learners get another chance to retrieve incorrectly again. Provide feedback on correct responses when:a. Learners experience difficulty in responding to questions or decisions.b. Learners respond correctly with less-than-high confidence.c. All the information learned is of critical importance.d. Learners are relatively new to the subject material.e. The concepts are very complex. Provide feedback on incorrect responses:a. Almost always.b. Except:i. When feedback would disrupt the learning event.ii. When it would be better to wait to provide feedback. When learners seek out and/or encounter relevant learning material either before or after feedback, this can modify the benefits of the feedback itself. When learners are working to support retrieval or fluency, short-circuiting their retrieval practice attempts by enabling them to access feedback in advance of retrieval can seriously hurt their learning results. When learners retrieve incorrectly and get subsequent well-designed feedback, they still have not retrieved successfully; so they need at least one additional opportunity to retrieve—preferably after a delay. On-the-job support from managers, mentors, coaches, learning administrators, or performance-support tools can be considered a potentially powerful form of feedback. Training follow-through software—that keeps track of learners’ implementation goals—provides another opportunity for feedback. Feedback can affect future learning by focusing learners on certain aspects of learning material at the expense of other aspects of learning material. Learners may take the hint from the feedback to guide their attention in subsequent learning efforts. Extra acknowledgements (when learners are correct) and extra handholding (when learners are wrong) are generally not effective (depending on the learners). In fact, when feedback encourages learners to think about how well they appear to be doing, future learning can suffer as learners aim to look good instead of working to build rich mental models of the learning concepts. Some of the concepts and language in the above recommendations may not be obvious until you actually read the research report. You can do that by clicking the link below. The link to download the feedback report