Making the Grade: Quality Assurance for eLearning Design at Penn State

Amy Garbrick, Penn State University

Already excited with this session because of the handouts provided: a course review worksheet, an inspector input worksheet and a whole thing on eLearning design standards. Hallelujah! Finally something I can use to make our courses better. I've been talking about trying to apply standards, but could never find standards to adhere to.

This session is all about how PSU developed quality standards, implemented them and the process of how they did it all.

The problem at PSU:

• course design & development varied among colleges, designers, ect. No one standard. Some developed by faculty, some by teams, some new, some old.
  
• Can PSU decide on design standards (visual).

PSU QA standards based on Quality Matters.

QA Standards are a set of 12 standards and determining what metrics will be measured to know if courses have met these standards, as well as developing best practices for course design. PSU went through a lengthy process to get the results they have today.

Quality Assurance documents at PSU:

• The Standards
• Instructor Input Form (for course review)
  completed by course faculty developer to provide additional information about the course
  
• Course Review Worksheet

They can be used for a new course, to review an existing course, to provide feedback to IDs.

While quality and consistence has benefits, there are some challenges are on flexibility, and internal 'political' hurdles.

Group Work to assess a PSU course based on one of the 12 standards. We're checking #9 Learning Activities and Assessment.

Biggest eyeopening moment: what corporate sees as interactive is very different than what higher education sees as interactive.

The group activity really didn't seem as effective as I expected. The focus of this was geared more towards higher education training and I'm on the corporate side. I think the best thing I got out of this session are the handouts. I would have been more excited if the session covered how PSU came up with the standards or presented a walk-through on how they applied the QA on a particular course.

I started to glaze over when the conversation turned to Moodle, Blackboard, type of course management systems. Again, not corporate based so less relevant to me.

Hopefully the next session is better for me.

The Powerful Webcomic Approach: Learning from Smithsonian

Daniel Bilton, Booz Allen Hamilton
Robert Costello, Smithsonian's National Museum of Natural History (NMNH)- responsible for electronic educational outreach

Stories are experience simulators. They provide the information, the context and the models that people need to make decisions. (There's that context word again!)

How do you turn a boring story into a sticky story that produces action?

Case study (boring)
Provide the teaching point, definition and the scenario. Too much facts, but not enough feeling or attention spend on the background stimulus. Where's the human factor?

Action story
Introduce the nurse and what she notices about the patient. ... (then into a compelling story of how the nurse saves the day and what happens). The discussion after the story talked about the learning point. The story provided simulation (knowledge and how to act) and inspiration (motivation to act).

Sticky Story SUCCESS Checklist
Simple
Unexpected
Concrete
Credible
Emotional
Sparks Action or, Provides a Solution
(from Made To Stick book)

Stories are good vehicles for many types of communications.

The Smithsonian needed to expand reach and teach scientific learning objectives. So they had to: increase the public understanding of the process of science, increase pubic awareness of NMNH research and increase the understanding of forensic science.

How evffective is storytelling? Among middle and high school students (NMNH target audience), they are technically savvy, connected and play video games, watch movies and read comic books.

So in the MNNH project, the main storyline hero is a character that the audience can relate to. Provide the storyline, give the hook and blend in the lessons, objectives, practice examples and scenarios. The story was based on a true story: a bone forensic mystery found in a cellar in Colonial America. The coolest part... the character even got her own Facebook page!

The webcomic got the audience to become emotional about the training. They became empathetic about the characters in the story. (As well as the other SUCCESS factors)

How popular are comic books? Have you seen the magna section in Barnes & Noble? And those reading... not just nerdy boys. There are plenty of magna for girls, too. Young female creators are also among the hottest new comic artists. So don't think that just because it's comic style that it won't be accepted!

Visual storytelling is sticky and an effective instructional approach.

NMNH didn't do a game because it needed to be done quick, and get acrosss a lot of information. And no audio either because of the cognitive load issues. Most read faster than the audio can run.

Sketches done on the comics. Then focus groups with middle and high school kids to get their reaction. The kids in the gropus played out the characters to see if the jokes and learning were received properly. Turns out, they got the story and provided great feedback on the language (lingo) used, font style, and other stuff that could only come from the target audience.

The story is a passive experience, but there was the goal to engage the learners into apply what they learned through exercises. They could dig deeper into the information, make their own judgement calls or compare their answers with other learners.

The story used collaboration in an online community and that made them pay more attention to the story (lesson). Try: instead of a regualar multiple choice question that provided an answer, make it a polling question where you can see the results of other learners.

The results of Google Analytics on the course:

• bounce rate low -- good! People are sticking around -- for about 9.5 minutes!
• the target audience is middle & high school kids, but survey respondents on "how old are you" indicate that 66% is older than 46!

Favorite parts of the comic:

• generates more interest
• liked the mystery aspect
• visuals great for memory and learning

The ABC Approach to telling a Sticky Story
Action - What is the starting event?
Background - What is the situation?
Conflict - What needs to specifically happen?
Development - What happens and who is involved?
End - What is the key take-away?

Lessons Learned

• ID's shoudl think of themselves as writers
• Apply SEction 508
• Try to use social media networks

Really interesting session!!

AG09 - Day Two: Opening & Keynote

Let's talk about Ah-Ha moments from yesterday.

• Jenna Delaney's ah-ha moment "I did it! Finally!"
• Sean: "I'm not as alone in joy/misery as I previous thought."

Shout out to the tweople and bloggers in the back of the room by the power outlets!

Recognition by LINGOs to Greg Davis - 2009 Volunteer of the Year. He delivers all his training online from Panama. Corporate partner of the year is Articulate. So now I need to check out LINGOs more. The concept is interesting.

LINGOs (Learning for International Non-Governmental Organizations) works to improve the performance, lower the cost and increase the effectiveness of Non-Governmental Organizations (NGOs.) Together, LINGOs member agencies provide humanitarian relief and development assistance in over 125 countries around the world. By working together, LINGOs agencies and partners increase the impact of their work by sharing learning resources.

Keynote Speaker, Bill Tancer
Click: What Millions ARe doing Online and Why It Matters to You as an ELearning Professional

Bill Tancer is the general manager of global research at Hitwise, the world’s leading online competitive intelligence services. Or really, a datageek. How so? He entered the talent show at Science Camp in high school and WON! Don't forget to celebrate Pi day! (3/14)

"To become aware of the possibility of the search is to be onto something." Walker Percy

He got started as a writer by given an assignment, then Googling "how to write a magazine column". Immediate need? Immediate search. Immediate learn.

Hitwise's data sampleis massive... can't get into the stats now. Can't read the slide.

So he gets stranger. He has this obsession with prom dresses (?!). Specifically the search terms on 'prom dresses'.

"What is the one thing people is searching to buy on the internet?" Prom Dresses. The spike occured the first week of January. Strange because retailers put out the marketing for prom dresses in March.

Turns out the answer was given by the GM or a teen magazine who approached Tancer at a presentation. The teen magazines started putting out their prom dress promotion in January to change consumer behavior. Solved.

Another data story: data also shows huge spike for the term "diet" starting Jan 1... but drops sharply on Jan 5. Go fugure. Know when "diet" is the lowest? Thanksgiving.

About engagement ring search data...
The search spike is the week before Thanksgiving. Jewelery stores marketing on engagement rings are right after Thanksgiving. WTF?

The theory is: girls bring guy home to meet family. Guys pressured to get an engagement ring. OR... guys are searching before Thanksgiving to get information on them, comparison shop, before they actually make the purchase after Thanksgiving.

An ah-ha moment: Dancing With The Stars is really a popularity content. Therefore, he predictited the winner by checking out the search term data on the contestents. Turns out he was wrong in his prediction. But he was excited that he was wrong. Why? Becasue he's a datageek.

On closer analysis, the data reveals the real reasons why the DWTS search results were skewed. The searchers weren't fans, but sex-starved males checking out the hot contestent. You have to pay attentionto the intent of search as well as the search results. "The Stacy Keilber Correction Coefficient (SKCC)."

Applying the SKCC to housing sales, Tancer predicted how the housing market was going to go. His prediction was wrong, but he was "almost right".

Cognitie Dissonance - the difference between giving an honest answer and giving an answer that you think is the one someone wants to hear. But really, actions speak louder than words. For example, if you ask "who visits porn sites?"... crickets. But when you look at the data of visits to pron sites they're through the roof. So... who's lying?

The Narrative Fallacy: creating a story after the fact so that an event will seem to have an identifiable cause. (See Nassim Nicholas Taleb and his book Black Swan).

The data also suggest that as the market tanks during this recession, eLearning enrollment increases. So, is eLearning recession proof? Enrollment at online universities and online components of brink & morter universities show it's a fact. The higher income people are taking advantage of going back to school.

The value of Twitter: you can listen and and to any conversation in the Twitterverse. Tancer found his value in it when he gave an immediate plug to his book to someone who had just asked if it was any good. For more data, check out search.twitter.com and trends.google.com.

Corporate eLearning isn't forgotten. They haven't gotten the same huge increase as university eLearning, but searches for eLearning platforms have increased. Technology companies are leading the way, followed by education, online retailers and the entertainment industry.

So...
Observed behavior provides unique insight into changing consumer sentiment.

Great Keynote!! I guess I'm a datageek, too!

AG09 - Wednesday Recap

Wow, what a full day. Now that I can relax in my pj's I can review my notes, tweets and blog posts and make some sense out of what I saw and heard today.

Jeff Howe's keynote presentation on Crowdsourcing was an interesting concept. Got a problem? Share it with the un-named, un-faced public and their collective knowledge and creativity can often return a soltuion that is revoltuionary. There is little to no cost for the problem poster, the talent pool for the problem solvers is endless and the recognition that the problem solvers receive is payment enough for their efforts.

I liked the examples he presented, threadless.com and istockphoto.com becasue I actually use both of those sites.

I snuck out a little early to attend Tom Kuhlmann's presentation at the ID Zone, "Is Rapid E-Learning a Myth?" What I enjoyed most about Tom's session is that it wasn't an Articulate advertisement, but practical application of using a rapid e-learning tool, any tool, to develop engaging learning courses. I read his blog religiously and greatly appreciate all the tips and tricks he provides. But I am a newly converted Articulate lover and I was able to get my product specific questions answered by Tom later in the afternoon. I can't wait to get my hands dirty with Quizmaker and Engage.

My first session of the day was with Patti Shank on "What Activities are Needed?" I never read her books or seen her before. My first thought was 'Damn! This woman has a mind that's all over the place!' Patti is an energetic person who self-admittadly does not think linearly. At times it was hard to follow where she was going, but overall, I enjoyed her session. I didn't blog or tweet while her session was going on because my laptop was recharging. I had to revert to notetaking 1.0 -- pen and paper.

Learning is about DOing (which will be a recurring theme today), so training should be about opportunities to practice what learners will ultimately DO on the job. She shared a cute and interesting tidbit: even songbirds need to learn how to sing. They practice just like the rest of us.

There are three steps to her process:

1. Uncover the DO
2. Convert the DO into activities
3. Select media to support the activity

Then came lunch. Yum!

After lunch I attended Will Thalheimer's session on Situation-based Learning Design. I was looking forward to his session since I've heard him present at a webinar and I follow his blog. He uses research to back up his message: if you make your learning context similar to the work context, there is higher incidence of recall and the learner does the right thing at the right time. I took notes and posted them separately.

I was looking forward to the session by the Suddenly Smart team "A Better Way to Design and Build Immersive eLearning". The premise was intersting: there was to be audience participation & presenter challenge -- collect a topic from the audience that presents a performance need. Then, while Robert does the presentation, Blake will build the course that meets the performance need using the rapid development tool, Smartbuilder.

However the first 15 minutes were back and forth conversations to a few individual members of the audience and completely left out the rest of the audience. They spent so much time trying to get the details of the course they were going to spontaneously build that they forgot that they had an actual session to present. I left quickly after that to find something that was more interesting, which is a shame because I would have really liked to learn more about immersive eLearning.

Instead, I went back to the ID Zone and listened to "Game On! Can You Have Fun and Learn?" with Metcalf and Oehlert. Turns out you can. I arrived late so I didn't do a full blog on this, but I did tweet a lot. Casual games (solitaire, tetris, etc) are simple games that aren't graphic impressive. But how many hours have you found yourself glued to the screen?

Games are really just a simulation with scoring. (See Kongregate for casual games with a social aspect.) They are naturally 'sticky'. People are more easily engaged in a game than in traditional training. What if you were to incorporate casual games with a social aspect (such as a chat window). Then you would see how people learn from each other and created teams. And if you think about it, that's how most people's work environment are as well, formed in teams. So, games for gaming sake probably won't be the answer. But if you align the game objectives with the learning objectives, then you have something that can be really effective.

My last session of the day was also an ID Zone presentation. "Once Upon a Time: The Impact of Storytelling on your Design". This also got its own blog post. I liked this session. They helped you plan out how to organize the components of developing a good story: the objective, facts, the situation, the charaters, the plot and finally the resolution.

It was a good, long day. Can't wait for the next one.

ID Zone: Storytelling in eLearning

You are a natural born storyteller! Every time you tell a joke, relate an incident, explain facts & figures, talk about your childhood, recreate the steps in a task... See, you can do it.

How does storytelling relate to eLearning?
For design:

• confirm what to cover (objectives)
• decide how to cover it (design)

For content:

• using scenarios

For motivation & evaluation

• sell the course to futures learners (WIIFM)
• learn how peiple are using the couse in real life

Stories and scenarios can be used interchangeably. Scenarios are real-file stories with a purpose. A really good story can become a really good scenario.

Nonfiction stories in eLearning
Those stories are captured from direct observation or interviews. These are true events. Nonfiction stories can put the learning into context for the learner and make the resulting training more relevant.

Fiction stories
They may be based on real-life scenarios. Often a composite of many experiences. They can be used as cautionary tales or to model exemplary behavior.

So, let's create a story...

First, let's create an objective.
Demonstrate a useful, non-traditional way to use a toothbrush.

A good story contains:

• facts
• situation
• characters
• plot (conflict)
• resolution (ending)

Next, the facts:
Who does this? Define the user categories.
What are their goals?
There are many useful ways to get more from life from a used toothbrush.

Where does this happen and in what circumstance?
You can reuse an old toothbrush at home, at work, for cleaning, or other creative projects.

Characters really bring the story to life.
Personas represent characters' similarities and differences. Personas are composites of user analysis and turning it into a particular character. Also a user profile. They're not one real person, not a sterotype but an archetype, and not your average joe. They have specific properties.
They include:

• name/picture
• job title
• demographics
• goals and tasks in relation to the subject
• environment
• a quote that sums up what matters most to the persona.

If you have a persona for your characters, they can be developed more 'realisticly'. Now, where can you find the domographic info to develop your characters? Think creatively. Customer profiles, Department of Labor statistics, direct observation.

Now, what's the conflict?
BAck to our example. Kathy, our character is having trouble keeping the areas of her sink clean.

The resolution?
Kathy heard from a friend that a toothbrush could help.

Now yout can take that, put it all together, and you can build a scenario that is more robust. Remember elementary school? A scenario is really a story problem.

Change the persona, and the whole scenario changes. Instead of grandma Kathy, how about Paris Hilton or MacGyver?

Go and have fun. How can you use it in your design, in your content and in your evaluation?