We’ve written previously about the results of our customer research with L&D professionals and the pain points we identified that needed solving.
From here, we've dedicated ourselves to learning as much as we can about learning, and how we can apply the best practices to help our customers' employees be the best they can be.
Let's jump in!
Quizzing
Multi-choice questions get a bad rap. Especially when they’re used purely for assessment, where the objective is to produce a score. I’m sure you have fond memories (as I do), of answering 150 multi-choice questions for an exam at school or uni. And at the end receiving a numerical score.
No feedback on correct and incorrect answers. Just a solitary score.
Great! Now what?
Multi-choice used in this way is a blunt-force assessment that results in zero learning.
However, there is a better way.
Multi-choice quizzes can be a powerful learning tool. They’re most beneficial when used as a learning technique to improve memory rather than as an assessment.
Quizzes can:
- Encourage repetition of learning concepts
- Present learners and instructors with immediate feedback
- Highlight common misunderstandings
Studies show that students who have been quizzed have a double advantage: a more accurate idea of what they do and don't know and improved memory as a result of having to regularly recall knowledge. (Brown et al.)
Questions can be used at any time throughout the training cycle, whether that’s pre, during or post-training interventions.
- Pre: Questions can be used to surface common misconceptions before training starts. This can assist the trainer in focusing attention on specific areas for improvement.
- During: Questions can be used to check knowledge and understanding of just-learnt concepts throughout the training. This can provide a useful indication to the instructor as to how well learners are receiving what’s being taught.
- Post: Questions can be used to reinforce learning concepts over time, providing retrieval practice and feedback.
Quizzes are Yarno’s primary learning method. We work with our customers to transform large blocks of existing training content into small bite-sized quizzes. And typically learners answer 3 to 4 questions each day.
To date quizzes in Yarno have reinforced existing learning concepts. And we’re super interested to see them being used pre and during training too.
Active recall
When you’re asked a question your brain searches for the answer. If it’s buried within long-term memory you may find yourself closing your eyes or looking away to give yourself maximum focus on retrieving the memory.
Although retrieving an answer in this way can feel like a real effort, the very process of recalling an answer is a valuable part of learning. And it’s called active recall.
Having to recall what you know rather than simply reviewing it strengthens memory, increasing the chances you’ll remember it again. And when you’re unable to answer a question, you know you need to revisit the material or relearn it.
Karpicke and Roediger(2006) performed a series of experiments in which participants learned lists of words and were assessed on their memory exactly one week after learning. They found that when people attempt to recall previous items during learning sessions, rather than simply studying them, retention was enhanced by more than 100%.
To be most effective, retrieval needs to be repeated again and again in spaced sessions, so the recall requires cognitive effort. After a lot of repeated retrieval, we get to the point where our knowledge and skills become reflexive.
Spaced repetition
The spacing effect is one of the oldest and best-documented phenomena in the history of learning and memory research. Harry Bahrick and Lynda Hall.
Research tells us that we learn best when we’re repeatedly exposed to knowledge and skills over time. The practice of regularly returning to previously learnt concepts creates new learning pathways, strengthening long-term memory.
Studies show that students who cram for final exams lose much of their memory within a few weeks, yet students who learn slowly with continuous reinforcement can capture skills and knowledge for decades.
Workplace training is no different. It’s common for training programs to embrace the “one and done” training approach - assuming that after one learning intervention (course or module) learners will be able to retain and apply everything they just learned.
Unfortunately, we know that this just doesn’t work.
So how does spaced repetition work?
Spaced repetition interactions may initially be a few days apart and then as memory strengthens moved out to once a month. You want to leave it long enough that it's a challenge to recall but not so long that you’re essentially relearning the material. In addition having to recall information just when you’re about to forget it strengthens knowledge – even though the actual recall may feel hard at the time.
Embedding knowledge in this way produces better mastery, longer retention and more versatility. Despite hundreds of research papers demonstrating the benefits of spacing in a variety of applications, it’s still rarely used in education. No longer!
The infamous forgetting curve
We forget everything we learn! (Disclaimer: that’s not entirely true)
No post about spaced repetition would be complete without mentioning the now infamous Ebbinghaus Forgetting Curve.
Hermann Ebbinghaus was a psychologist way back in 1885, who self-experimented to understand the workings of the brain. He tested his ability to remember random strings of words. He found that his ability to recall the words declined consistently, and quickly. His memory of what he had learned dropped significantly within a few days and reduced to almost zero after 30 days.
The natural conclusion from his research is that“we forget 100% of what we learn!”. However, it’s not quite that clear cut.
Learning researcher Will Thalheimer explored 14 research articles representing over 1,000 learners, to see how much forgetting occurred.
He found that the amount a learner will forget depends on many things, such as:
- The type of material being learned
- Their prior knowledge
- Their motivation to learn
And he also found that learners forget at varying rates. For example, in the 1-2 day range learners forgot from 0 to 73%. And learners in the 2-8 year range forgot from 16-94%.
So it turns out that forgetting is unique to individual learners, and there is no magic one-size-fits-all forgetting equation. Thalheimer concludes by saying, in general, the longer the spacing interval, the longer the information will be remembered.
Yarno currently re-presents incorrectly answered questions after 3-5 days and correctly answered questions after 10-15 days. In future iterations of the platform, these spacing intervals will be automagically tailored to each individual learner, thereby optimising knowledge retention.
Interleaving
Much of our learning is presented to us sequentially. We attempt to master one topic before moving on to the next. Textbooks are a great example of this, presenting one chapter at a time with a quiz at the end (if you’re lucky). This approach is called blocking, and it’s very useful for cramming and recalling the next day.
Blocking isn’t so useful in the real world, however, when we want to recall knowledge into the future. That’s because blocking doesn’t require us to know the difference between learning concepts. We just have to know what order to use them in.
Enter interleaving. When many different but related learning concepts are presented to us at the same time, it’s called interleaving.
Interleaving feels harder at first because our brains have to make sense of how the different concepts we’re learning are related. But over time we get progressively better at each one, producing long-lasting effects.
Returning to our textbook example, if we ripped out all the chapters and worked through them in random order, that would be interleaving. This method works best when we already have a foundational understanding of the topic we’re learning.
Yarno leverages interleaving by presenting questions from various topics in random order. Most campaigns have around 10 topics, so in any given day a learner could be answering 3 to 4 questions across 3 to 4 topics. Typically within 2 weeks, we find a 10-20% spread between the strongest and weakest topics for learners and teams.
For more info, have a read: How to use interleaving to win long-term results.
Immediate feedback
One of the key features of Yarno is that we provide immediate feedback to a learner on how they answered. We included this feature originally based on learning research, which showed immediate feedback can correct misconceptions and help learners build correct mental models.
Several learners on our first pilot were surprised at some of the questions they were answering incorrectly. These were 10-15 year veterans, often managers, who assumed that the questions would be a breeze. After recognising that this wasn’t the case, they were motivated to focus their attention on these discrepancies and resolve them.
While beneficial, this realisation can also be confronting. No one likes to be proven wrong, especially not in front of our colleagues. So we make a real effort with Yarno to create a safe and supportive learning environment. We do this by framing Yarno as practice rather than a test.
And an important part of practice is making mistakes. It’s how we all learn.
Conclusion
With Yarno we have tried to facilitate a new way of learning, based on the latest learning research. And we really feel that the potential is limitless.
