Last year, I wrote in Experimenting with GenAI about how we planned to approach understanding Generative AI and how it could benefit customers and Yarnoers. This post updates our progress since then and includes an exciting announcement!
Update
Over the past 12 months, a core AI project team has met every fortnight to experiment with and share what we have learned from playing with Thimble. Thimble?! Yep, that's our secret squirrel codename for the AI feature in Yarno. The team consists of the following roles: Instructional Designer and Content Lead, CTO, Product Manager, Managing Director, and Head of Sales.
Our goal with Thimble is to improve our customers' content creation experience in Yarno. Some of our customers find creating their own Yarno content time-consuming and don't have the resources to invest in creating content to the high standard they'd like.
We knew the Generative AI space and the large language models (LLMs) were likely to change frequently, so we built a lightweight application to swap LLMs as desired. We also wanted to produce Yarno-ready content with AI, so we split out the elements of a Yarno question, answers, and explanation and created a separate prompt for each.
We tried to capture our approach to content creation and everything we coach our customers to do in these prompts. We experimented with Anthropic's Claude and OpenAIs GPT. In the initial user prompt, you'd type in the audience, the goal for the campaign, and any other contextual info. Yarno AI created 10-15 questions and correct answers. You'd pick one and ask it to generate incorrect answers and an explanation.
We've iterated this process over the past 12 months with many learnings and challenges!
Learnings
Specificity is best
The more specific the prompt, the better the output, e.g., being particular about the audience and topics in the campaign, which gets better results than being vague and/or using generic language.
Writing in the style and tone of celebrities can work (and is good value!)
Asking it to write a Yarno question in the tone and voice of a well-known personality can work if there's enough information about them. We experimented with a few different celebs. Here’s an example in the style of UK comedian Bob Mortimer:
You've discovered that the office plants are actually made of plastic, but everyone thinks they're real. How do you handle this revelation without breaking trust?
And
Oi, you lot! What's that fancy word for connecting with others that smartphones are mucking up for kids?
This example, using the tone and voice of Robin Williams lays the language on pretty thick:
Welcome to Jumanji! I mean, your quarterly budget meeting. You need to cut costs, but your team is already stretched thinner than a rubber band on Mount Everest. How do you approach this while maintaining trust?
LLMs are updated regularly
The LLMs are updated regularly, so we had to keep checking and testing the latest versions to see if previous issues had been resolved and if anything had broken. When Claude 3.5 was installed, the "refine incorrect answers" functionality no longer worked.
Challenges
Explanations not generating
Sometimes, the question explanations wouldn't generate at all or would partially generate. This happened intermittently, every 10-15 tries. It looks as though the JSON response didn't come through as our system expected, so the output failed.
Implausible incorrect answers
One of our most significant challenges has been getting the LLMs to produce implausible incorrect answers. By this, I mean non-obvious answers that the learner can't discount immediately.
Question: You've just bought a new laptop and want to keep it secure. What's the first step you should take?
Incorrect answer choice: Share the laptop with friends and family members.
And non-sensical:
Question: Why is using unique passwords for every account important?
Incorrect answer choice: It's easier to manage all your passwords if they are the same
Broken links in the explanation
We like to include links to websites and articles for additional information in our explanations. However, the links that the LLMs return are often broken. We've added additional checks to account for this.
Counting is problematic
It's well known that counting is not an LLM's strength, though this is improving. We discovered this when asking for 15 multi-choice questions, usually receiving over 30. However, it's not a big deal for us.
Exciting news
After two years of experimentation and testing, Thimble has been renamed Yarno AI and will be live on the Yarno platform in February 2025! We’re thrilled to offer our customers the chance to generate high-quality, customised learning content in seconds.
Ready to transform your training? Get started with Yarno today and see how easy and effective learning can be.
