Teacher Advisor With Watson
Background
Teacher Advisor With Watson is a free, K-5 lesson planning tool that leverages IBM's cognitive artificial intelligence to make planning elementary math lessons fast and easy for teachers all over the world. Teacher Advisor uses Watson's understanding of natural language to return targeted and increasingly personalized results for our users.
Our objective: Enable teachers to find high-quality, pedagogically sound instruction faster and easier than any other lesson planning tool on the market.
Awards: Teacher Advisor With Watson was selected as a 2018 Webby Award Honoree for both Best Education Website and Best Use of Machine Learning.
Personalized Content
Teacher Advisor is built with help from a coalition of current and former teachers. Everything in our tool is built to enable teachers to quickly access the materials they value in a convenient, prioritized and individually resonant way.
Supportive Context
All of our lessons, activities and strategies are paired with complementary supportive content - aligned standards, related materials, tactics for implementation and recommendations from other teachers - empowering and supporting teachers where they need it most.
My Responsibilities as Design Lead
Led a complete, end-to-end redesign of Teacher Advisor With Watson.
Including but not limited to designing the deliverables for: end-to-end search, discovery and download product experience, prioritized content via "Watson Recommends", My Library saved content, landing page for teacheradvisor.org and much more.
Partnered with the development and offering leads to collectively define the strategic road map for TA, including defining the vision of our product, scoping our quarterly deliverables and estimating sprint work allocation for entire team.
Contributed to product strategy, generative and evaluative research, wireframing, prototyping, testing and QA.
Worked closely with UX Research in the development and evaluation of personas, usability studies, and interviews.
Managed multiple full-time designers and oversaw their work from ideation, to creation, to implementation and refinement.
Worked in concert with our development team on a daily basis to ensure that designs were technically feasible and the work was being implemented to meet our users' diverse requirements.
Acted as a bridge between the content team (SMEs) and development, which required a thorough understanding of complex user needs, existing content and metadata and the subsequent translation of those requirements to our technical team.
Table of Contents
Background: What is Teacher Advisor?
The Problem for Teachers Today: How hard is it to find quality lesson planning materials today?
The Before State: How our product worked before I joined the team
Design Thinking Drives Redesign: User centered design principals lead to product transformation
The After State: What was built and why
Outcomes: How do we measure our success?
Lessons Learned and Further Opportunities: Where could we have been better?
The Problem for Teachers Today
How Teachers Find Lessons
Teachers today have a wealth of digital tools to help them in and outside of the classroom. From crowd sourced lesson planning tools (Teachers Pay Teachers) to free inspiration boards (Pinterest) to attendance and professional development tracking tools. Teachers have more digital tools than they know what to do with and less time than ever to use them. It's not uncommon for teachers to resort to Google as the fastest way to quickly find what they need. Unfortunately, the content they often find is often severely lacking in quality.
Searching "3rd grade math lessons on fractions" yields nearly 1.5 million results
“It's not usually that things don't come up but that often times a lot of things come up and, to be blunt, most of it is shit…It just takes time to find a reliable and free source.”
- J.D., teacher from research group
The time and energy it takes to find that reliable and free source is time teachers do not have.
Determining the pedagogical quality of the materials is painstaking. As you can see in the screen to the right, if you search "fractions" for 3rd grade on Teachers Pay Teachers, you're still dealing with over 20,000 options, most of which have very high ratings.
Relying on pedagogically dubious, crowd sourced resources puts the quality of your in class experience in the hands of strangers. There had to be a better way.
Enter Teacher Advisor With Watson
Across the country, many states have leveraged Open Educational Resources (OER) for their in class instruction. These resources include lessons, activities and teaching strategies that have been vetted, of the highest quality, and best of all -- are free. The challenge for teachers prior to Teacher Advisor was that these materials were often scattered and many of the content repositories weren't searchable. If you were a 3rd grade math teacher planning a lesson on "fractions and trains" there was no easy way to find the materials you needed. You could browse countless lessons from one of the individual OER providers, but that could take a lot of time.
Teacher Advisor centralizes all of this content and makes it universally accessible.
The Opportunity
When it comes to lesson planning, it's not about whether resources exist on a given topic. At this point, that's a given. It's about whether these diverse resource meets the pedagogical standards of quality and excellence that teachers demand.
Thankfully, Teacher Advisor partners with the best Open Educational Resource providers in the country. Their quality and standard has been attested to by a team of former educators that ensure the resources that we include are pedagogically sound. We knew if we could just make this content easily accessible, there was tremendous potential value for our users.
The Before State
I joined the team at a pivotal, transitional time in our product's life cycle. With nearly 300 users on the beta product, TA was little more than a proof of concept. The tool promised to do a lot for teachers, but didn't actually fulfill their needs. We were delivering on our promise of offering the best free, vetted pedagogical resources. Unfortunately, that wealth of content was very hard to access.
I was tasked with assembling a design team and crafting a product vision and strategy that could lead us to over 10,000 users by the end of the year. It was an ambitious goal for a product that at the time had very little definition, traction or plan for the future.
Finding Content on the Old Teacher Advisor
One of the first things I did as Design Lead was perform a comprehensive audit of the current beta product in order to identify our strengths, weaknesses and areas of opportunity. Here are some of the things I learned about the product I inherited:
Hard to use filters
A user could only filter lessons, activities or strategies by a single common core math standard and the content provider. This didn't refine the results efficiently or allow for cross comparison of options.
Endless scrolling
Even if a user selected a standard or a provider to filter by, she had no way of targeting individual topics or areas of focus and had to manually scroll through hundreds of lessons. This was a major pain point and wildly inefficient for our users.
Confusing content previews
When a user finally found a lesson, activity or strategy, previewing the content in question was cumbersome, disorienting and laborious. Similar content flanks both sides of the preview with little structure or organization.
Unhelpful chatbot
Despite our best intentions, the inclusion of a dialog tool ended up functioning as an unhelpful chat bot. Clunky, frustrating and ultimately ineffective, the tool ended up rerouting our teachers to formulaic, basic content.
A depiction of an unfortuanately common outcome
Why Use Dialog?
The idea behind the dialog tool, which was selected prior to me joining the team, was to create a more guided content discovery experience. For even the most seasoned teachers, guidance is critical. Our product goals were centered on the right things, but the execution left a lot to be desired.
Watson Dialog could only decipher key words and provide rudimentary answers in the format of a firmly rooted decision tree. In the example above I ask for the tool to, "Help me plan a lesson around fractions and trains." Instead of leading a teacher to resources dedicated to that end objective, a user would get rerouted to base categories of "fractions." If a user were to ask for more options, she would get a message such as, "I'm sorry I didn't understand that," and the whole process would start all over again.
Which begged the question - could there be an way to marry the specificity users want with the light guidance that they need?
Feedback From Our Users on Beta Site
It was important for the design team to engage with our users early and often in order to better understand what was meeting their needs and what wasn't. Here are some of the things we heard:
Design Thinking Drives Redesign
The massive overhaul of Teacher Advisor was driven by IBM Design Thinking. After I joined as Design Lead, our entire team gather together to discuss the future of our product. We determined that In order for us to grow our user base and deliver our promised value to our users, we had to understand them better. We also needed a set of principals to adhere to as we set out to transform our product experience. Those tenets included the following:
The user is our focus. They are our north star and guiding light.
Leverage the collective expertise of our diverse team. Design, engineering, content, offering management all have different ways at looking at a problem or an opportunity. This is a strength and not a liability.
Have a willingness to learn, improve and fearlessly reinvent ourselves.
With those principals in mind we we're poised to take action. We left our kick off with a methodology that we could employ that would help us get to better outcomes faster. This entailed the following:
Observe - understand our users through direct and indirect observation
Reflect - synthesize their needs
Make - constantly build, iterate and test in order to validate our assumptions and our progress
This virtuous cycle is intended to be repeated.
Although we had received some pointed feedback about the current state of the tool, we doubled our efforts with regards to learning about the day to day of our teachers and their pain points, irrespective of our current solution. We then transferred all of our questions, observations and ideas from freehand notes to sticky notes. We put all the stickies up on the wall and then grouped them by topic to identify trends. Finally, we discussed the topics and found insights that lead to areas of opportunity.
This work helped us not only identify areas of opportunity, but also helped us update our understanding of our core user. We then in turn revised our personas, empathy maps and journey maps. These helped us to better understand the depth and contextual complexity of a teacher's planning experience.
Once we established a better sense of who our core user is and how we needed to serve her, we started to ideate on a different way to access content. At this point our team was interested in switching to a search based content discovery experience, but there were major hesitations. The team had invested nearly a year building out the dialog content experience and there was lots of hesitation around whether or not we should just invest in improving what we had, or cut our losses and start over. It was not easy.
As design lead, my role was to assess all of our options and advocate for a calculated risk. In my view, abandoning dialog in exchange for search had the highest upside. But the move had to be sold to our executive sponsors and I had to really rally the team to invest in this platform change, which included the addition of new Watson APIs. The analogy that I told was one of a meandering road. Dialog was inefficient and indirect, full of potholes that our users had to drive around. In the end, it took far too long to get to the final destination. A good search experience, when done well, could act as an HOV lane for our users to get to the value we promised them faster and easier than before.
Thankfully, our users agreed.
Having a good search feature is really important to me…The faster I can get what I want, the happier I am.
Kristen, Grade 3 teacher, NJ
So we began to think through the architecture of what our product could be with search as the cornerstone of our content discovery.
At the same time, we began to do some low fidelity wireframing, which we tested with teachers and evolved iteratively. We then spent multiple sessions with teachers in order to validate our understanding our their pain points and to find dynamic solutions to solve them. This comprised in person paper prototype tests, phone interviews, InVision, Sketch, XD prototype tests and follow up conversations.
We also began to think about the brand and visual identity of the product we wanted to build. We held several workshops on the desired Voice and Tone of our product that were facilitated by our visual designer, Alison Entsminger.
We started broadly and then created groupings based on reoccurring themes.
Through those working sessions and several subsequent prioritization exercises, we were able to distill the three most important descriptors for our tool - supportive, efficient and personalized.
The After State
Because Teacher Advisor leverages Watson's ability to analyze our high-quality content corpus for a wide-range of key concepts and keywords, teachers can search a wide-variety of math terms and concepts, and gain immediate, targeted results faster than anywhere else on the web. The results are efficient, targeted recommendations for our teachers.
Unlike Pinterest, Teachers Pay Teachers, or other tools, [with Teacher Advisor] I can easily browse through lessons and activities that I that I know will be accurate. There is no fluff to pick through.
Tyler Erickson, Teacher from Oakland, CA on the new Teacher Advisor
Single grade focus
Most lesson planning tools allow default to all content from K-12. We took a decidedly different approach by filtering all content through a single grade filter. This eliminates unneeded materials and allows teachers to focus on the content most relevant to them.
Showcase our best content
Teachers don’t have time to be scrolling or searching or browsing dozens of options. If we can put our most relevant content on a pedestal we can be engendering confidence and trust while saving our teachers time. Enter Watson Recommends - selective recommendations enabled through Watson Artificial Intelligence.
Contextually relevant support
All of our detail pages are built with complementary tools to help teacher find related materials, prepare their students and optimize the efficacy of their in-class instruction.
Content pages that empower our Teachers
In addition to standards-aligned materials from OER sources, Teacher Advisor provides instructional context and support to help our users understand their grade-level content in-depth—so a user can feel confident that her in-class implementation will go smoothly and work for her students.
Landing Page Redesign
What is 'Watson Recommends' and why does it matter?
When our team decided to invest in Search as the primary means to discover content we knew it had to meet two goals.
One, it had to deliver content to our teachers quickly and reliably. It had to be intuitive and built upon common patterns and frameworks that are familiar to our users.
It also needed to use Watson's Artificial Intelligence to empower the efficacy of our search.
But there was an opportunity for it to do something more - instill confidence in Watson and our discovery experience as a whole. Watson Recommends is one of many critical, deliberately designed touch points that enable our teachers to make quick, confident decisions that translate into effective in-class instruction. Thus helping to lessen the stress, strain and time that saps a teacher's energy.
Our research with teachers had proven to us the need for teachers to differentiate options quickly. If a user searches for "Measure Areas," she can't spend 20 minutes weeding through dozens of options. We needed to impose a clear hierarchy to the search results in order to provide guidance and empower selectivity and precision.
When a user executes a search, we scour our database of content and Watson scrapes the data and returns results in order of relevancy to the concept that is being searched for. This is done by assigning concept match relevancy scores to all of our search results. Watson Recommends is the search result with the highest concept match - placed on a pedestal of our teachers to quickly access.
The Future of Watson Recommends
Our plan is for Watson Recommends is to become increasingly personalized. We want to better understand a teacher's needs, the make up of her class, they types of materials that are most effective for her - thus saving her more time by anticipating her unique requirements.
Before we get there, we are beginning to showcase why Watson is recommending something. By providing this level of transparency, we are better able to illustrate the link between a user's intent and their end outcome.
Outcomes
Feedback from our Users
Since the redesign, we have been in constant communication with our users. We know that our success is completely tied to their success. Although we are thrilled with the progress we have made, we know that we have to remain connected to them in order to continue to grow the tool in ways that continue to serve them best.
These are excerpts that we heard from real teachers using the new and improved Teacher Advisor With Watson:
Metrics of Success
My goal as design lead is to enable design to dynamically improve the value of our offering by creating the most direct, consumable and powerful experience for our end users. We are far from finished, but I'm encouraged by our early results.
Before redesign: 323 users
After complete product redesign: 10,000+ users
Over 25% of our 10K+ users are repeat users
In August of 2017, we had 775 visits to the landing page with 70 registrations, a conversion rate of 9%. Once the complete redesign went live in September, we had 15,341 visits to the landing page with 2354 registrations, leading to a conversion rate of 15% - an increase in conversion rate of 6% and 14,566 more visits month over month.
In an end of 2018 survey (with a diverse sample size of 100 current users), 90.8% of teachers reported that they would recommend Teacher Advisor to other K-8 math teachers, 79.8% say that the search function in TA helps them find the right resources faster and better than they could on other websites and 83.5% say Teacher Advisor saves them time in their lesson planning process.
In a further distillation of the value of the new Teacher Advisor, recent research suggests that teachers could save upwards of six hours per year using our tool.
Lessons Learned and Further Opportunities
There's always room to improve and there are several things we could have done differently given the chance.
Our registration process is still too complicated and laborious. There are IBM security and infrastructure requirements that are unavoidable, but with growth and outreach as our collective focus for 2018, the user ingestion pipeline is too cumbersome. To say we ignored this reality isn't true. We simplified and improved parts of registration process where we could, but we likely should have dedicated even more effort and focus in finding creative ways to get users registered faster.
Secondly, we worked in relative partnership with the marketing team to coordinate our efforts in messaging the value of Teacher Advisor to an outside audience. This was predominantly effective, but there were plenty of instances where the brand representation outside of our product felt distant from what we were building within the tool. Improving our cross platform communication and streamlining our messaging, marketing and outreach to have the same singular purpose and tone is another goal for this year.