Teacher Advisor With Watson
Teacher Advisor With Watson is a free, K-5 lesson planning tool that leverage's IBM's cognitive artificial intelligence to make planning elementary math lessons fast and easy for teachers all over the world. Teacher Advisor leverages Watson artificial intelligence’s understanding of natural language to return targeted and increasingly personalized results for our users.
Our objective: Enable teachers to find high-quality, pedagogically sound instruction faster and easier than any other lesson planning tool on the market.
Teacher Advisor is built with help from a coalition of current and former teachers. Everything in our tool is built to enable teachers to quickly access the materials they value in a convenient, prioritized and individually resonant way.
All of our lessons, activities and strategies are paired with complementary supportive content - aligned standards, related materials, tactics for implementation and recommendations from other teachers - empowering and supporting teachers where they need it most.
My Responsibilities as Design Lead
- Led a complete end-to-end redesign of Teacher Advisor With Watson.
- Delivered complete end-to-end search, discovery and download product experience, prioritized content via "Watson Recommends", My Library saved content page, landing page for teacheradvisor.org and much more.
- Contributed to product strategy, generative research, user testing, low to high-fidelity wireframes, prototyping and quality assurance.
- Partnered with the development and offering leads to collectively define the strategic road map for TA - including defining the vision of our product, scoping our quarterly deliverables and estimating sprint work allocation for entire team.
- Managed multiple full-time designers and oversaw their work from ideation, to creation, to implementation and refinement.
- Worked in concert with our development team on a daily basis to ensure that designs were technically feasible and the work was being implemented to meet our users' diverse requirements.
- Acted as a bridge between the content team (SMEs) and development, which required a thorough understanding of complex user needs, existing content and metadata and the subsequent translation of those requirements to our technical team.
Table of Contents
- Background: What is Teacher Advisor?
- The Problem for Teachers Today: How hard is it to find quality lesson planning materials today?
- The Before State: How our product worked before I joined the team
- Continuation of Research: research leads to pointed objectives
- Teacher Advisor Redesign: Content model, Information architecture, Wireframes, Prototyping
- The After State: What was built and why
- Outcomes: How do we measure our success?
- Lessons Learned and Further Opportunities: Where could we have been better?
The Problem for Teachers Today
How Teachers Find Lessons
Teachers today have a wealth of digital tools to help them in and outside of the classroom. From crowd sourced lesson planning tools (Teachers Pay Teachers) to free inspiration boards (Pinterest) to attendance and professional development tracking tools. Teachers have more digital tools than they know what to do with and less time than ever to use them. It's not uncommon for teachers to resort to Google as the fastest way to quickly find what they need. Unfortunately, the content they often find is often severely lacking in quality.
“It's not usually that things don't come up but that often times a lot of things come up and, to be blunt, most of it is shit…It just takes time to find a reliable and free source.”
- J.D., teacher from research group
The time and energy it takes to find that reliable and free source is time teachers do not have.
Determining the pedagogical quality of the materials is painstaking. As you can see in the screen to the right, if you search "fractions" for 3rd grade on Teachers Pay Teachers, you're still dealing with over 20,000 options, most of which have very high ratings.
Relying on pedagogically dubious, crowd sourced resources puts the quality of your in class experience in the hands of strangers. There had to be a better way.
Enter Teacher Advisor With Watson
Across the country, individual states have leveraged Open Educational Resources (OER) that provide lessons, activities and teaching strategies that have been vetted, of the highest quality, and best of all -- free. The challenge for teachers prior to Teacher Advisor was these materials were scattered and many of the content repositories weren't searchable. If you were a 3rd grade math teacher planning a lesson on "fractions and trains" there was no easy way to find the materials you needed. You could browse countless lessons from one of the individual OER providers, but that could take a lot of time.
Teacher Advisor centralizes all of this content and makes it universally accessible.
When it comes to lesson planning, it's not about whether resources exist on a given topic. At this point, that's a given. It's about whether these diverse resource meets the pedagogical standards of quality and excellence that teachers demand.
Thankfully, Teacher Advisor partners with the best Open Educational Resource providers in the country. Their quality and standard has been attested to by a team of former educators that ensure the resources that we include are pedagogically sound. We knew if we could make this content easily accessible, there was tremendous potential value for our users.
The Before State
I joined the team at a pivotal, transitional time in our product's life cycle. With nearly 300 users on the beta product, TA was little more than a proof of concept. When I joined, the tool had the promise of doing a lot for teachers, but didn't actually fulfill their needs. We were delivering on our promise of offering the best free, vetted pedagogical resources. Unfortunately, that wealth of content was very hard to access.
I was tasked with assembling a design team and crafting a product vision and strategy that could lead us to 10,000 users by the end of the year. It was an ambitious goal for a product that at the time had very little definition, traction or plan for how to get there.
Finding Content on the Old Teacher Advisor
One of the first things I did as Design Lead was perform a comprehensive audit of the current beta product in order to identify our strengths, weaknesses and areas of opportunity. Here are some of the things I learned about the beta product I inherited:
Hard to use filters
A user could only filter lessons, activities or strategies by a single common core math standard and the content provider. This didn't refine the results efficiently or allow for cross comparison of options.
Even if a user selected a standard or a provider to filter by, she had no way of targeting individual topics or areas of focus and had to manually scroll through hundreds of lessons. This was a major pain point and wildly inefficient for our users.
Confusing content previews
When a user finally found a lesson, activity or strategy, previewing the content in question was cumbersome, disorienting and laborious. Similar content flanks both sides of the preview with little structure or organization.
Despite our best intentions, the inclusion of a dialog tool operated as an unhelpful chat bot. Cumbersome, clunky and frustrating, the tool ended up rerouting our teachers to formulaic, basic content. It essentially worked like a very indirect search experience.
The idea behind the dialog tool, which was selected prior to me joining the team, was to create a more guided content discovery experience. For even the most seasoned teachers, guidance is critical. Our product goals were centered on the right things, but the execution left a lot to be desired.
Watson Dialog could only decipher key words and provide rudimentary answers in response, in the format of a firmly rooted decision tree. In the example above I ask for the tool to, "Help me plan a lesson around fractions and trains." Instead of leading a teacher to resources dedicated to that end objective, a user would get rerouted to base categories of "fractions." If a user were to ask for more options, she would get a message such as, "I'm sorry I didn't understand that," and the whole process would start all over again.
Which begged the question, was there an easier, faster way to access materials in Teacher Advisor, all while providing light guidance?
Feedback From Our Users on Beta Site
It was important for the design team to engage with our users early and often in order to better understand what was meeting our teachers' needs and what wasn't. Here are some of the things we heard:
Continuation of Research
What we learned from teachers was that most current or potential users saw TA as a stop gap tool in their everyday curriculum and lesson planning ecosystem. Nearly all teachers are provided curriculum materials from their districts, yet the desirability and efficacy of these materials varied widely. As a result, every so often, a teacher needs to find another lesson or activity for their students that matches her state standards, is reliable, vetted and easy to find. This is where Teacher Advisor With Watson fit in.
So we scheduled many different interviews with teachers. We asked them a variety of questions. Topics included:
- Planning Strategies
- What are your primary needs when it comes to planning?
- How and when do you plan your lessons?
- How do you find resources and teaching strategies?
- Where do you find supplemental content?
- Class Makeup
- What are the levels of students in your class?
- Which students do you struggle most to serve?
- What is the biggest gap in your planning process?
- What do you do when a lesson fails to meet your needs?
Distillation of Research
We then took the answers we received from these questions and transferred each of our individual notes to sticky notes. We put all the stickies up on the wall and then grouped them by topic to identify trends. Finally, we discussed the topics and found insights that lead to areas of opportunity.
Teacher Advisor Redesign
Design Thinking Drives Redesign
The massive overhaul of Teacher Advisor was driven by Design Thinking. As outlined above, this was the first release under my watch as the Design Lead on TA. This led us to a new, pronounced effort to test the usability of the aforementioned changes, but getting our hands dirty alongside teachers to ensure our value is being translated directly into their experience with the tool. This comprised in person paper prototype tests, phone interviews, InVision and XD prototype tests and follow up conversations.
Part of our process included crafting and or updating our understanding of our core user. This was developed via personas, empathy maps and journey maps through our product experience.
Once we established a better sense of who our core user is and how we needed to serve her, we started to ideate on a different way to access content. At this point our team was interested in switching to a search based content discovery experience, but there were major hesitations. The team had invested nearly a year building out the dialog content experience and there was lots of hesitation around whether or not we should just invest in improving what we had, or cut our losses and start over. It was not easy.
As design lead, my role was to assess all of our options and advocate for a calculated risk and in my view, abandoning dialog in exchange for search had the highest upside. But the move had to be sold to our executive sponsors and I had to really rally the team to invest in this platform change. The analogy that I told was one of a meandering road. Dialog was inefficient and indirect, full of potholes that our users had to drive around. In the end, it took far too long to get to the final destination. A good search experience, when done well, could act as an HOV lane for our users to get to the value we promised them faster and easier than before.
Thankfully, our users agreed.
Having a good search feature is really important to me…The faster I can get what I want, the happier I am.
Kristen, Grade 3 teacher, NJ
So we began to think through the architecture of what our product could be with search as the cornerstone of our content discovery.
At the same time, we began to do some low fidelity wireframing, which we tested with teachers and evolved iteratively.
The After State
Because Teacher Advisor leverages Watson's ability to analyze our high-quality content corpus for a wide-range of key concepts and keywords, teachers can search a wide-variety of elementary math terms and concepts, and gain immediate, targeted results faster than anywhere else on the web. The results are efficient, targeted recommendations for our teachers.
Single grade focus
Most lesson planning tools allow default to all content from K-12. We took a decidedly different approach by filtering all content through a single grade filter. This eliminates unneeded materials and allows teachers to focus on only the materials most relevant to them.
Showcase our best content
Teachers don’t have time to be scrolling or searching or browsing dozens of options. If we can put our most relevant content on a pedestal we can be engendering confidence and trust while saving our teachers time. Enter Watson Recommends - selective recommendations enabled through Watson Artificial Intelligence.
Contextually relevant support
All of our detail pages are built with complementary tools to help teacher find related materials, prepare their students and optimize the efficacy of their in-class instruction.
Content pages that empower our Teachers
In addition to standards-aligned materials from OER sources, Teacher Advisor provides instructional context and support to help you understand your grade-level content in-depth—so you can feel confident that classroom implementation will go smoothly, and work for your students.
- Features engaging Teaching Strategy videos to spark ideas
- Allows you to quickly preview content before downloading
- Pairs related resources together for context and discovery
What is 'Watson Recommends' and why does it matter?
When our team decided to invest in Search as the primary means to discover content we knew it had to meet two goals.
- One, it had to deliver content to our teachers quickly and reliably. The expectation is every search needs to work as well as Google. Thus it had to be intuitive and built upon common patterns and frameworks that are familiar to our users.
- The second goal is we wanted to use Watson's Artificial Intelligence to empower the efficacy of our search.
Enter Watson Recommends. Our research with teachers had proven to us the need for teachers to differentiate options quickly. If a user searches for "Measure Areas," she can't spend 20 minutes weeding through options. We needed to impose a hierarchy to the search results in order to provide guidance and empower selectivity and precision. As a result, we employed the WDS API (Watson Discovery Service) in order to create a ranking of search results. When a user executes a search, we scour our database of content and Watson scrapes the data and returns results in order of relevancy to the concept that is being searched for. This is done by assigning relevancy scores to all of our search results results.
Plan for Watson Recommends is to become increasingly personalized - understanding a teacher's needs, the make up of her class, they types of materials that are most effective for her.
Feedback from our Users
Since the redesign, we have spoken with teachers to constantly assess the efficacy of our work. These are several excerpts that we heard from real teachers using Teacher Advisor:
The impact of design on the success of TA
Design's impact upon Teacher Advisor is proportional in how the product we designed leads to dynamic outcomes to our users and our business as a whole.
- Before redesign: 323 users
- After complete product redesign: 8,000+ users
- Over 25% of our 8K+ users are repeat users
- In August of 2017, we had 775 visits to the landing page with 70 registrations, a conversion rate of 9%. Once the complete redesign went live in September, we had 15,341 visits to the landing page with 2354 registrations, leading to a conversion rate of 15% - an increase in conversion rate of 6% and 14,566 more visits month over month.
- Notably, leading research suggests that teachers could save upwards of six hours per year with Teacher Advisor.
Lessons Learned and Further Opportunities
There's always room to improve and there are several things we could have done differently given the chance.
Registration is still too complicated and laborious. There are security mandates and infrastructure requirements that are undeniable, but with growth and outreach as our collective focus in 2017, the user ingestion pipeline is too cumbersome. To say we ignored this truth isn't true. We simplified and improved parts of registration process where we could, but we likely should have dedicated even more effort and focus in finding creative ways to get users registered faster.
Secondly, we worked in relative partnership with the marketing team to coordinate our efforts in messaging the value of Teacher Advisor to an outside audience. This was predominantly effective, but there were plenty of instances where the brand representation outside of our product felt distant from what we were building within the tool. Improving our cross platform communication and streamlining our messaging, marketing and outreach to have the same singular purpose and tone is a major goal in 2018.