Generative and evaluative research for a new enterprise system

I (Debbie Levitt) started a contract with the Enterprise Technology department at Arizona State University (ASU) in August 2023. The contract would end with the fiscal year in June 2024.

I was brought in as Lead UX Researcher because a project I’ll call “ABC” had already been in-flight for some years, but had never been put in front of target users. 

Utilizing internal funds as well as grant money, ABC was one of the most high-priority and mission-critical projects at the University, and had been aiming for a Fall 2024 launch. Users included students, advisors, and success coaches.

 

 

The Team

My manager and I quickly realized that doing ongoing generative and evaluative research cycles for ABC was more than one person could handle in a timely manner.

I was allowed to hire three workers to add to my team.

• We received hundreds of resumes for the student worker position, interviewed fewer than 10 people, and selected Katelyn Dang, who was allocated part-time to our project.
• UXR Mia Logic was also allocated to the ABC project part-time.
• I handpicked two other Researchers to come in as contractors for the remainder of the fiscal year (through June 2024). Kelene Lee was our Mid-Level UX Researcher, and Sirocco Hamada was our Associate UX Researcher.

Creating an ongoing cadence of research insights allowed our larger team to continually pivot toward features, designs, and interfaces that worked best for target audiences. At all levels, everything in the project could be done with a strong foundation of evidence and knowledge. Nobody on the team needed to guess; they could tell the Research team what they needed to know or understand, and we would have that information to them some weeks later.

Our research work drove decisions and important changes:

• Gave the team the evidence they needed to completely pivot the interface’s layout.
• Assured our Data and Analytics partners that the information visualized in the tool matched what our target audiences needed.
• Saved Development cycles by giving the Designer the information she needed to get well ahead of Engineering.
• Supplied information about how target audiences behaved, allowing better decisions to be made earlier, and less for Engineering to redo or fix.
• Surfaced an interesting bi-directional trust issue between students and their academic advisors.

We set the larger ABC team up for continued success and a process including ongoing cycles of UX research and testing.

• The ABC outer circle now better understands the importance of having UX Researchers answer important questions before making key decisions. Cross-functional teammates even started asking for blank “Knowledge Quadrant” boards for future studies before I was thinking about studies that far in the future.
• I nudged Project Managers to invite UX Researchers and our Designer to more meetings with our higher-level Provost stakeholders. This allowed the larger UX team to be more involved in UX-related decisions.
• Previously, the Designer was mostly informed of decisions, and needed to make the matching screens. I wanted to ensure that our fantastically talented Designer was empowered to be more of a problem solver and less of an order taker. We delivered research reports that didn’t suggest exact solutions or mechanisms, but laid out the problems to solve and insights from participants.

Project Manager (representing the Provost’s Office)

“The UX research studies that Debbie and her team have done for our application over the past 8 months have helped us understand the behaviors, needs, and desires of students and their advisors.

The UX team worked collaboratively with functional stakeholders to understand the goals of the studies and sought input, feedback, and approvals for each study. They provided detailed reports at the end of each study and were always willing to answer any follow-up questions.

As a result of these studies, we have been able to fine-tune the user interface and are confident that the application will serve the purpose and goals it set out to achieve.”

 

Moira McSpadden, Business Analyst

“Debbie has been great to work with and brought many skills to the table as interlocking teams worked to build a student-centric application. Her insights were invaluable to the work.”

 

Dania Albasha, Technical Product Manager

“Debbie’s impact as a Lead UX Researcher on our [ABC] product at Arizona State University has been transformative. Within just months, she navigated the product from a cumbersome experience to one of clear value and intentionality. Her expertise in UX research fostered cohesive collaboration across our development teams, stakeholders, and university research groups, elevating our organizational mindset towards user experience.

Debbie’s approach was inclusive and dynamic, actively involving stakeholders in research planning, delivering rapid insights for informed decision-making, and crafting actionable recommendations grounded in robust data. Her ability to influence leadership strategy and decision-making was evident as we embraced experimentation and welcomed diverse user perspectives, leading to tangible improvements in the product’s usability and overall user satisfaction.

We witnessed the evolution of the product after just three successive research studies which demonstrated the significance of understanding our users and the problems we are trying to solve. Debbie’s dedication to user-centric design drove meaningful enhancements that directly benefited our users and advanced our organizational goals.

I have no doubt that Debbie’s exceptional skills and unwavering commitment to UX research will continue to make a profound impact in any role she undertakes. She truly impacted ABC in ways I can’t even figure out how to write!”

 

Linda Nguyen, UX Designer

“Insights from the UX research team have been pivotal in the design process. With their findings, I feel empowered to make decisions based on the needs and behaviors of real users, as opposed to guesses and assumptions. The research is also thorough, presented clearly, and provides actionable recommendations.

Additionally, the research team has been very open to new ideas and willing to test them. This has saved our product team a lot of time and effort since those ideas can be efficiently evaluated through prototypes prior to development.

I am grateful for the excellent work that the research team has done and I feel confident that with each iteration, we are getting closer and closer to making a product that makes people’s lives easier.”

 

Victoria Polchinski, Lead UX Researcher (and our Manager)

“We hired Debbie to lead large generative and evaluative UX Research projects for a large, complex, critical product. Through a collaborative planning process and by delivering on promises, Debbie and the team were able to deliver actionable insights that have led to data-informed decisions that are making a huge impact on the customer experience. Their first study highlighted major usability issues and mild customer sentiment related to the experience.

By partnering closely with UX Design, users are now saying “OMG! I love this.” And “When will this be live?” ”

We had an “inner circle” of approximately eight stakeholders including our Technical Product Manager, a Project Manager from the Provost’s office, a Project Manager from the Enterprise Technology department, the Director of Actionable Analytics, a Researcher and Research Director from another department working on a variation of this project for prospective students, and some Assistant Vice Provosts.

Our “outer circle” included others from the Provost’s office, Business Analysts, people working on the partner program, and others heavily involved in the ABC project.

We collaborated at key points during and after each research study:

 

• The outer circle was invited to participate asynchronously in the Knowledge Quadrant (KQ) exercise, which heavily influences the plan for each research study. We want to ensure that we are answering the questions our cross-functional team needs to create better strategies, priorities, decisions, and product.
• The inner circle was invited to comment on our research plan to make sure that we didn’t miss anything that they needed to learn. This also allowed them to suggest changes such as better ways to phrase things, given the target audiences and internal lingo they may or may not know.
• The outer circle could leave post-research questions in another frame of the Miro board that we used for the KQ. At the end of each study, we delivered a report in Google Slides, some video clips, and a video version of us talking through the key points of the report. Rather than holding a meeting where we read a report to people, we provided all of the documentation, and then scheduled a meeting a week or so later to take questions on our findings, insights, or suggestions.

Co-Researchers Kelene and Sirocco were equal partners in every task and phase of the research work. They hit the ground running incredibly quickly, already contributing to the study 2 research plan on their second day on the job.

They both exhibited strong mid-level UX Researcher work in all areas and tasks, including planning, moderating observational and interview sessions, analyzing and organizing hundreds of notes on a Miro board, finding important themes and patterns, capturing and compiling video clip montages, and writing informative and compelling report slideshows.

Kelene and Sirocco also took the lead on writing the Study 3 and Study 4 reports, both over 80 slides each due to the number of questions stakeholders wanted answered. I worked on planning the next study with stakeholders.

Kelene and Sirocco also played a larger role in presenting Studies 3, 4, and 5 in our video presentation and stakeholder meetings.

Kelene and Sirocco are highly skilled teammates with whom I would love to work again on any project. Special thanks to Mia and student worker Katelyn for their great help with planning, recruiting, sessions, analysis, synthesis, and reporting!