UXR Spike: Configuration, AI, ML and Mean Time-To-Value
Business Transactions are quite literally, the value proposition of the AppDynamics APM tool. So when I learned that the BT configuration experience was giving our expert Sales Engineers the blues, I wanted to learn more. What was going on that expert users were constantly reporting pain with the configuration process? Why weren’t their concerns and suggestions showing up as UX improvements or documentations of constraints? And, how was this impacting other users?
I initiated this research spike out of curiosity on the topic, compassion for my colleagues, and a life-long passion for healthcare accessibility.
Scenario:
While exploring Webex channels, I stumbled across the #healthcareobservability channel, where Sales Engineers were discussing many things and also their frustrations around proving AppDynamics product value to customers. This was directly impacting their ability to close sales and renewals.
My background in healthcare made me particularly interested in supporting the efforts of these SE’s — observability in healthcare means that patients, clinicians and hospitals will have reliable access to their Electronic Health Records/ EPIC systems ensuring optimal patient care delivery.
And, my skills as a design researcher meant I could actually help identify and create potential solutions to solve what seemed like a straightforward problem, for users who were also colleagues.
The Challenge:
The Business Transaction (BT) configuration process was manual, and lacking an intuitive user experience workflow and in-product feedback or guidance. Complicating things, the configuration process did not scale; SE and users could not copy or duplicate a configuration from Application to Application, or even rule to rule. This slowed time to value and was a blocker for onboarding as well as renewals.
The Pain:
AppDynamics Sales Engineers are resourceful and innovative. To tackle these challenges, they developed their own ‘Center of Excellence’ process for executing Proof of Value (POV) with customers that provided some direction. Still, because the Business Application configuration process required access to and understanding of the customers environment, pain still existed. Many of these mid-sized customers, in the healthcare observability space used integrated environments, consisting of Custom Off The Shelf (COTS) applications managed by a third party vendor.
This dynamic requires that SE’s to have regular access to the customer, and the third party vendor in order to understand the code architecture in order to successfully configure BT’s. A long arduous process, complete with being handed off to the wrong person and delayed actions due to chain of command protocols required. Finally, once all of the background work was done, SE’s could begin configuration—an unguided, manual, trial and error process where success is determined by checking the outcome and iterating upon it, time and time again.
Story 1: Sales Engineers need a way to automate Business Application configuration so that they can be freed up to optimize Business Transaction Detection Rule performance and win new and renewal deals.
Story 2: Sales Engineers need a reliable way to identify and validate entry and exit points, in real-time for effective Business Transaction Detection Rule configuration, to win new customer and renewal deals.
““It’s hard to feel confident and gain user confidence because the process is a manual game of trial and error” ”
Understanding the Problem
I had informal meet-and-greets with two internal users to learn about their needs, motivations, and pain points.
Core Business Problem
The complexity associated with business transaction configuration took too long and made it hard for SE’s to look like experts, which negatively impacted customer perception of our product.
Research Question
How might we simplify business transaction configuration and allow our SREs to maintain a positive reputation with our customers?
Research Approach
To answer "how to simplify," I first needed to understand what users found complex about business transactions. I conducted the following activities:
User interviews (n=4) with SEs to understand workflows and pain points.
Contextual inquiry (n=3) to observe real-time struggles with BT setup.
Heuristic Evaluation: of the configuration workflow myself.
I performed thematic analysis of interview data to identify and understand trends in user behavior and needs, and product performance and usability. The following became clear: a lack of automation, confusing IA, hidden entry points, and zero in-product guidance was complicating an already complex user experience.
5
Minimum number of screens to traverse in order to create an Application—the prerequisite to creating and configuring Business Transactions
10+
Estimated number of BT's to configure per application service, per Application. This equates to 100s of BT per a single user application.
3
The number of foundational configurations a user must complete before attempting BT configuration (Application, Tier, Node must be accurately configured, first)
Medium
Business deal size impacted typically gross between $5K - $100K+, annually.
5+
The number of different application languages and frameworks that a Sales Engineer must deeply understand in order to effectively configure BTs
0
The number of times “Configure Business Transaction” or “Create/Add Business Transaction” occurs in the UI.
Research Insights
Need for Smarter Agents
BT configuration process requires analyzing and organizing unstructured code, into something that makes ‘business sense’ to the user/customer. There was no clear workflow, guidance, or ability to intuitively automate the ingestion of unstructured data, agnostic of application framework.
Need for improved data collection, processes
Each configuration is specific to the company’s business use cases and dependent upon their application frameworks. This information needs to be well understood by SEs and the user/customer in advance for efficient configuration. SE’s need clear business goal understanding in order to configure the most accurate BT’s.
Usability is a major blocker
No obvious CTA for Business Transaction configuration exist in the primary, secondary or tertiary navigation. Further complicating this, AppDynamics information architecture ontology was unclear and the navigations’ logic was unclear: organization, number and position of menus changed depending upon which page users were on.
Connecting the Dots
My research and insights galvanized disparate efforts across the organization, and gave clarity around prioritization:
Data Science team had an older initiative investigating this area, which had not been prioritized
Another designer was working on tangentially related configuration efforts, with tremendous insight about design solutions
The Product Manager now had business context, user and stakeholder insights about why solving this challenge mattered
Once I shared my research findings, we could put forward a clear rationale for how improvements would help all teams. This research spike successfully influenced the lead PM to re-prioritize BT configuration initiatives, using some of the ML strategies that I proposed for BT grouping and to explore how we might use AI to deliver insights about the accuracy and usefulness of user-chosen groupings.
In regard to usability, this work also reinvigorated a team working on improving the information architecture of the product. While they had already discovered many of the usability challenges that I reported, this body of work lended additional business and user context to their efforts.