Automatic and Intelligent Observability: Fast Track to Better User Experience RESEARCH BRIEF
Automatic and Intelligent Observability: Fast Track to Better User Experience
1
Executive Summary Government organizations are under constant pressure to make enterprise operations and services more robust and resilient. The goal is to advance agencies’ missions amid shifting and frequently unpredictable external circumstances — military interventions, natural disasters, public unrest, economic vicissitudes and the occasional pandemic — and to be ever-more responsive to constituents and employees. Today, the measure of institutional responsiveness is the customer experience, which Digital.gov defines as a constituent’s overall experience of interacting with an agency. A key part of that is the user experience (UX), which looks at a constituent’s interaction with a particular application or service. UX uses such metrics as success rate, error rate, abandonment rate and time to complete a task. A robust UX makes it possible for thousands of government workers to transition from traditional offices to working from home during the health crisis. It improves the effectiveness of warfighters, who rely on instantaneous technology upgrades to maintain an advantage over adversaries, and it reduces stress in the lives of people applying for unemployment benefits at a time when millions are suddenly out of work.
2
Research Brief
Delivering a satisfactory UX to government workers and the people they serve, especially in trying times, requires fast and seamless deployment of enterprise applications, from management and analysis of critical data — biomedical research, environmental health, public health data, human genome research and infectious-disease control research, for example — to public access and dissemination. Organizations must provide consistent and robust application monitoring to maintain superior application performance UX. But as agencies adopt dynamic, modern IT technologies and infrastructure — hybrid cloud architectures, cloud-hosted and serverless apps, container-based orchestration — establishing the observability needed to understand and improve UX has become difficult or impossible. To learn more about the challenges of providing world-class UX, Dynatrace and GovLoop, a digital knowledge network for government, surveyed more than 65 members of the federal, state and local IT government communities. The survey found that agencies increasingly recognize the importance of UX — and the need to invest in the right solutions and to commit to continuous improvement.
User Experience — An Emerging Best Practice In the past decade, the concept of UX in government IT has gained widespreadacceptance and moved into the mainstream. Long considered an afterthought or not even worthy of consideration, UX has emerged as a principal metric for the effectiveness of systems and applications.
Such sentiments would have been inconceivable until fairly recently, said Willie Hicks, a Senior Application Performance Engineer at Dynatrace, a technology company that specializes in software intelligence. Hicks remembers trying to interest engineers at a federal agency in the value of UX management about a decade ago. They weren’t buying it.
Almost seven in 10 respondents said that understanding customers’ experience would be “very valuable” (see Figure 1), while more than 60% said that their agencies consider providing a high-level end-user experience to be “important” or “very important” (see Figure 2).
“‘You know, user experience doesn’t really matter because it’s not like [the user] is going to another agency,’” Hicks recalled one of the engineers saying. That notion of government agencies serving as unique providers of services and therefore inoculated against poor performance reflects a mindset that Hicks has seen change in recent years.
Automatic and Intelligent Observability: Fast Track to Better User Experience
3
Indeed, the shift has been dramatic. When asked if they expect their agencies to prioritize UX in the next year or two, 84% of respondents said yes (see Figure 3). Several trends have elevated UX as a priority: executive orders and other government regulations, lessons learned from high-profile IT failures, lofty UX standards set by private-sector companies, and the inherent complexity of deploying and managing modernized IT infrastructure. The Office of Management and Budget, in OMB Circular A-11 Section 280, set out clear guidelines that define federal government customer experience (CX). The guidelines provide expectations for the role of CX, what it should accomplish, and how to measure it. Noting that 67% of trust in government can be explained by customer experience, OMB concluded that “improving the public’s trust in government happens interaction by interaction,” and that “customer experience has also been demonstrated to improve outcomes such as saving costs, reducing risk, and more effectively achieving stated missions.” Private-sector companies such as Google and Amazon are strongly influencing the expectations end users have for government interactions. Those companies use artificial intelligence (AI), algorithms, and biofeedback to understand and improve exchanges with users. “Private sector companies do a lot of analysis to optimize user experience,” Hicks said. “Government agencies historically haven’t had the competitive pressure to put much effort into UX. But the private sector has set customer expectations at a high bar, so now agencies have to provide a better experience.” Further motivating agencies to improve UX is a desire to not get caught on the wrong end of an embarrassing UX debacle.
4
Research Brief
When HealthCare.gov launched in 2013 — on the first day of a government shutdown — technical problems abounded. Built to enroll millions of people in health care plans following the Affordable Care Act, the site successfully signed up only a small percentage of would-be applicants in its first week of operation. The episode has become a cautionary tale of poor UX and how not to roll out a new government program. “It was a watershed moment that demonstrated how complex applications require hundreds of interfaces into backend applications,” Hicks said. “If they are not built, tested and presented properly to constituents, they are bound to fail. That really drove user experience into the forefront of people’s minds. It’s become kind of a bellwether.”
Calibrating the Cloud for UX Conventional wisdom holds that cloud solutions are essential to enabling a good UX. But usability is not as simple as just adopting cloud-based apps. From an end-user’s perspective — whether that user is a constituent or an employee supporting operations or services — the characteristics that define good UX are consistent. Asked to identify markers of exceptional UX, more than 60% of survey respondents identified “easy to use” as important. More than 30% also identified “clear” and “accessible.” At least 20% of respondents named other characteristics such as “intuitive,” “fast,” “consistent” and “efficient” (see Figure 4).
Modern enterprises, hybrid clouds and digital transformation initiatives have vastly expanded the potential for improved UX — when those initiatives are done well. Poorly executed initiatives, however, can have the opposite effect. Consider the government’s Cloud First and Cloud Smart guidance that encouraged agencies to move workloads to the cloud and become more digitally oriented. “The challenge is if you simply grab a workload and move it from Point A to Point B, you still have a legacy application that doesn’t perform that well,” said Steve Mazzuca, Director of Federal Defense and Intelligence at Dynatrace. “You have to find a way to analyze and ensure better overall performance and health of services.”
Automatic and Intelligent Observability: Fast Track to Better User Experience
5
The survey revealed uncertainty about the relationship between modern, complex, clouddriven IT environments and UX. Almost two-thirds of respondents said it is “difficult to say” whether modern environments make it easier or harder to deliver outstanding UX because the impact “depends on other factors.” About one-third of respondents said modernization makes UX easier (see Figure 5). Asked about the link between IT infrastructure, applications and UX, almost two of every five survey takers said IT and UX are “separate issues” at their agencies. Other respondents acknowledged that IT influences UX, both directly and indirectly (see Figure 6). However, in recent months, many agencies have begun to realize that UX is both an important indicator of overall IT health and a risk factor for enterprises. During the government’s response to the COVID-19 pandemic, legacy applications that were not designed to handle surging workloads couldn’t keep up and crashed.
6
Research Brief
“You’ve got to build applications that will scale to need, and then you have to be able to instrument those applications so that you can identify problems before they start to affect the end user,” said Bob Dorch, Director of Federal Sales at Dynatrace. AI and continuous automation are the only practical way to manage dynamic and complex multicloud environments. “Traditional tools can’t keep up with the complexity of modern applications and infrastructure,” Dorch said. “AI and automation provide visibility into applications and hybrid IT environments so you can catch problems before they affect end users, whether they are constituents or employees.”
Failure to deliver a positive experience carries the risk of users disengaging with an organization. More than two-thirds of survey respondents cited frustration as an unwelcome outcome of poor UX. Other potential outcomes most frequently cited are “inefficiency,” “confusion,” “low morale” and “misunderstandings” (see Figure 7). The survey also indicated that poor UX correlates with negative customer behaviors and outcomes, including confusion, complaints and repeat calls. Fully 87% of people taking the survey said inadequate UX led to such behaviors, which ultimately led to inefficiency and wasted resources. Such outcomes indicate that agencies may be falling short of mission goals (see Figure 8).
Automatic and Intelligent Observability: Fast Track to Better User Experience
7
Finding the Optimal Solution The business case for improving UX through application performance monitoring isn’t new. First- and second-generation solutions have existed for more than a decade. Until recently, however, those tools produced results that frequently were underwhelming. “It wasn’t a question of whether application performance monitoring was a necessity. It was,” Dorch said. “But could organizations afford the time and labor required to instrument applications in order to effectively deploy monitoring tools?” In the past, he noted, only about 5% of mission-critical applications would get monitored, because “it was so difficult to deploy the monitoring solutions that were available.” Almost three in five survey takers reported that their agencies use a mix of manual and automated application monitoring tools — and one in four don’t know what kind of tools they use, which could indicate high levels of inefficiency (see Figure 9). Microservices, serverless apps, Kubernetes and other cloud-native technologies help organizations achieve the agility they need to accelerate innovation and achieve more successful business outcomes. But they also bring hundreds of thousands, millions, sometimes billions of dependencies that come and go in seconds. Agencies need a solution that uses AI to simplify the complexity and improve the user experience. “AI makes it possible to gain observability into the user experience, and provide direct feedback to the people who support the end user so they can understand when something goes bump in the night, when something breaks, when a user is not happy,” Hicks said. “They don’t have to wait until they get a complaint. They will get that feedback immediately.”
8
Research Brief
Two-thirds of people who took the survey said they had used monitoring tools that generated more data than insights. That’s because most solutions gather data in dashboards, but users often don’t know how to interpret information on those dashboards. Almost nine in 10 said it would be helpful to have tools that monitor applications and UX in real time. Insights from real-time data make it easier to quickly correct issues that could compromise delivery of services and mission attainment (see figures 10 and 11). When the COVID-19 health crisis hit, the ranks of Veterans Affairs (VA) teleworkers grew from about 35,000 to about 150,000 in a matter of days. To ensure that workers’ experience at remote desktops matched their office experience, VA used an automatic, AI-assisted technology to monitor and instrument the performance of the new remote workforce. “That process normally would have taken an army of engineers months to complete,” Mazzuca said. Using the AI-driven solution, they automated the process in days. The Social Security Administration used the same automatic, AI-assisted technology when it successfully rolled out the second generation of its disability claims processing system. The AI assistance and automation in the new deployment delivered performance that exceeded all prior iterations. “Those disability claims are processed by the 50 state governments and Puerto Rico and rolled back to Social Security for payment, so it’s a huge integration, a monumental effort,” Mazzuca said.
Automatic and Intelligent Observability: Fast Track to Better User Experience
9
Two-Factor ROI Many factors go into choosing an application monitoring
Slow performance can negatively affect end-user
solution, but two stand out: ease of deployment and
experience. A change in one area may cause inadvertent
return on investment (ROI). The two issues are related: A
performance consequences in another, which may not
robust application monitoring solution should generate
be detected until the problem affects users. Migrating
ROI by simplifying the deployment of other applications to
an app to AWS GovCloud West, for example, could slow
conserve financial resources. “When an error happens in a production application that’s
performance if most of its users are at the Pentagon, and traffic must go all the way across the country.
used by thousands of folks in an organization, it costs
“Somebody could get a black eye if they did not think
millions of dollars in downtime and lost productivity,”
to check whether moving to the cloud changed the
Mazzuca said. “When you find those issues earlier in the
characteristics of how something works,” Hicks said. “If
development cycle, it just costs a handful of developers
you don’t test these things, you’re not going to know you
rewriting code.”
have a problem until you start getting phone calls.”
More than 50% of survey respondents said ease of
Detailed data about user experience can further improve
deployment was a “very important” feature of UX-
operations, almost two-thirds of survey respondents said
improvement tools (see Figure 12). Those respondents
(see Figure 13). Dynatrace’s AI- and code-level analytics
understand that complex, drawn-out deployments
solutions make it easy to identify potential performance
can drain financial resources, demoralize staff and
problems before they affect end users.
compromise business functions. An observability solution should be easy to install, delivering immediate telemetry without configuration.
10
Research Brief
In the final analysis, simply getting more data won’t help agencies. Organizations are drowning in data. The survey revealed that organizations’ application monitoring tools generate actionable insights. More than three-fourths of respondents said monitoring tools, in addition to improving UX, could help agencies by simplifying cloud computing and advancing digital transformations. More than 60% of respondents said it is very important for federal agencies to have tools that will provide deep insight into application performance (see Figure 14). Agencies that have robust data and deep insights can detect and remediate issues before they become apparent to end users. Relying on data without insights can result in avoidable scenes, such as those that unfold after an IT meltdown. “Typically, stakeholders gather in war rooms to understand what went wrong and how to fix it,” Dorch said. “You’ve got people from operations, from the network, from the applications team, the business owners, and they’re all scratching their heads and pointing fingers at each other, saying, ‘Hey, the data I’m collecting says it’s not my problem.’” “What teams need is a single set of metrics that everybody in the room can use that provides insights and pinpoints the actual problem,” he said.
Automatic and Intelligent Observability: Fast Track to Better User Experience
11
How Dynatrace Delivers Better UX Dynatrace is the industry-leading provider of software intelligence products and services, providing application-performance management, AI for operations, IT infrastructure monitoring, digital experience management and digital business analytics. Businesses and government agencies trust these services to simplify cloud complexity and accelerate digital transformation. Dynatrace’s all-in-one platform uses AI with automatic and intelligent observability at scale to deliver actionable insights about applications’ performance, underlying infrastructure and UX. The right data enables organizations to innovate, collaborate and deliver
value with fewer impediments, from modernizing and automating cloud operations to speeding the release of improved software and delivering digital experiences that delight end users. The Dynatrace Software Intelligence Platform discovers, maps and monitors enterprise cloud environments, on-premises deployments and hybrid cloud environments. It also provides automated problem remediation and delivers observability of the full solution stack to simplify cloud complexity and speed organizations’ digital transformation to cloud-based architectures and microservices running in containers on orchestration platforms.
Curious about how to achieve your digital transformation and mission goals with greater efficiency, simplicy, and speed? Let Dynatrace show you. Visit their government solutions page for resources and a 15-day free trial.
About Dynatrace Dynatrace is a software-intelligence monitoring platform that simplifies enterprise cloud complexity and accelerates digital transformation. With Davis (the Dynatrace AI causation engine) and complete automation, the Dynatrace all-in-one platform provides answers, not just data, about the performance of your applications, their underlying infrastructure, and the experience of your end users. Dynatrace is used to modernize and automate enterprise cloud operations, release higher-quality software faster, and deliver optimum digital experiences to your organization’s customers. To learn more visit www.dynatrace.com.
12
Research Brief
About GovLoop GovLoop’s mission is to inspire public sector professionals by serving as the knowledge network for government. GovLoop connects more than 300,000 members, fostering cross-government collaboration, solving common problems and advancing government careers. GovLoop is headquartered in Washington, D.C., with a team of dedicated professionals who share a commitment to the public sector. For more information about this report, please reach out to info@govloop.com. www.govloop.com | @GovLoop
1152 15th St. NW Suite 800 Washington, DC 20005 P (202) 407-7421 | F (202) 407-7501 www.govloop.com @GovLoop
Automatic and Intelligent Observability: Fast Track to Better User Experience
13