TOMAT AU
IN
G
M
IT MON
OR
ORMANCE RF
PE
LOY P E D
ION
Embed quality and accelerate development throughout your mobile SDLC
BUIL D
Continuous Quality
DEV EL OP
Deliver Better Mobile Apps Faster with
OR T I ON
TEST
Table of Contents
DEV OR T I EL ON Mobile is Now Mission-Critical................................................................................................................................................................. 6
OP
M
Defining Continuous Quality..................................................................................................................................................................... 5
Components of the Continuous Quality Lab............................................................................................................................................... 11 The 4 Phases of Continuous Quality......................................................................................................................................................... 18
TOMAT AU
ION
IN
G
OR
PE
ORMANCE RF
TEST
IT MON
BUIL D
LOY DEP
Summary............................................................................................................................................................................................... 23
2
3
Executive Summary Mobile apps have dramatically changed the way users interact with brands. But with the rapid pace of mobile hardware and software innovation, many organizations struggle to meet user expectations. These challenges are well understood, it’s the solutions which have proven to be elusive. In order to ensure application quality in the modern age of mobile, a new approach is required. But what? We believe that the answer can be found in the principles and practice of Continuous Quality (CQ). Written in plain language – but with enough technical details to satisfy seasoned development and QA professionals – this short book provides a definition of Continuous Quality in theory and gives a practical overview on how to implement its methodology.
Topics Covered: •
New Dynamics and Challenges of Mobile Apps
•
An Introduction to the Principles of Continuous Quality
•
Components of the Continuous Quality Lab (CQ Lab)
•
The 4 Key Phases of Continuous Quality in Practice
4
Definition Continuous Quality (CQ) is a methodology for embedding quality activities into every step of the SDLC process from design through build to production.
TM
CQ shrinks time-to-market, drives faster release cycles and minimizes escaped defects to production.
5
No Longer a Nice-To-Have, Mobile is Now Mission-Critical Mobile is becoming the main product platform for enterprises and is driving SDLC transformation This time, it really is different. We’re talking, of course, about the rapid adoption of mobile devices and applications – a claim backed up by the numbers:
8x
Mobile adoption is growing 8 times faster than web adoption did in the 1990s and 2000s
60%
Today, 60% of online traffic now originates from mobile devices More than 80% of mobile time is spent within native apps
73% of Millennials are already transacting directly on their mobile devices. 75% of Millennials have their mobile devices glued to their palm while in store as a trusted personal shopping assistant. 6
These are just a few (of many) recent statistics that have led brands to reach two important conclusions: 1.
Mobile is now the preferred consumer brand engagement point.
2.
As consumer preferences change, so too must corporate business models: keep up or fall behind.
This brief book is a primer for those seeking to stay ahead of the innovation curve by adopting the principles of Continuous Quality. We wrote this as a guide for Dev, QA, Operations and other departments in charge of application development and quality, but believe it offers useful insights for anyone involved in the design or production of mobile apps.
7
New Dynamics = New Dev/QA Strategies New dynamics raise the stakes when balancing velocity and quality Time is not on the side of today’s Dev and QA teams – they must adapt to a number of emerging dynamics including:
Rapid Innovation New devices, OS enhancements, wearables, sensors and third-party plug-ins are continuously being released into the market.
Cloud Deploy
Mobile app development teams must constantly update their release plans to ensure compatibility with a growing number of variables. Feedback
These external pressures have compressed release schedules to weeks instead of months.
Client Deploy
Story backlog
Code Committed
Code Built Ver X+1
Client in-App Discovery Upgrade
Publish & Review
In-store General Availability
8
“After two bad experiences, I’ll try something else” Users uncover half of application defects
44% OF DEFECTS
ARE FOUND BY USERS
44%
24% Direct User Feedback
20% Public Users Reviews
http://info.perfectomobile.com/wpr-why-apps-fail.html
Skyrocketing User Expectations
Increased Complexity
Typical commerce websites will fully load pages in about 6-10 seconds. Mobile users expect faster results. Although few enterprises are achieving Google’s oft-cited one second (above the fold) content load target, studies show that users will quickly abandon applications with any response delays.
The desired mobile experience is dependent on a combination of factors – battery, memory, CPU / GPU, network connectivity, screen size, sensors and even app style (Web vs. Native). Ensuring quality in mobile environments requires enhanced testing, complex analysis and client side visibility.
9
Continuous Quality Is NOT Another Buzzword To effectively handle the increased velocity, quality and complexities associated with mobile development, industry leaders are turning to an innovative practice: Continuous Quality. Continuous Quality is a methodology for embedding quality-activities into every step of the SDLC process from design to deployment based on automated processes, tools and a lab infrastructure tailored for mobile. Successful Continuous Quality optimizes time-to-market, drives faster and more frequent releases and minimizes the defects that escape to production. The Continuous Quality (CQ) methodology is built on the notion of a robust test lab capable of supporting the full end-to-end development, continuous integration and delivery cycle. It specifies a model in which a single lab solution is capable of supporting all aspects of testing – unit, functional, build verification, performance testing and proactive UX monitoring. In addition, it is fully accessible to every tool and user in a company’s application development workflow. The result is an automation-ready iterative dev/test workflow capable of supporting a full range of mobile initiatives. 10
Making Continuous Quality a Reality
1. Real Environments
Anywhere, anytime, on-demand access to a CQ Lab capable of supporting all development activities from end-to-end is critical to the success of a CQ Program. The CQ Lab provides the required infrastructure and tools to conduct tests, measurements and analysis which are fully embedded and integrated into the Agile / DevOps process and able to replicate end user conditions throughout the lifecycle.
2. Real Devices
The CQ Lab supports a continuous, 24/7 operation with a production grade SLA and consistency across the different phases in the lifecycle, allowing reproducibility and minimizing time-to-resolution to maximize time-to-market.
Key building blocks of an effective CQ Lab solution includes
The key building blocks of an effective CQ Lab solution for mobile app development include:
11
1.
Real environments
2.
Real devices
3.
Coverage across the SDLC
4.
Integration with existing Tools
3. SDLC Coverage
4. Tool integration
What Got You Here Won’t Get You There As mobile requirements increase, companies face greater challenges in the way they develop, build, test and deploy end-user applications – forcing them to:
EXPERTS AGREE
Rethink Methodologies Enterprises that currently base their entire SDLC on traditional waterfall or preagile practices are quickly realizing that such solutions, which worked well in the web and desktop era, are inadequate in today’s mobile space.
Evolve Testing Coverage means more than just the code. For mobile, coverage necessitates implementing a device coverage strategy that addresses the current Android and growing iOS market fragmentation. Effective app testing requires mimicking real-world conditions including varied network conditions, device stress (e.g., background streaming media apps) and back-end load.
“The rapid pace of change in the mobile market is putting pressure on development and operations teams to adopt rapid development and deployment practices that constantly iterate their mobile applications as expectations change.” – Gartner, Traditional Development Practices Will Fail for Mobile Apps, 2014
Dev teams therefore need access to a variety of devices running varied OS profiles, actively connected to networks in target geographies.
12
Eliminate Obstacles Companies looking to capitalize on mobile opportunities cannot afford barriers that impede the velocity of development – especially those created by ineffectual internal processes.
EXPERTS AGREE
Typical challenges that organizations must overcome to effectively manage mobile app development include: •
Insufficient test coverage due to time constraints
•
Lack of stability and test consistency of the in-house test lab
•
Inability to enable cross-team collaboration
•
Gaps in production feedback being passed to product teams
“Poor app quality is the primary reason behind the weak adoption, usage and even failure of many mobile apps. Enterprises that are developing mobile apps need to employ mobile app testing tools and services to help mitigate the risks associated with testing increasingly sophisticated contextual mobile applications.” – Gartner, IT Market Clock for Enterprise Mobility, 2014
13
CQ Lab Component #1 Real Environments Design and development teams cannot effectively create mobile apps without an understanding of how server-network-device “wind tunnel� load factors impact performance. The CQ Lab must be able to execute tests on actively connected devices capable of reporting data from different network profiles (3G, LTE, WiFi) as implemented across the globe.
Server Domain
Network Domain
Device Domain
Back-end Servers
3rd Party / Cloud Services
ISPs
Web Services
Content Delivery Networks
Mobile Carriers
Customers
End-User Devices
Employees
14
CQ Lab Component #2 Real Devices One of the greatest challenges for CQ initiatives is ensuring that tests can be executed on a wide range of device/OS profiles. Not only must the CQ Lab provide robust device pools, it must also ensure quick access to the newest hardware and operating system releases to keep proactive, externally-driven release plans on schedule.
TM
This is doubly true for iOS updates. Upwards of 50% of Apple users will upgrade to the latest operating systems within the first 48 hours of availability.
15
CQ Lab Component #3 Coverage Across the SDLC The CQ Lab must be able to support all test scenarios from sprint zero to post-production including individual unit tests, functional testing on target devices, load testing for of production readiness and proactive user experience monitoring on live devices. Those evaluating mobile apps must be able to provide both object properties on all elements as well as visual results that allow testers to see exactly what the user sees on their device. With consistent results from an SLA-backed test environment, operations and development groups can more effectively collaborate throughout the SDLC.
16
CQ Lab Component #4 Integration With Existing Tools Test labs that require users to leave their known environments or learn new skills inherently restrain the velocity of a Continuous Quality initiative. To be effective, the CQ Lab must provide seamless, out-of-the-box integration with an organization’s existing development stack including open source and commercial programming languages, IDEs, CI servers and test frameworks. The days of enterprises having a single standard tool set is waning. Open source tools are changing the game and development teams expect integration with industry standard interfaces as table stakes.
17
CQ requires integration with market leading quality tools
In Practice The 4 Phases of Continuous Quality CQ Enhances Every Phase of Software Development Integrated quality enhances every aspect of software development, including design, code, debug, build, stage, deploy and production management. Regardless of individual implementation variances, CQ provides overarching benefits across the SDLC.
18
Phase #1 Development and Debugging Fast feedback Fuels Velocity: Mobile apps behave differently when run on different device/OS profiles and across varied network and location conditions which can introduce additional performance and usability issues. Therefore, developers need actionable data on performance impacting factors as early as possible during the app development phase. To provide developers with immediate feedback throughout the development process, build verification testing on target devices must be integrated into the organization’s chosen development environment. Executing unit tests on actual devices not only ensures early defect detection, but also provides critical data on device vitals, network bottlenecks, response times and back-end connectivity.
19
From a workflow perspective, CQ best practice recommends integrating test code development and execution into the developer’s preferred IDE (Eclipse, Android Studio, Visual Studio, etc.). It is also important to select the CQ Lab environment that supports the programmer’s preferred programming language (C#, Java, Ruby etc.) to reduce learning curves. Any incompatibility that delays the dev/test feedback loop will impact quality and/or delay release schedules.
Phase #2 Code Builds
When developers commit code and trigger the build process, they need a full report to quickly correct any issues that may impact user experience. Reports should include both test details (device OS logs, device vitals, network conditions, back-end delays) as well as visual recordings that enable developers to easily pinpoint the source of any failures.
A key piece of any CQ program is the ability to reproduce problems and ensure that they have been corrected before changes are recommitted into the master code repository. Because the CQ Lab supports the entire SDLC, developers can take any test code created during the during development and automatically include it in the build verification process. Even if teams have to perform additional interoperability manual tests to assure specific functions, a true CQ Lab solution will support those capabilities.
TM
Automation Drives Efficiencies: Because Continuous Integration (CI) is critical to agility, organizations need to integrate mobile testing into their individual code build process. To ensure efficiency, CQ practice recommends integrating the CQ Lab tests into CI process, regardless of which server – Jenkins, Team City, TFS – is used.
20
Phase #3 Integration & Pre-Production Scalability Ensures Readiness: In a CQ workflow, developers (either Android or iOS) perform code commits into a master code repository (SCM like Git, SVN, etc.) and each performs local build acceptance testing and debugging through unit tests and other verifications, usually on a small number of mobile devices. As development progresses to the point where the quality and robustness of the mobile app is expected to deliver against all functional specifications, any artifacts must be tested at scale for higher functionality and performance. To ensure quality under tight deadlines, the CQ Lab should serve as the common foundation for automating and performing these tests.
21
With seamless support from within the build environment for functional and non-functional performance testing, standard industry tools such as Selenium, HP UFT, Appium and Calabash, allow teams to validate quality and performance at scale across multiple device/OS/network profiles. Without this level of testing, companies cannot verify the release readiness of a mobile app with confidence.
Phase #4 Proactive Assurance Preempts Complaints Proactive Assurance Preempts Complaints: Production is a critical phase which can determine business success and end-user adoption. To stay ahead of issues, the product team needs pro-active user experience monitoring embedded into the CQ process. Such capability will assure that the team gets production insights and real time alerts as soon as something goes wrong. When proactive monitoring is performed on real devices in a governed, secured CQ Lab, mobile DevOps teams have full control of the entire workflow from the early warning through resolution, typically before endusers are impacted.
22
Continuous Quality A Framework For Growth
EXPERTS AGREE
Summary As development work evolves to encompass mobile environments, technical and business challenges mount. Release cycles accelerate and user expectations climb.
Recommendations
Poor app quality undermines adoption, conversion and ultimately becomes a barrier to the achievement of business objectives.
EMBRACE THE MOVE TO “CONTINUOUS QUALITY�
To meet rapid release schedules without compromising quality, it is imperative that development groups transition to CQ practices by employing a robust CQ Lab. This provides a consistent quality foundation across the entire Build-TestDeploy workflow. More importantly, it ensures that performance issues do not impact adoption, conversion or innovation KPIs.
In this first decade of mobile mass adoption, many developers have snuck by without comprehensive approaches to quality. With app and device complexity increasing, and consumers less forgiving of clumsy mobile apps, quality is quickly moving from a position near the end of the development life cycle to an embedded and continuous process throughout the cycle. Source: Forrester, Market Overview: Mobile app Testing, 2014
23
About Perfecto Mobile Perfecto Mobile provides a hybrid cloud-based Continuous Quality Lab that enables mobile development and testing teams to deliver better apps faster. The Continuous Quality Lab supports testing processes earlier and more often in the development cycle, giving way to faster feedback and improved time-to-market. Users can access an exhaustive selection of real mobile devices connected to live networks around the world and leverage them for testing and monitoring throughout the mobile application development lifecycle – from development, functional and performance testing to monitoring and support. More than 1,500 customers, including 50% of the Fortune 500 across the banking, insurance, retail, telecommunications and media industries rely on Perfecto Mobile to deliver optimal mobile app functionality and end user experiences, ensuring their brand’s reputation, establishing loyal customers, and attracting new users. For more information about Perfecto Mobile visit www.perfectomobile.com or follow us on Twitter at @PerfectoMobile.
24
Notes
25
Deliver Better Mobile Apps Faster Mobile apps have dramatically changed the way users interact with brands. But with the rapid pace of mobile hardware and software innovation, many organizations struggle to meet user expectations. These challenges are well understood; it’s the solutions which have proven to be elusive. We believe that the answer can be found in the principles and practice of Continuous Quality. This short book provides an introduction to Continuous Quality for the mobile executive, developer and QA manager including an overview of the framework behind Continuous Quality and a practical methodology for implementation. About Perfecto Mobile Perfecto Mobile, the world’s leader in mobile app quality, provides a hybrid cloud-based Continuous Quality Lab that enables mobile app development and testing teams to deliver better apps faster. Learn more at www.perfectomobile.com
© Copyright is 2015, Perfecto Mobile Inc.