HARNESSING THE POWER OF BIG DATA
IS GOVERNMENT READY FOR BIG DATA?
Big data has revolutionized the way government agencies store, manage and collect data. Agencies can no longer rely on stable and structured data as a means to meet mission need. The way citizens engage with government has evolved, and the kinds of data processing services available have changed.
In our survey of 152 public-sector employees, GovLoop explored how agencies are defining big data. (See Figure 1.) “Everyone defines big data with the V’s: volume, velocity and variety of data. I would focus more on the variety,” said Kevin Shelly, vice president of the public-sector business at MarkLogic. “Traditionally, most of the data in databases has been of the structured variety. The structured data is now only 20 percent of the world’s data, and the other 80 percent of the data is unstructured.”
Today, agencies must be able to quickly and efficiently capture and manage a multitude of data types, including unstructured data, such as e-mail and social media. Agencies must filter through the noise and turn volumes of information into actionable insights. Through qualitative and quantitative analyses, this executive research brief will highlight the common challenges, best practices and pertinent issues surrounding big data adoption in the public sector. Our report includes: •
Results from a big data survey of 152 public-sector employees.
•
Insights from industry expert Kevin Shelly, vice president of the public-sector business at MarkLogic.
•
Best practices for big data adoption.
It’s clear that big data offers the opportunity for government agencies to improve their missions performing and service delivery. But we wonder: Is government ready?
Figure 1. What level of government do you work for?
The growth in unstructured data has led governments to think differently about the tools, opportunities and challenges in turning data into meaningful information. According to our survey, 58 percent of respondents believe government is not ready for big data to be fully adopted. “I have found that most department and agency [information technology] infrastructures are subpar, and the majority of users don’t have more than a basic knowledge of how to use the tools available to them,” one survey participant said. Other responses indicated that information silos, data quality and governance are lacking across government. Several respondents indicated that because big data is still new to government, many agencies are just in the early discussion phases of how to use it. “We need to make sure government is optimized for big data,” Shelly said. “There are some infrastructure and skill sets that the government needs to embrace to optimize maximize the value of big data to their specific organization and mission objectives.”
Figure 2. What are your big data challenges?
How are you defining big data?
Is government ready for big data?
1
|
Harnessing the Power of Big Data
What is preventing you from adopting a big data solution?
HOW GOVERNMENT IS LEVERAGING BIG DATA The survey dug deeper into the big data challenges that agencies face. Figure 2 shows that respondents cite a lack of understanding of how big data can transform an agency as the biggest, closely followed by the need for improved management support and clarity of value (59 percent). Respondents identified additional challenges as well, such as accuracy and completeness of data, inconsistent data, and a lack of resources or time to allot to big data programs.
In our survey, we found that most agencies are obtaining limited insights from data on their day-to-day operations. (See Figure 4.) For big data enthusiasts, and for those who recognize the power of big data to transform agencies, these numbers should not be disheartening – they reveal the incredible opportunity to champion big data initiatives within government.
Figure 3.
How would you assess your ability to do the following?
Figure 4.
What are the benefits of big data?
Despite these challenges, some big data solutions have taken root in government and many agencies are becoming more data-driven. Seventy-one percent of respondents noted the leading benefit of big data as “increased ability to find information that is critical to organization success,” followed by “improved efficiencies and decision-making” (68 percent) and “timely decision-making” (56 percent). (See Figure 3.) We believe government has to be ready for big data because it provides an unprecedented opportunity for organizations to leverage information to drive improved operational insights. “Government needs [to embrace big data] whether they like it or not,” a survey participant said. “By using big data they will reach a wider audience and be more transparent, which will bring about a better trust in government.” But first, government needs to learn from how others are excelling with big data and discover the core elements of a big data solution.
But leaders must make the business case for big data. One of the best ways to do this is to identify specific use cases to show the value of properly managing and using data. Our survey provided insights as to how big data has been used to support mission-essential functions. Examples include: •
Allocating scarce resources.
•
Making budgetary decisions.
•
Changing labor hours based on time, location and demand to reduce working hours.
•
Targeting recruitment to increase results where there are skills gaps.
Executive Research Brief
|
2
Figure 4. (cont’d) What statement best describes your current use of big data?
“[Big data] has allowed a laser-like focus on utilizing trends to manage resources better prior to events, rather than reacting to events as they happen,” a respondent said. As agencies start to adopt big data technology, we asked what they require most in a solution. Consistency and standardization of data are the leading demands (65 percent), followed by access to information and collection from various data sources, both at 44 percent, according to the survey results. (See Figure 5.) “When looking at big data solutions, it’s not a simple rip and replace,” Shelly said. “Agencies need to find technologies that you can plug into their current ecosystem rather than go out and build completely new enterprise-scale systems. As you deploy new big data technology, it’s more a low-risk approach to leverage existing infrastructure, and as you see the fruits from your labor, you can continue to go down that path and even accelerate adoption.”
3
|
Harnessing the Power of Big Data
What are your challenges in leveraging big data?
As government continues to adopt big data solutions, it’s essential to consider the proper best practices and lessons learned to assure effective deployment.
Figure 5. What do you require most in a big data solution?
YOUR BIG DATA BEST PRACTICES Big data can become a reality at your agency. Government is already collecting and managing more data than ever before, and big data can help you continue to improve citizen services. To see what big data can do and how you can start using it, consider the following best practices from your peers, as gleaned from our survey.
What kinds of data types do you currently search across your agency?
Make the Big Business Case To gain support from managers, you must have a strong business case. “Plan before implementation of new programs and [know] how to integrate that new program within the structure of the whole,” a survey respondent said. “Most often, it is left to those without the knowledge of what is needed to simply plug it into the existing structure.” Making the business case was a very strong theme in our survey. It includes showing how big data solutions will ease the burden of limited financial and workforce resources and lead to significant return on investment because of better efficiency.
Are government personnel or contractors building your applications?
Know Your Data To fully leverage data, agencies must start by assessing who uses it, how they use it and how they access it. This will help agency officials understand what problems to tackle, the best solutions and the right vision going forward. “Know what data is available and what it means,” a survey respondent said. “Clearly define who is responsible for the analysis and who is responsible for using the results, and then spend the money for the right tools.”
Identify a Clear Organizational Problem Thousands of problems can be solved or addressed using big data. First identify the problem to ensure you are using the best data and addressing the right issues. This also means that you will have to focus on collaboration and team building to know that you have all the right voices at the table. “You must be learning more about the people you serve and what works,” a survey respondent said.
What are some of your challenges with application development?
Part of making the business case also involves identifying the right skill sets and workforce needs. Big data solutions and technology are often cutting-edge and a great opportunity for government to showcase the unique data-driven programs it has undertaken not only to other agencies, but the public at large. “The way to attract new talent into an agency is to do interesting programs and use new and interesting technologies, and if you can do that, you will attract some of the most talented workers for big data,” Shelly said. “So I think publicizing some of the initiatives and using the newer technologies will attract new people to government.”
Storing Data in a Single Repository Although collecting data is important, having the ability to store or accessing information and from a single platform is essential. Our respondents noted the many benefits to having data stored in one place:
Do you aggregate data from multiple sources?
Start With the Low-Hanging Fruit With big data, it’s best to start small: Identify some important use cases, obtain a few wins and then scale out your program. This will also help build consensus on the team, and show that efforts are successful early on.
•
“[A single repository] would free up a lot of people’s time to do other things.”
•
“With [a single repository] we can develop a common understanding of need that may be shared by stakeholders.”
Executive Research Brief
|
4
•
“A single location helps improve decision-making and remove inefficiencies.”
•
“[A single repository] would be fantastic; it would add many more dimensions for better decisionmaking.”
One recurring theme from the survey was the ultimate reason for using big data: improved decision-making. “When you integrate data from various sources, you obtain the ability to make sense of the data from a variety of disciplines and have a complete view of an issue,” Shelly said. “When you have more data and a complete view, you can make more fact-based decisions and identify the best way to solve an issue.”
Finally, by adopting big data solutions, you can start to provide information anytime, anywhere. “When you ingest data from a variety of data sources, you then need to make it available to stakeholders whether they are on a PC, tablet or a mobile device,” Shelly said. “Information should not be hampered by your device.”
Meeting Government’s Data Management Needs Making sense of so much data – structured and unstructured alike – can be daunting. MarkLogic can help.
How often do you need to search across data sources?
But having a single repository isn’t feasible if you don’t collaborate and work to break down silos. “People should not only be acquiring the data, but also sharing the data. It’s really only when you combine different datasets that you get the true insights and the power of big data,” Shelly said.
Master the Art of the Possible The final best practice is mastering the art of the possible – that is, being able to show your peers and colleagues what can be done with data. Having a clear vision and communicating that vision effectively is essential to big data success.
How long does it take you to aggregate data from multiple sources?
To help you think about what is possible, we asked our survey respondents to tell us what they wanted to do with their agency’s data. Here are some of their answers:
5
|
•
Collect and display it on a regular basis to allow for analysis and quality improvement.
•
Consolidate data across disparate systems.
•
Develop programs that will have a clear ROI as opposed to doing things that have been done in the past.
•
Identify trends, reduce costs, improve processes, create new revenue streams.
•
Make timely decisions and evaluate programs.
•
Performance management, revenue allocation, audit selection.
•
Plan infrastructure investments and collect input from users.
•
Detect fraud, waste and abuse.
•
Improve public outreach.
•
Help create a picture of what is going on in day-today operations to determine a real-time response.
Harnessing the Power of Big Data
“The opportunities for government to derive value from big data are limitless,” Shelly said. “By accumulating the data and making better fact-based types of decisions, we can better address issues such as health care, fraud protection and protecting our country from those who wish to do us harm. We can also make better decisions on the environment and the world in which we live, and use data to better protect against disease outbreak and traffic patterns.” Big data is changing the way our government engages with citizens and delivers critical services. It’s also playing an essential role in improving the efficiency and business of government. For government leaders today, understanding data and how it can improve their agencies is critical.
How valuable is the opportunity to write once, and publish everywhere?
“MarkLogic not only helps government,” Shelly said, “but we improve the lives of citizens by harnessing the power of data and helping communities make factbased decisions.”
ABOUT GOVLOOP
ABOUT MARKLOGIC
GovLoop’s mission is to “connect government to improve government.” We aim to inspire public-sector professionals by serving as the knowledge network for government. GovLoop connects more than 140,000 members, fostering cross-government collaboration, solving common problems and advancing government careers. GovLoop is headquartered in Washington, D.C., with a team of dedicated professionals who share a commitment to connect and improve government.
For more than a decade, MarkLogic has delivered a powerful, agile and trusted Enterprise NoSQL database platform that enables organizations to turn all data into valuable and actionable information. Organizations around the world rely on MarkLogic’s enterprise-grade technology to power the new generation of information applications. MarkLogic is headquartered in Silicon Valley with offices in Washington D.C., New York, London, Frankfurt, Utrecht, and Tokyo.
For more information about this report, please reach out to Patrick Fiorenza, senior research analyst, GovLoop, at pat@govloop.com.
For more information, please visit www.marklogic.com.
GovLoop 1101 15th St NW, Suite 900 Washington, DC 20005 Phone: (202) 407-7421 Fax: (202) 407-7501 www.govloop.com Twitter: @GovLoop
Executive Research Brief
|
6
1101 15th St NW, Suite 900 Washington, DC 20005 Phone: (202) 407-7421 Fax: (202) 407-7501
7
|
Harnessing the Power of Big Data