bridging the gap between IT operations & security industry perspective bridging the gap between IT operations & security 1
IT operations and security have evolved as separate silos within the government enterprise, but they depend on the same information and ultimately share the same goal: Enabling the agency mission. Establishing a working relationship between these functions and sharing common intelligence can create a whole that is greater than the sum of its parts.
In the beginning there was IT operations.
the gap These are the people charged with making sure that networks and systems work and that IT resources are available. Reliability is the goal and uptime is the metric. Security was the province of the guys with badges who guarded the gates and made sure the doors were locked. Then about 20 years ago—give or take a few years—the internet changed everything. Networking became global and the nature of threats changed. People could reach out and touch you from almost anywhere in the world and it no longer was enough to secure your physical location and your enterprise perimeter. Cybersecurity came into its own, charged with looking for vulnerabilities and blocking threats and attacks. Uptime had to be balanced against the needs of security. Given their origins, it’s not surprising that these two functions – IT operations and cybersecurity – developed in silos in the public sector, with little cooperation and communication – and all too often seemingly working at cross purposes.
industry perspective 2
There are some legitimate differences in their functions. IT operations manage and monitor the day-to-day activity and performance of the infrastructure, handling the provisioning of capacity to ensure that end users have what they need to do their jobs. As agencies depend more on the information infrastructure, operations becomes even more critical to the agency mission. Security, on the other hand, is more concerned that access, data and resources are protected from outsiders and from insiders without legitimate need. To users, security often is the shop that says “no” when something new is needed. Ultimately, however, IT operations and security share the same goal: Enabling the agency mission. A system or network that is not adequately secured is not reliable, and one that is not properly managed and monitored cannot be adequately secured. Splunk, the leading platform for both operational and security intelligence, provides a platform to bridge the gap between these functions by allowing agencies to collect data from systems once and use it many times. In this industry perspective, GovLoop and Splunk look at how government organizations can bridge the gap between IT operations and security to better meet agency mission and serve citizens.
two sides of the coin Despite their different points of view, IT operations and security depend largely on the same information in carrying out their jobs.
John Stoner, Security Architect for Splunk Public Sector, estimates that there is an 80 percent overlap in the data being gathered in the Network Operations Center (NOC) and the Security Operations Center (SOC). Each needs end-to-end visibility of the infrastructure to see user behavior, spot suspicious activity and understand system and device status. Each uses this data to answer its own questions:
IT Operations: Are the systems running? Are they available? Security: Is the enterprise secure? Can we detect intruders and suspicious activity? It makes sense and would improve efficiency if the two sides shared not only the data being collected but also the answers to their questions, said Bill Babilon, IT Solutions Architect for Splunk Public Sector. “But more often than not we find that data sharing is not happening as much as we would like.”
legitimate differences Like the two sides of a coin, each is different. There are situations in which security needs to limit access to certain data. These can include maintaining a chain of custody for forensic investigations and ensuring that sensitive data such as personally identifiable information is not being exposed.
On the other hand, system data generated to help IT operations make informed decisions contains critical information on users, customer activity and capacity consumption that is not routinely needed by security personnel and that could pose privacy risks if not properly handled. Each side is rightly jealous of this data and the privacy of its constituents. But while acknowledging the specific needs of each function, failure to recognize common goals and to take advantage of the common resources creates a sub-optimal situation for both IT operations and security.
common ground scenarios Machine data -- data created by the activity of computers, mobile devices, embedded systems and other networked devices -- is critical to operational intelligence. Operational intelligence turns machine data into valuable insights that gives you a real-time understanding of what’s happening across your IT systems and technology infrastructure and can provide a wealth of information on network activity that does not directly affect performance, but might be valuable indicators of malicious activity for the security team. And some security issues, such as a Distributed Denial of Service (DDoS) attack, have an immediate impact on availability. In these situations, sharing information can help both sides. As the first line of defense, security uses tools that identify known threats—those threats that already have been identified and can be detected by their signatures. However, this approach is ineffective against advanced threats that utilize new and stealthy exploits as well as rapidly morphing malware that makes signatures obsolete. Operational data can be helpful in hunting out these unknown threats.
bridging the gap between IT operations & security 3
Advanced threats can be delivered through a variety of vectors to slip past defenses. Gaining access to operational intelligence produced from host and network data can help defenders detect these threats. This data includes: êê Suspicious file names in system logs, êê Unusual executables and processes in process logs and registries, êê Unusual administrator activity in event logs, êê Malicious command and control traffic from web proxy and firewall logs, and êê Malware delivery from web proxy and firewall logs. Domain Name Service (DNS) traffic is an increasingly common way for attackers to hide communications with compromised servers and to exfiltrate data. DNS traffic often is not secured and data masquerading as DNS requests can be sent past the enterprise’s firewall without detection. This malicious traffic can be difficult to detect because DNS is ubiquitous and these requests form routine traffic to the Internet. But if not spotted it can put an agency’s most sensitive data at risk. But operational intelligence can contain indicators of DNS exfiltration that might be ignored by the operations team because they do not create performance problems. These can include: êê The presence of encrypted DNS traffic; êê Repeated requests to a single domain, a restricted domain, or to rapidly shifting domains that could be hiding botnet activity;
Server errors are another type of operational information that could point to security incidents. Error messages can cover a range of problems from the inconsequential to serious. Errors that do not have an impact on performance might not be investigated, but log analysis could reveal patterns or unusual activity that could be indicators of malicious—or at least suspect—activity on the network.
Freeing data from their silos can provide end-to-end visibility and offer insights, increasing the value of data being collected and empowering workers in both IT operations and security.
êê Recognizable patterns in requests; and êê Unusual packet sizes or spikes in traffic.
Splunk gives you the following functionalities to help improve your security posture:
incident review & classification
reports & security metrics
risk-based analysis
industry perspective 4
threat intelligence framework
unified search editor
bridging the gap Organizational structures vary, but IT operations and security typically are housed in separate offices within the agency, with separate chains of command with little common oversight
These barriers reinforce the silos in which they operate. Many agencies have established, or share with other agencies, Network Operations Centers (NOC) and Security Operations Centers (SOC) to carry out these functions. Bridging the gap between them does not have to mean merging these separate organizations. Building bridges is about sharing of data and information rather than integrating functions. “There needs to a working relationship,” Babilon said. “It’s about building trust and knowing that they have each other’s back.” When each team is aware of the other’s concerns and of how they can help each other, they can develop a holistic picture of the enterprise by making machine-generated data available to both rather than keeping it in silos. This holistic view allows IT operations to act as security first responders, providing eyes and ears on the front line that can help spot unknown threats through unusual activity. This enables a response before threats can create a breach. This awareness reduces both the meantime to investigate incidents and the mean-time to recovery for operations, allowing them to fix problems faster and keep systems up and running.
a new context Providing a first line of defense for security does not have to add to the tasks of the IT operations team, said Stoner. “It just means asking a few additional questions and adding some things to consider when evaluating a system. Is there a security context?”
Babilon recalled when he became aware of the security context when he was a systems administrator. When performing updates, he found that not all patches were being applied. He approached the problem from a purely technical perspective, but after he happened to mention it to a member of the security team and it was soon discovered that a malicious insider had disabled patching so that vulnerabilities could not be fixed. Sharing what was a technical problem from an operational point of view resulted in the discovery and remediation of a serious security issue before it resulted in a breach. That was a matter of chance, Babilon said. Institutionalizing this type of sharing can produce lasting improvements. Security is a triad of people, process and technology. When teams look beyond the technology issues, the security context becomes clearer. This allows the organization to move beyond reacting to threats when they appear to proactively managing risk. “I can be smarter in how I approach a situation if the risk is understood,” Stoner said. But, breaking down silos takes time he said. Building trust and expanding awareness of security issues cannot be done all at once, even at the highest levels of the organization. “What you need to do is light small fires,” Babilon said. “Start by finding points of commonality with peers on the other side.” There is a cost to getting it going, but when the advantages of breaking down data and organizational silos become clear the task becomes easier.
“There needs to be a working relationship. It’s about building trust and knowing they have each other’s back.” Bill Babilion, IT Solutions Architect for Splunk Public Sector
bridging the gap between IT operations & security 5
the bridge Taking full advantage of the data already being generated by systems to benefit both IT operations and security requires a platform to gather this information once and maintain it in a form that allows it to be used many times.
This supports analysis to create both operational and security intelligence, providing end-to-end visibility across all of your platforms. The Splunk platform enables this by gathering data from all IT layers and keeping it in their original formats, allowing you to collect it once and make it available for analysis across the enterprise. Because the data remains in its native format and does not have to be modified to fit into a database, the platform maintains a wide range of unstructured data and formats that can be used by each team as needed. The data is correlated and synchronized by time, and powerful keyword searching allows analysts to connect the dots to identify problems and track end-to-end performance. Splunk is a single integrated platform that provides a full range of functionality. Because data is gathered from across the enterprise and maintained in its native format, it can be used for monitoring for real-time situational awareness, for analysis to create intelligence and spot threats or malicious activity, and for forensic investigation. This collect once, use many times model requires the ability to fully manage data and track its use. Role-based access controls enable dashboards that provide appropriate views for each stakeholder. Sensitive data is tagged so that users access only relevant data, and user activity is logged.
industry perspective 6
The result is a holistic view of the infrastructure that allows each team to do its jobs more effectively without compromising privacy. This generates a greater return on investment from data being generated by systems. Each team also is able to help the other, backing each other up to dramatically increase the efficiency of both IT operations and security. Money that would otherwise be spent on maintaining separate systems can be used more productively. This helps the agency to focus on its primary mission of delivering citizen services rather than merely “keeping the lights on.” The information needed to enable these advances already is there. The challenge—and the opportunity—is to free it from its silos to provide end-toend visibility and intelligence that both IT operations and security can use.
about govloop about splunk Splunk Inc. provides the leading platform for operational intelligence. Splunk® software searches, monitors, analyzes and visualizes machine-generated big data from websites, applications, servers, networks, sensors and mobile devices. More than 11,000 organizations use Splunk software to deepen business and customer understanding, mitigate cybersecurity risk, improve service performance and reduce costs.
GovLoop’s mission is to “connect government to improve government.” We aim to inspire public-sector professionals by serving as the knowledge network for government. GovLoop connects more than 250,000 members, fostering cross-government collaboration, solving common problems and advancing government careers. GovLoop is headquartered in Washington, D.C., with a team of dedicated professionals who share a commitment to connect and improve government. For more information about this report, please reach out to info@govloop.com. www.govloop.com | @GovLoop
better IT operations through operational intelligence 7
1152 15th St. NW, Suite 800 Washington, DC 20005 Phone: (202) 407-7421 | Fax: (202) 407-7501 www.govloop.com @GovLoop
industry perspective 8