How Salesforce Built the Fort Knox of Data Security

How Salesforce Built the Fort Knox of Data Security process.st /salesforce-data-security/ 11/15/2017

Think about what’s at stake if Salesforce suffered a breach…

American Express, Philips, Vodafone, Virgin, Western Union, GE, the U.S. government, and 150,000 organizations risk their communications leaking to the public This includes trade secrets, financial documents, passwords, and bank details Hackers could believably impersonate bank employees or government officials It’s a terrifying concept for Salesforce customers, and just as frightening for Salesforce themselves; if a breach like that happened, their $61b company would suffer tremendously.

A company as well-known as Salesforce has a huge target on its back, so it needs to have the highest security standards even while scaling up and innovating rapidly.

How complex are Salesforce’s IT operations?

Salesforce has 100,000 pieces of hardware across data centers all over the world, serving almost 5 billion daily transactions to 150,000 customers.

Since founding, its hardware infrastructure has grown from being manageable in a single spreadsheet to requiring the assistance of a powerful internal suite of tools. To give you an idea of the scale we’re talking about, here are some facts about their data center operations: A piece of data center hardware is received, moved, relocated or decommissioned every 30 seconds.

90 pieces of hardware are serviced every day at Salesforce data centers Change management processes run every 30 minutes 1/10

Salesforce has nine data centers: four in North America (the region that brings in 71% of the revenue), three in Europe, and two in Japan. Each data center hosts anywhere between 8 and 66 instances of Salesforce, depending on the demand for services in that area.

Hosting SaaS at this scale presents complex problems. As Salesforce gets more popular, transaction rate and data size increases. To handle higher demand without increasing latency, disrupting service or opening security vulnerabilities, a SaaS company needs to elegantly scale its IT operations.

As the grandfather of SaaS and the world’s most popular CRM, Salesforce was forced to scale its cloud services to loftier heights than any technology company that came before. There was no blueprint, playbook, or best practice guide — the company devised new methods from scratch. How did it manage? In this article, I hope to tell that story.

Scaling Salesforce to serve 3.5m users In the formative years of SaaS, the concerns of corporations centered around security and reliability. Companies were used to housing their own data, and sending it off to a black box where they couldn’t be sure it wouldn’t get lost or corrupted was a nerve-wracking idea.

As Salesforce tried to scale IT in line with its lightning fast adoption rates worldwide, it suffered inevitable initial disruptions. In a 2007 paper, THINKresearch reported the launch of trust.salesforce.com in reaction to a series of technical issues:

“During a three-month period from December 2005 to February 2006, Salesforce.com experienced service disruptions which limited customer access to the company’s online solutions and renewed the debate about the reliability of SaaS in general. In acknowledging the problems, Salesforce.com’s founder and CEO, Marc Benioff, made a public commitment to work toward achieving 100% service availability and setting a new standard for service level accountability in the process.

2/10

In addition to making significant investments to fortify its data center and service delivery infrastructure capabilities, Salesforce.com also took the bold step of establishing a public website, called trust.salesforce.com, to continuously report its service availability and performance levels.”

Salesforce’s trust page was, as they claim, the world’s first system status page. In it, you can see that they’re serving just three instances: Asia Pacific, EMEA, and North America 1. Salesforce instances are systems running everything a customer needs to use the service. They are independent, which provides extra security because a single compromised instance can be taken down and diagnosed while users are switched to their closest active instance.

In a 2014 Dreamforce presentation, Salesforce architect Ian Varley explained that the decision to provide Salesforce in instances is directly related to scalability. Scaling, he says, is simply a matter of creating more identical instances hosted on data centers in heavily trafficked locations.

Creating more instances in more data centers rather than upgrading to more powerful machines is horizontal scaling. Scaling horizontally is safer because it reduces the points of failure in the system. If, for example, you were serving an application to 1,000,000 people on a single powerful server, it’d be safer to serve it 250,000 people on four servers that each had a quarter of the power.

If Salesforce can make a single working instance that serves 10,000 customers, serving 20,000 customers is simply (spoken like a true non-techie) a matter of creating an identical instance and serving that instance to the nearest 10,000 customers to distribute the load between the two.

3/10

The data centers that host Salesforce’s instances are just as — if not more — impressive. Next up, we’ll look at the physical systems that protect their data.

Inside a Salesforce data center

Salesforce scales its servers horizontally to increase redundancy (image source)

4/10

Earlier in the post, I explained the consequences of a Salesforce breach. Instanced systems are part of their safety precautions, but they also need to consider the physical security of their servers.

So, how does Salesforce protect its data centers against physical attacks?

“The exterior perimeter of each anonymous building is bullet resistant, has concrete vehicle barriers, closed-circuit television coverage, alarm systems, and manned guard stations, all of which help defend against non-entrance attack points. Inside each building, multiple biometric scans and guards limit access through interior doors and cages at all times.”

Let’s break that down. First of all, information on exactly where these data centers are is not available . The most accurate estimate comes from a Stack Exchange user who traced their Chicago data center to… somewhere in Chicago.

Next, they’ve got a bullet resistant perimeter and vehicle barriers.

Vehicle barriers are deployed against attacks at airports, banks, military facilities, and locations that could be vulnerable to a vehicle smashing down the gate or wall.

In the image below, an 80,000lb vehicle is immobilized by a concrete vehicle barrier.

5/10

(Source)

In the event that an intruder manages to get to the door of a data center, they’ll either be caught by the exterior guards or blocked from entering by the facility’s biometric scanners.

The reason that physical security relies on biometric scanners over passcodes is that they are convenient for the staff. In the event of an emergency, getting a staff member to punch in a secure password would surely impede their 6/10

ability to respond in a timely manner. According to Apple, the probability of a fingerprint match is 1 in 50,000, whereas the probability of an intruder guessing a 4-digit passcode is 1 in 10,000.

Salesforce doesn’t specify whether the scanner accepts a fingerprint, an iris, or something else, but whichever it is, it’s more secure than any practical passcode.

As far as we know, Salesforce has never suffered a physical breach at one of it’s data centers. This could be because of the tight security, but it’s also likely because it’s much safer to attack SaaS remotely.

Data security at Salesforce In this section, I’m going to move past the design of Salesforce’s systems and their physical security measures, and instead focus on what we know about data security and their penetration testing practices.

Broadly speaking, Salesforce ensures the security of user data in two ways: 1. Stateful packet inspection at firewall, bastion hosts, and TLS/SSL encryption (more on this later) 2. Training users to set up secure permissions and look after their data Other than a phishing scam email opened by a Salesforce user in 2007, there are no formal reports of Salesforce being breached in any way. The company has always taken security seriously, partly because it was such a major concern for the platform’s early adopters.

Stateful packet inspection 7/10

A stateful firewall inspects data coming into a network, and filters out anything that doesn’t come from a trusted connection. Stateful firewalls were previously unworkable in practice because they required so much memory to operate, but now they are standard high-grade security for business applications.

The firewall stores a connection’s attributes including the port, IP address, and packet sequence numbers. Anything contrary to the attributes of its trusted connections is filtered out before it goes any further into the network.

Bastion hosts Bastion hosts are computers that have been specially designed to withstand security threats. They run the minimum amount of software to decrease their vulnerability, and are situated as barriers between the perimeter and core firewalls. Any attacks on Salesforce must first breach these machines — so far, none have managed.

TLS/SSL encryption TLS/SSL is a standard for a reason. When data is transmitted between Salesforce and a user, it is hashed into a string like this: 3727de7e10777a1c90634d22701ee22a84e9ec136a6a847694cbe7417d47d6e0 This hash is irreversible unless the host and client share keys which tell the connection how to decrypt it. Cracking SSL is theoretically possible, but it would require every supercomputer on the Bitcoin network working for hundreds of millions of years straight.

Penetration testing Like many other software companies, Salesforce hires a team of people tasked with attempting to hack its systems, and a team of people who try to (Source) thwart the attacks. These teams — red team and blue team — are constantly simulating and defending against risks. In the Def Con presentation below, two ex-Salesforce red team members talk about their cutting-edge methods. The video is well worth watching, but the tone might make you cringe.

If you want to avoid the video, here’s the summary. Josh Schwartz, director of offensive security, and John Cramb, a senior offensive security engineer, reveal Meatpistol — a tool to manage and track the malware they create and try to implant into Salesforce.

The pair of presenters revealed so much about Salesforce’s security methodology, they were fired directly after the conference.

8/10

“Meatpistol, while still in its early stages of development, had already improved the efficiency of the Salesforce red team. “Malware implant creation used to take days,” Schwartz said during his presentation. “Now it takes seconds,” he said, cutting “weeks off our operation time.”

Not only are Salesforce servers protected by layers of encryption and bulletproof coating, they have also withstood assaults that are so complex and innovative that they need to be managed with a fast-paced internal malware creation tool.

Training users and administrators to keep their data safe A system is only as secure as its weakest link. Even with the Fort Knox of data centers and encryption which would take hundreds of millions of years to crack, there’s nothing stopping an

(Source)

(Source)

irresponsible admin giving login permission to a malicious third party, or sharing confidential data by accident.

To prevent that, Salesforce puts a lot of effort into training its administrative users. Trailhead is a Salesforce site designed to help users learn the platform. In it, you’ll find hours of data security training for admins which details how they can block IPs, block log in times, set password rules, and more.

Salesforce also has hundreds of hours of video content on data security, some aimed at training developers and admins working in the Salesforce ecosystem.

How can you build a secure, scalable system like Salesforce?

I’m not going to go any deeper into the technical specifications of Salesforce’s security and scalability, but I will give 9/10

you a few general takeaways to keep under consideration: Build scalability into your systems by reducing them to their smallest working instances, and then scaling horizontally Run instances independently to stop the spread of vulnerabilities Protect data with multiple layers of digital and physical security

(Source)

Train developers, admins and users on data security best practices

10/10