shooting: Your Equipment

“We are very integrated with our friends in IT,” the engineer says. “In most organizations, there are two di erent worlds—IT lives in their own world, and OT, which is the operational technology world, lives somewhere else. In our case, we are very converged.”

The shop floor has tended to avoid IT as much as possible, comments Dan Barrera, product manager for ctrlX Automation at Bosch Rexroth. “But COVID accelerated things; it created that demand to really incorporate the IT and the OT world together.”

It’s important to maintain transparency between IT and OT, and between brand owners and their OEMs. “If you are not clear, if you say, ‘We are just going to have a device,’ the IT organization will never allow you in,” the CPG engineer says.

The cybersecurity part of this discussion could fill its own article (see “Put Your Cyber Defenses Up Before They Take You Down,” pfwgo.to/7504), but it’s a vital consideration as any CPG works toward a troubleshooting solution. According to a 2020 sur- vey of CPGs done by PMMI, cybersecurity concern was the top barrier to setting up remote access.

“Users are very suspicious of hooking anything up behind their firewall. And there’s a raging battle right now between the IT department and the OT department about who even should get to control the what and who hooks up the equipment on the factory floor behind the firewall,” says Spencer Cramer, founder, chairman, and CEO of ei3 , and a pioneer in remote connectivity. “So if you’re a machine builder, and you don’t have a fully featured sta of experts who are putting together your IoT solution, you’re going to find that the end customer is going to be very suspicious of putting your system online.” troubleshooting can enable technicians to get support from experts, even if they’re not physically in the same room with them.

There is plenty of reason to be concerned about the security of connections within your manufacturing facility. “Many in the food and beverage industry will be concerned about losing intellectual properties if they connect equipment and put the wrong data in the wrong hands,” Cramer says.

The benefits are too real to be ignored, however. So rather than deny the connections, Cramer recommends partnering with companies that can help address vulnerabilities.

A key element of remote access is that everyone has a unique capability and also their own way of doing it, notes Mark Fondl, vice president of product management for remote access at ei3. He also leads the Digital Transformation Workgroup at The Organization for Machine Automation and Control (OMAC), working on best practices for remote connectivity. “All these di erent variations can be very di cult,” he says. “There needs to be a collaborative acceptance between the IT organization and the OT organization for how to handle safe and secure access and then limit that to a specific type of technology—so that you can control it and manage it.”

Key to this success is coming up with a connectivity solution that not only satisfies the demands of IT but also enables OT to be flexible when there’s a problem. Otherwise, OT will go right back to ignoring IT and putting in a solution that bypasses the IT organization, Fondl says.

Need for standardization?

The vast majority of OEMs will adapt to a food or beverage producer’s needs, according to the CPG. “But there are some who are new to this. They may have developed one solution and they just o er that solution,” the engineer says. “That is where we will get into trouble, if they are not nimble.”

Though a less experienced end user might welcome accepting whatever the OEM is prepared to provide, a larger company with a higher level of maturity will likely have di erent demands, he notes. “That company will say, ‘We are not going to allow a foreign device that can get unrestricted access to our network. You have to come through our system so we can manage who comes into our system, and we can monitor what is being done.’”

That’s exactly the sort of thing this large multinational company is able to do. To standardize on how OEMs from all over the world can access their respective equipment, the CPG developed engineering stations at each of its 16 plants and asks that the OEMs log in through those stations. The CPG uses Rockwell Automation’s FactoryTalk AssetCentre as a repository and disaster recovery for all its software. It’s through this that OEMs get access—after logging onto the server through the VPN—to the software and documentation they need to get the job done.

The right way to connect

At the end of the day, there is likely not only one way to make the right connection to machines to enable remote troubleshooting. But there’s usually a way to get it done safely.

Remote connections have somewhat of a renegade history, with people throwing a cellular connection on a machine to get access. “Through lack of anything else, that’s the way people were doing it, and it’s really very insecure,” Fondl says. “And so the industry tended to be very much against remote capabilities in manufacturing.”

There’s also a general skepticism in the industry about whether the OEM will be able to correctly diagnose a machine remotely, according to Bryan Gri en, senior director of industry services for PMMI. “When I was with Nestlé, there were times that we would bring OEMs in to fix the machine, and the guy on site would really struggle trying to fix his own machine,” he says. “So there’s some lack of confidence that the remote connectivity is actually going to solve the problem.”

Prior to COVID, remote connectivity tended to be a taboo subject, with customers refusing to connect their machines, says Justin Garski, Americas OEM segment manager for Rockwell Automation. “And then the world shut down, and all of a sudden people were like, ‘Man, our factory’s not going to run unless somebody fixes this thing.’”

FPS provides a remote monitoring system that enables its technicians to monitor customer freezer units.

It was in this atmosphere, in late 2021, that Rockwell launched a remote access solution to help OEMs more easily realize the benefits of using remote access. It combines cloud-based software with a remote access router. “The OEM connects to the cloud, the machine connects to the cloud, they get married there, and then you’ve got this tunnel back and forth,” Garski describes.

Though the food industry might’ve been a little late to connect, the pandemic opened some doors to technology that had previously been blocked, Gri en notes. “All of a sudden, sometimes my only option was: I’ve got to open a channel here and let somebody into my equipment.”

But that kind of hurried response to connectivity introduced exposed operations to possible attack. “We were in an emergency mode. We were just going and doing what needed to be done,” Gri en says. “Now let’s clean this up. Let’s make sure that it’s a good option going forward.”

Because of the growing importance of remote access in manufacturing, OMAC put together a workgroup in September 2020 to address the need for best practices. The group created a Practical Guide for Remote Access to Plant Equipment, which details many of the considerations that manufacturers will need to think about.

There are several options for connecting to equipment, including direct VPN, converged network, cell modem access, and others. A large majority of respondents to OMAC’s study use VPN along with other methods.

“One of the challenges is that the OEMs, if not given some level of guidance, will choose a way that they will do it on their own. And so it creates a little bit of challenge in that particular regard,” Fondl says, noting that this was a key goal for OMAC creating the remote access best practices. “Some end users, they may say what is the best, most secure solution that is agreeable to both the IT organization and the OT organization, and then try to do some level of standardization.”

Bosch Rexroth released a new control platform, ctrlX Automation, around the end of 2021. Along with that is CtrlX World, an ecosystem that allows third-party vendors to create apps for their technologies. One of those apps is a VPN, but there is a range of apps specialized in various aspects of connectivity, says David Boeldt, product manager, ctrlX Automation. “This could be for doing MQTT and other types of communication protocols, but it also opened the door for vendors that are specialized in security,” he says.

This makes it much easier, for example, for an OEM to incorporate VPN capabilities into its machine. “The OEM doesn’t have to create anything; they just have to select the apps that are required for them,” Boeldt says.

“The OEM can create a VPN connection to the customer and be able to translate themself into the machine without physically having to be there,” Barrera adds. “That allows us to really see the code, look at the alarms, and then we’re able to do that in our computers. We can even ask, remotely, the maintenance guy to run the machine in a certain way so we can actually review what the code is doing or what the drives are doing.”

Connecting with a brand’s own expertise

The expertise might not always be coming from OEMs, but from subject matter experts within the CPG itself. Through its Movicon industrial software platform, Emerson has a customer implementing a pilot project with an expansion to 50 plants.

“They’re using augmented reality to share subject matter experts between plants for maintenance,” says Daniel Millar, business development manager of factory automation at Emerson.

Such visibility could enable an expert at another location to get a better view of what the technician on site is seeing. Though augmented reality is certainly not yet universal, Millar estimates that three of the 10 projects he’s working on now include augmented reality. “For two of them, it was helping maintenance maintain the equipment