Special Report
IT-OT Convergence in Focus Solutions bridging the gap between Information and Operations Technology for smart manufacturing.
OPC DAY
INTERNATIONAL
IT meets Automation
APR 25 – 29, 2022 Registration: https://opcfoundation.org/opcday
DIGITAL EVENT
ONE HARMONIZED SOLUTION FOR PROCESS & FACTORY SCALING FROM FIELD TO CLOUD Criteria to get listed Feature available / on roadmap
OPC UA FOR CLOUD
Efficiency/Process Optimization Condition Monitoring Predictive Maintenance
CLOUD
Semantics
©klagyivik / stock.adobe.com
Digitalization
PROCESS AUTOMATION
5G
Security
IIoT
OPC UA over MQTT
FIELD APL
SPE
Safety
TSN
Motion
FACTORY AUTOMATION
www.opcfoundation.org
IT-OT Convergence
IT-OT Convergence Special Report Industry experts weigh in and provide their perspective on the state of IT-OT Convergence in the modern manufacturing plant. This special progress report takes a look at cybersecurity, enterprise data connectivity and how common network diagnostic and management tools can increase IT-OT effectiveness.
Visionary operations leaders recognize that reams of operational data used to support real-time decision making could create additional value for companies. INFORMATION TECHNOLOGY AND OPERATIONAL technology departments within industrial manufacturing plants have traditionally functioned independently. The goal of OT was to keep the plant running smoothly, and to support IT-managed business applications within the enterprise. But the world of manufacturing is changing. And to keep up, IT/OT relationships and deployment of technology must change as well. In this special report, IEB reached out to industry experts to get their insights into the latest insights into IT-OT convergence, cybersecurity concerns, and all of the technical megatrends that are providing new management and diagnotic tools for helping to meet this challenge. Together, IT and operations teams must go beyond responding to problems. Instead, they are key players in their companies’ transformations, helping to seize new business opportunities that make companies more competitive, more efficient, and more secure.
2
Focus on Security
Forging stronger partnerships. According to Barry Turner, Technical Business Development Manager at Red Lion, cybersecurity is a primary issue for strengthening the IT-OT, and the ability to implement a defense-in-depth strategy. Information Technology (IT) and Operations Technology (OT) personnel within a manufacturing environment often have different priorities, but generally IT departments have experience in this specific area. Education and communication enables them to work together more effectively, and especially to build the future the industrial networks that 5G and Web3 will require.”
IT-OT solutions
“Traditionally the IT and the OT have misaligned priorities when it comes to security. The OT is responsible for making it work as quickly and efficiently as possible,
which in many cases is in direct conflict with the IT department's highest responsibility, security,” Turner told IEB recently. “For that reason, it will be important that OT understand and implement key concepts the IT department has been using for quite some time. For example, defense in depth is a concept that increases the security of a network or application by adding multiple layers of protection from attacks. These layers often come in the form of VLANs, firewalls, and strict user access control.”
Technology benefits
Turner stated that increasing the network and application security by implementing a defense-in-depth strategy will help align both IT and OT and create a more robust application in general. A great way to get started is to build zones and conduits using VLANs, routers, and ideally, monitored access control. So, in terms of hardware, we are talking about routers, layer 2 Ethernet & layer 3 Ethernet
in d u s t r ial et h er ne t b o o k
02.2022
SOURCE: ROCKWELL
IT-OT Convergence
There is an acute need for providing enterprise connectivity between systems that create data, and the users that distill data into relevant information. switches. Using the Ethernet switches to create VLANs to make smaller networks or zones. Then using routers and firewalls to build a path in and out of those zones. It is also important to be able to have access control and some type of monitoring and alerting. “Many times, access control can be accomplished by using access control lists which would allow access based on IP or MAC address of a network device. Lastly, the network needs to log and alert when unusual activity is detected,” Turner said. He added that Syslog is one of many options here. It's a technology that the IT department is already using and many industrial controls products support including Red Lion's FlexEdge controller or NT24k Ethernet switch. Adding a layer that can detect and alert when bad things happen will ensure a controls engineer has as much time as possible to take action and hopefully mitigate costly downtime.
Convergence challenges
“Getting this very critical piece right is about education and communication. The IT department has been in this cyber security battle for a few more years than the OT. It's a good idea to learn from that experience and build it. The IT department typically does not have unmanaged switches in their network as they do not provide features needed for a defense in depth strategy,” Turner said. “Likewise, controls engineers would be 04.202 1
better served to replace unmanaged switches with managed ones that would offer features like VLANs, Access Control Lists, and Syslog. Having managed switches in the network will enable controls engineers to configure their networks in alignment with the IT department. Ensuring they are able to work together, and build the future-state industrial networks that 5G and Web3 will require.”
Enterprise data connectivity
Software at the edge can break down barriers. Arvind Rao, global business leader, Digital Solutions for Rockwell Automation emphasized the need for providing enterprise connectivity between systems that create data, and the users that distill data into relevant information. They key is software for taking operational data, and developing methods to take advantage of it at the enterprise level to drive decision-making.
IT-OT solutions
“Industrial data from plants and equipment is generated in huge volumes, every millisecond, which makes it difficult to capture accurately. When that data is captured, it is often stuck in siloes that make it hard to bring different data sources together,” Rao told IEB recently. “Even when you do accurately capture data and bring it together, a lot of work remains. You need to contextualize data from disparate
i n d u str i a l e th e r n e t b o o k
sources, align timestamps, and prepare data models.” Rao said that new software at the edge can help break down these barriers by providing connectivity between the systems that create data and the consumers of that data. Once the data can flow easily, data scientists can create actionable industrial performance insights. “This is the future of IT/OT convergence – connecting data as it’s created to the analytics engines, quality systems, traceability records, and optimization programs that drive industrial efficiency,” he added. Software that collects, organizes and contextualizes OT data and makes it available to higher-level IT applications and databases, such as FactoryTalk Edge Gateway software from Rockwell Automation, can unlock actionable plant and enterprise-level insights to accelerate IIoT digital transformation. It provides the right foundation to drive edgeto-cloud IT/OT convergence at the enterprise level, so everyone from the shop floor to the top floor can use data to make smarter decisions.
Technology benefits
Every industrial facility has numerous data sources and, many times, these data sources use different communication protocols from different vendors. To get data from the control system to the edge, multiple ingress sources are needed. This can be addressed in many
3
SOURCE: OPTO 22
IT-OT Convergence
ways, such as native support for the most common industrial communication protocols like EtherNet/IP and OPC-DA, or with multiprotocol solutions like Kepware. The ingested data will be mapped to data models. More advanced solutions can set up the ingress pathways and create a data model from a controller project file at design time, reducing the need to create data models during commissioning. Once the data is ingested, it can be processed at the edge or sent to other egress destinations. The more formats and destinations that are supported, the better. Some common destinations include specific applications, like Azure IoT Hub from Microsoft and ThingWorx from PTC and may also include generic protocols to support any destination, like MQTT. Your priority should be to use one solution that can send data to an on-premises data center and/or the cloud.
Integration challenges
“Using the controller project file allows a more literal translation of the industrial control layer to the data layer, helping OT engineers and IT analysts speak the same language. Instead of ingesting raw, unstructured and uncoordinated data from the control system, using a data model defined in the controller can more accurately and more quickly contextualize information,” Rao said. “That means less time spent preparing or grooming data, and more time on generating meaningful results.” The FactoryTalk Edge Gateway software leverages a unique smart object capability. This capability is built directly into the industrial control layer and transforms raw data from tags into a logical data structure that is completely configurable by the OT engineers based on the needs of IT analysts and data scientists. This allows you to automatically build in OT data context at the source, at high speeds of sampling, and maintain that context throughout the IT layer. This increases the data’s readiness for ingestion into the next level of analytics, removing up to 70% of the effort from data engineering.
Convergence challenges
“Industrial organizations have a lot of operational data, but they don’t know how to take advantage of it at the enterprise level to compete better, such as by improving their organization’s resiliency, sustainability and supply chain agility,” Rao said. “While digital transformation initiatives are on the rise in manufacturing, CIOs and CTOs are struggling to make OT and IT worlds play well with each other and tap the hidden insights from the vast industrial data sources.” The key reason for these roadblocks is the lack of proper correlating context, such as data
4
Opto 22’s custom built Linux OS for the groov EPIC includes only the packages required for EPIC’s functionality. This makes it easier to keep installations secure, stable, and up to date. source, type or timestamp, while capturing high-speed OT data at the edge. Without this critical OT context, captured industrial data has low data integrity, which translates into significantly higher data preparation efforts while building analytical models. To extract actionable insights from this OT data, it needs to be packaged in an interchangeable and flexible data format that can be easily shared between OT and IT applications. “As it becomes easier to connect the data across the enterprise, the value of IT-OT convergence becomes greater. Smart connectivity at the edge creates that ease,” Rao concluded.
Common tools & procedures IT-compliant standards & toolchain.
An important aspect of IT-OT Convergence is not only working more effectively together, but also adopting a common set of standards and tools, especially to deal with cybersecurity but also to manage the network infrastructure as well. Examples would be use of industry standard open protocols, and convergence on a toolset and standards that work for both IT and OT.
IT-OT solutions
The Eclipse Foundation, in its 2021 IoT & Edge Developer Survey Report, noted that Linux is the most popular operating system for constrained devices, edge servers, and the cloud. If this kind of adoption were mirrored by OT hardware vendors, many options would become available to foster convergence with IT systems. Key among these are networking and cybersecurity technologies, which over time have created sharp cultural divides between OT and IT organizations due to a lack of interoperability. Instead of pushing proprietary protocols, OT should embed IT-compliant standards like DNS/DHCP, network firewalls, and SSL/TLS encryption, so that all devices can be managed using the same tools and procedures.
Technology benefits
Linux is an open source operating system with a significant presence throughout IT and communications systems, including much of the internet; a supportive professional and enthusiast community; and a broad range of libraries for every kind of application. These factors represent a significant difference
in d u s t r ial et h er ne t b o o k
02.2022
SOURCE: OPTO 22
IT-OT Convergence
from the proprietary, closed source operating systems that power many automation devices from the Industry 3.0 generation. “An IT-compatible toolchain in OT eliminates questions about how to integrate the two organizations, provides greater assurance that they can operate with the same integrity levels, and moves us toward a shared vocabulary and culture about networking and security,” said Josh Eastburn, Director of Technical Marketing, Opto 22.
IT-OT integration challenges
Eastburn said that Linux was designed as a modular, open-source implementation of the UNIX operating system. It has a small kernel of essential functions that allow software to communicate with underlying system hardware and a broad library of modules (called packages) supplied by other open source contributors, including enterprises and non-profit foundations, which manage much of the source code. As such, Linux has been extended, modified, and re-released many times for different purposes and powers everything from small consumer electronics, to websites, to the servers that manage the internet, and even spacecraft. Different flavors of Linux are referred to as distributions. Opto 22 uses tools provided by the Yocto Project to create a minimal, secure OS for our products based on the popular Debian Linux distribution.
to optimize industrial operations, and manage the effective transfer of valuable data from the plant to MES, ERP, and cloud systems
Convergence challenges
IT-OT solutions
EtherNet/IP has played a critical role in IT-OT convergence by providing industrial operations the information needed by management to continue to drive growth and constrain
Standard Ethernet Solutions Network diagnostic and management tools.
Dr. Al Beydoun, President and Executive Director at ODVA said that IT-OT convergence can be enhanced not only by mutual understanding of key technologies, but also a common set of diagnostic tools and capabilities. Consistent content in a common location, from different devices and different vendors, can be shared using IT tools to create greater IT-OT visibility. Ultimately, a broad set of skills are needed 04.202 1
Standard Network Diagnostic Assembly Concept.
i n d u str i a l e th e r n e t b o o k
SOURCE: ODVA
“Since questions about IT-OT integration come up very often for Opto 22’s customers, it’s typical for us to meet with IT representatives to address concerns about cybersecurity and network architectures,” Eastburn noted. “Our decision to build our edge devices on Linux makes this conversation very straightforward. Instead of explaining why IT needs to adapt to special OT protocols and procedures, we simply demonstrate how our devices can be managed in the same way that the rest of the network is managed, including things like centralized user access rights (LDAP) and automatic addressing (DHCP). This puts IT at ease and allows our customers’ projects to move forward.”
A Linux repository is where code packages are stored ,so developers can add them to applications. Opto 22 signs all code in with a digital cryptographic key indicating it has been approved for use. costs in ever more challenging business environments. EtherNet/IP uses standard IEEE 802.3 Ethernet technology and Internet Protocol suite standards to solve critical industrial automation applications. “Reliance on standard Ethernet technology reduces the differences between the plant floor OT network and the broader enterprise IT network making it easier to transport critical information wherever it needs to go,” Beydoun said. “Furthermore, the object-oriented design of EtherNet/IP via the underlying Common Industrial Protocol or CIP allows for the services and device profiles needed for real time control in an interoperable environment. The object-oriented nature also makes it easier for those with an IT background to work with and to understand EtherNet/IP.” CIP Security is also a key component of IT-OT convergence as a last layer of defense for EtherNet/IP devices to protect critical infrastructure investments and worker safety. Approaches like permit and deny listing via firewalls along with detection tactics like deep packet inspection are vital components of network security that can be enlisted in the switches that connect OT and IT networks. However, IT centric approaches alone aren’t enough with bad actors constantly searching for weaknesses in OT networks, developing new malware, and finding more and more success with ransomware attacks. A defense in depth approach that includes the EtherNet/IP device level must be a part of a comprehensive security plan.
5
EtherNet/IP Confidentiality Profile
Device Authentication
4
Trust Domain
Broad – group of devices
Device Identity
CIP User Authentication Profile
Resource-Constrained CIP Security Profile 4
Narrow – Users/Roles
Broad; option to be Narrow via Gateway or Proxy
4 (Identity of User)
4 (Via PSK)
Device Integrity
4
4
Data Confidentiality
4
4
User Authentication
4
Via Gateway or Proxy
Fixed
Via Gateway or Proxy
Change Detection (Audit) Policy Enforcement (Authorization) CIP Security Profiles.
Technology benefits
EtherNet/IP has added a Standard Network Diagnostic Assembly to enhance the diagnostic capabilities of EtherNet/IP devices. The Standard Network Diagnostic Assembly creates a scaled architecture concept where network and device diagnostic data that was once spread out in many different locations is now combined together to be more easily accessible. This approach reduces messaging and user programming required to access data. This provides for consistent content in a common location from different devices and different vendors, which can be exposed via IT tools to create greater IT-OT visibility.
Standard Network Diagnostic Assembly Concept
Beydoun said that, with CIP Security, a selfdefending device can reject data that has been altered (integrity), reject messages sent by untrusted people or untrusted devices (authenticity), and reject messages that request actions that are not allowed (authorization). CIP Security for EtherNet/ IP devices makes use of the IETF-standard Transportation Layer Security (RFC 5246) and Datagram Transport Layer Security (RFC 6347) protocols in order to provide a secure transport for EtherNet/IP traffic. TLS is used for the TCP-based communications such as diagnostics and commissioning, and DTLS for the UDP-based transport communications such as I/O data. User Authentication is provided via the OpenID Connect standard, with Role-Based Access
6
Control (RBAC) being enforced with the CIP Security capable devices.
IT-OT integration challenges
The EtherNet/IP Standard Network Diagnostic Assembly combines ten different attributes of three different object classes into one location. One message is required instead of 10 messages per device in order to retrieve all of the information. Making it easier for diagnostic and prognostic analysis tools/systems to utilize data that is already available in EtherNet/IP devices is one way to realize the benefits of IT-OT integration. The reduction in messages minimizes traffic and therefore required bandwidth, which is important for existing OT networks that are seeing the addition of new devices and traffic. “Reduction in user programming, application development complexity, and changes to devices on the network allows for more valuable data to be transported from devices to the cloud for analysis and action to improve the production process,” Beydoun said. “The Standard Network Diagnostic Assembly is the first profile-independent definition of assembly instances in CIP that have data applicable to any device providing a higher level of data standardization.” From a security perspective, CIP Security provides the last mile of security-related requirements and capabilities for EtherNet/ IP devices. With the reality that not every EtherNet/IP device requires the same type of security or can support the associated resource
requirements, CIP Security defines the notion of a Security Profile. A Security Profile is a set of well-defined capabilities to facilitate device interoperability and end-user selection of devices with the appropriate security capability.
Convergence challenges
Beydoun added that “IT-OT convergence is a requirement to stay competitive in a future where cost pressures and resource constraints will continue grow. Data must securely traverse from the plant floor to the edge and the cloud in order to continuously optimize the business.” He said that the challenges of differing priorities regarding data confidentiality versus network uptime have become a moot point as converging networks have proven that data must remain confidential for security purposes and the production line has to keep going for cost reasons. The merging of expertise ranging from well experienced controls engineers with mechanical and electrical engineering backgrounds to workers graduating today with software training is a requirement for IT-OT convergence. “Ultimately, all of these skillsets are needed to optimize industrial operations and to transfer valuable data from the plant to MES, ERP, and cloud systems,” he added. “EtherNet/ IP and CIP Security are key underpinnings to the connection of lower level to upper-level networks to stay globally competitive and to ensure the preservation of both workers and assets.”
in d u s t r ial et h er ne t b o o k
02.2022
SOURCE: ODVA
IT-OT Convergence
Security Properties
IT-OT Convergence
Barilla implements digital connectivity at sauce plant SOURCE: SIEMENS
Digital connectivity at the Barilla plant provided key solutions including elimination of times of inactivity, real-time analyses of huge amounts of data, more effective way of working thanks to remote control and teleservice, repaid investment and maximum cybersecurity and data protection standards.
Barilla’s production gains advantages from a new OT network. BARILLA, AN ITALIAN COMPANY FOUNDED in 1877 and based in Parma, is the world leader in the food and pasta goods sector and produces pasta, sauces, and baked products at various production sites both in Italy and abroad. Its newest plant is in Rubbiano (PR) and is the only one used to produce sauces, dressings, and pestos, which includes its flagship product: Pesto alla Genovese with fresh basil and Parmigiano Reggiano cheese. The ingredients, recipes, technology, and “know-how” are 100% Italian, as well as the meat, which is sourced entirely from animals reared in Italy, with full traceability. Built in 2012, the plant was subject to expansion in 2018, which saw the introduction of an additional two lines, doubling production capacity to 120,000 tons of sauce per year. The plant comprises three macro 04.202 1
departments, one for cooking, one for production, and one for packaging. In cooking, the different ingredients that go into the sauces are blended together and then cooked inside cookers. The prepared sauces are then sent to the packaging department.
Manufacturing challenges
The challenge for the project was clear: automation and state-of-the-art remote control without any downtime. The expansion of the plant goes along with an ever greater proliferation of networked field devices whose task it is to exchange data with SCADA systems and plant management software. It was therefore necessary to develop a robust and organized industrial communication network to ensure the necessary scalability and flexibility to manage
i n d u str i a l e th e r n e t b o o k
the new technology currently on the market in order to be able to archive, process, and analyze the huge amounts of field data (Big Data) in real time and thus allow companies to generate values using this data. The introduction of new cybersecurity standards in an industrial setting, such as the IEC 62443 standard, also prompted the client to revise its industrial network standards. The plant also required a remote control and teleservice system for the devices that can guarantee both simultaneous technical interventions by suppliers and security, in terms of access control and monitoring. The aim was to transform an uncontrolled access system consisting of a flat network of communication into a network that, by the end of the project, is segmented and structured using a state-of-the-art access method.
7
SOURCE: SIEMENS
IT-OT Convergence New systems created more effective way of working thanks to remote control and teleservice. technology) and OT (operational technology) talking, as before this project the whole automation network in the factory was almost entirely the domain of IT Barilla. The aim of this project was to separate the IT world from the automation world whilst at the same time making them interface in a functional way.” In fact, Siemens recommended to Barilla the most innovative OT networking and cybersecurity technologies currently on the market, ensuring complete compatibility of
New network safeguards production continuity
The whole project was carried out by three parties: Siemens, Barilla, and ITCore. ITCore, as solution partner of Siemens for the “Industrial Strength Networks”, has taken a central role in the planning, realization, and maintenance of the whole network infrastructure and the teleservice systems installed at the Rubbiano plant. SOURCE: SIEMENS
Another challenge was to avoid downtime situations and hence prevent disruption of production, as Andrea Di Nicola, Automation Manager at the Barilla plant in Rubbiano, explained: “A key issue for this project has been the implementation of Siemens’ teleservice infrastructure, based on the Sinema Remote Connect solution combined with the Scalance S firewall. The most difficult part was converting the network architecture while the systems were in operation or in the few times production could be shut down. Thanks to the expertise of Siemens and their competent partner ITCore, we managed to achieve this with zero downtime, thanks to the careful management of the operations and network configurations, which made the transparent and optimally controlled transition phase possible. Both companies helped us throughout all the phases of the project, from the start right up to the final training part, the plant’s technical area and maintenance, thus creating a harmonious infrastructure.”
Siemens’ devices with regard to systems and components from other automation suppliers. The strength has been the components and competence offered by Siemens, and the possibility of easily managing communications from other vendors, guaranteeing adequate continuity of production. Marcello Scalfi, Sales Specialist Team Leader Digital Connectivity and Power at Siemens Italia, noted: “Integration work between the information technology and operation technology teams, carried out by Siemens, required a long internal evaluation phase together with Barilla, who considered various solutions presented by us via training workshops targeted at Barilla colleagues in the engineering and OT departments for the purpose of increasing their networking and cybersecurity skills. Knowing the technology available has also enabled them to acquire the right skills to be able to assess the best solution to adopt. For Siemens, it was essential to open up the discussion to the possible scenarios so the client could make a conscious choice. It has been an important transition, a training ground for everyone involved.”
Innovative OT network meets high cybersecurity standards The partnership with Siemens comes along in a natural way; the collaboration between Barilla and Siemens has been producing winning results for years. This is where the strong motivation was born to continue in this way and prefer Siemens over other automation companies. “Our trump card has been the full participation and collaboration in the pre-analysis phase and technical and economic assessment of the investment with Siemens, right from the start,” continued Di Nicola. “We tackled this project together. The main idea was to get the worlds of IT (information
8
Scalance XM416-4C: the core of the production backbone that provides high availability and redundancy to the plant network. in d u s t r ial et h er ne t b o o k
02.2022
SOURCE: SIEMENS
IT-OT Convergence
Scalance SC632: the array of industrial firewall enables the right communication path from plant network to IT and vice versa. The new network, structured with rules and routing plans, now has more than 1,000 interconnected intelligent nodes. The process envisaged the creation of a fiber optic ring backbone network (also called “backbone”) managed with MRP (media redundancy protocol) and the segmentation of the automation network in VLAN (Virtual LAN) split into production cells. Each of these cells is segregated from the others in terms of network and communicates using the following devices from the Scalance product family at Siemens: Industrial firewalls: a system for network security that allows monitoring of incoming and outgoing traffic using a predefined series of security regulations to allow or block events. Industrial switches: network devices responsible for switching at datalink level, which is used to manage the data traffic when there are more connected nodes, separating the so-called collision domains connected to its ports. Routers: electronic devices that route data between the networks, distributing the connection between different terminals. To complete the solution, a centralized management and control system for remote access to the production network (teleservice) 02.202 2
has been integrated according to a defined double jump host model. The implementation of the solution was made possible with the functionality provided by the SCALANCE S equipment from Siemens (industrial firewall) used upstream of the cells and the use of the SINEMA Remote Connect software. Networking and cybersecurity in the industrial sector are themes that ITCore deals with on a daily basis together with Siemens. “We believe that in the coming years constant monitoring of all devices connected in the network will be essential because, while we cannot think of stopping the digitalization process, we must do everything possible to safeguard production continuity, both in terms of functional operation and protecting the data generated,” stated Federico Tarzia, Chief Technical Officer of ITCore.
Results above expectations
The solution put in place by Siemens has made it possible to achieve higher cybersecurity and data protection standards at a factory networking level feasible to date. This makes the Rubbiano plant a real role model for the brand's other plants. As well as easy maintenance and thus the plug & play replacement of components, and
i n d u str i a l e th e r n e t b o o k
consequently the reduction/elimination of downtime due to malfunctions, another point that has been gaining ever greater value over time and which has made it possible to repay Barilla’s investment in full has been the support. “With the spread, on the one hand, of the pandemic and the necessity to guarantee plant production on the other, investing in a reliable teleservice infrastructure has allowed us to “be in the factory” even if off-site, when smart working,” commented Di Nicola. “At the same time, equipment suppliers, who were not always able to go to the factory to make any changes or interventions due to the pandemic, could use teleservice to be in environments that are segregated from each other, guaranteeing information confidentiality between one supplier and another.” The timely control of this access and the possibility to smoothly control and manage external supplier operations has been another successful aspect of this project. Marcello Scalfi, Product Management Team Leader, and Giorgio Santandrea, Account Manager, Siemens. Visit Website
9
IT-OT Convergence
Software technology solution bridges gap between IT & OT New exOS software offers machine builders a solution to the divide traditionally separating IT and OT. Using exOS gives Linux software developers an option to develop, compile and debug their code using their preferred environment, in coordination with B&R's Automation Runtime real-time operating system.
Flexible exOS technology offers a crossover solution that bridges the divide between IT and OT.
Exploring new horizons
The challenge is to overcome the obstacles that stand in the way of harmonious, effective IT/OT collaboration. Traditionally, the two domains have existed more or less as parallel universes. "But with the rise of the Industrial IoT, the time has come to change that," said Stefan Bina, B&R's product manager for SOURCE: B&R
HOW DO COMPANIES BRING IT AND OT together? It's a question that must be answered by any company hoping to realize the visions of Industry 4.0. It’s a puzzle, so far, without a real solution. But new technology is now bridging the IT/OT divide, and offering machine builders access to the world's largest software ecosystem. To pull off today's more advanced manufacturing solutions, machine builders need experts from the fields of IT and OT collaborating on interdisciplinary teams. That's the key to leveraging the full potential of the Industrial IoT. But it also becomes an arena of conflict between two different worlds. While OT experts are well-versed in machine design, PLC programming and commissioning, their IT colleagues are more at home using high-level programming languages like C++, Python or JavaScript and working with open-source software and Linux. Common ground can prove hard to find. A team is only ever as good as its members. But picking the best and brightest from individual fields and gathering them around a table will only get you so far. You also need to give them the tools and opportunities to apply their craft toward a common solution. And all too often, there lies the rub.
The exOS technology gives control systems the ability to access and understand any Linux software.
10
Industrial IoT network solutions. "It's time to erase the lines between IT and OT." Among the biggest obstacles is a lack of support in today's control systems for the full bandwidth of high-level programming languages. As a prerequisite for true IT/OT connectivity, control systems must therefore be given the ability to access and understand Linux software. This is exactly where new technology from B&R comes in – with its new enhanced crossover Operating System, or exOS for short. "With exOS, we bridge the divide that has traditionally separated IT and OT, and finally give machine builders a connection between the two worlds," said Bina. "In doing so, we give them access to the world's largest software ecosystem." exOS gives every Linux software developer the freedom to develop, compile and debug their code using their preferred environment. Then, the Linux application can be easily integrated into the control system with the help of exOS and used hand-in-hand with B&R's real-time operating system, Automation Runtime. "This offers the huge advantage that developers can work in their familiar environment, like Eclipse or Visual Studio, where they can make the best use of their knowledge and experience," explained Bina.
in d u s t r ial et h er ne t b o o k
04.2021
SOURCE: B&R
Developers simply import their completed Linux application into the B&R system as an exOS package. From that point on, they can use Automation Studio as a central engineering tool. Managing and debugging software solutions becomes simple and straightforward. No other tools are required. "What that means is that the developer creating the Linux software and the engineer commissioning the machine can each work in the environment they're used to," noted Bina, "because they have the exOS crossover solution bringing it all together seamlessly."
Efficient maintenance
Integrated functions simplify hybrid solutions combining machine control and Linux applications. remote access.
user-friendly API for high-performance data exchange. The API ensures easy, buffered process data communication between the controller operating system and Linux, consistently in the millisecond range.
Futureproof solution
B&R's crossover solution offers numerous integrated functions that make it easier to implement hybrid solutions combining machine control and Linux applications. The integrated functions are executed symmetrically in Automation Runtime and Linux. "exOS uses technology and tools from the B&R system and integrates the Linux application into the B&R engineering environment. All the freedom of Linux remains," said Bina. Features include uniform project management for Linux and control applications in Automation Studio as welas automatic transfer of Linux applications from Automation Studio to the controller and the Linux environment. exOS also includes a
Scalable for any application
SOURCE: B&R
In addition to streamlined development and commissioning, another hallmark of an intelligent machine is that any required maintenance can be performed quickly and easily. In the event of a malfunction, service technicians need to be able to locate the problem and contact the machine builder as quickly as possible. exOS also offers an array of diagnostics functions for installation and code execution. The B&R system provides the user with a complete log of all error messages from both the Linux application and the controller operating system. Hardware components are also easy to replace. The machine is back up and running again in minutes without any reprogramming. "exOS makes maintenance much more efficient and significantly reduces downtime," says Bina. Not only is it easy to replace hardware when necessary, any updates to the machine software can also be installed, for example via
IT-OT Convergence
One central engineering tool
Machine builders can choose between two different types of exOS implementation. "One option is to run Automation Runtime and Linux on the same device, such as an Automation PC," explained Bina. This is useful if the Linux application is to be executed hand-in-hand with cyclic control applications, as is the case with automated guided vehicles (AGVs). Such systems are often based on the open-source robotics platform ROS. In ROS, dynamic path planning is executed in Linux, while motion control tasks are handled in Automation Runtime. With exOS, the two can be optimally synchronized in a hypervisor configuration and installed on one Automation PC. Another way to implement an exOS automation solution is to combine an X20 PLC and an Automation PC. In this setup, the control system and Linux system are connected via an Ethernet interface. Having the two systems running on separate hardware maximizes utilization of the available processing capacity.
IT and OT hand-in-hand
As a flexible crossover solution, exOS breaks down longstanding barriers between IT and OT to create common ground for effective collaboration. "With this move, B&R opens up the world's largest software ecosystem – where the best of both worlds can contribute to convergent, adaptive machine concepts," says Bina. OEMs can fully leverage the know-how of their developers and engineers in both domains and implement innovative machine concepts for smart factories. Machine builders can choose different types of exOS implementation: Automation Runtime and Linux. 02.202 2
i n d u str i a l e th e r n e t b o o k
Carola Schwankner, editor, B&R.
11
IT-OT Convergence
Edge computing for smart factories and smarter machines
SOURCE: STRATUS
Edge computing platforms offer new revenue and cost reduction opportunities for machine and equipment manufacturers. They also offer OEM machinery builders a technology pathway to developing new generations of smarter equipment that meet changing customer demands and the need for digital transformation.
From this topology, it is easy to see that the Device Edge through the Compute Edge are applicable to machine and equipment builders. DIFFERENT TYPES OF OEM MACHINERY builders face one challenge that is always the same; the market is highly competitive. Each generation of equipment offers price/ performance improvements over previous generations. Design, engineering and product management departments must constantly evaluate the tradeoffs between new technology improvements and associated cost
increases as they bring new designs to market. Automation has been an integral part of equipment design for many years, but new concepts such as digital transformation, Smart Manufacturing, Industry 4.0 and the Industrial Internet of Things (IIoT) are being increasingly applied in the manufacturing world to deliver competitive advantages. This article focuses on what machine and equipment manufacturers (OEMs) can do to
Machine builders can choose different types of exOS implementation: Automation Runtime and Linux.
12
leverage these new concepts and how Edge Computing can meet their unique needs as well as their customers’ requirements.
The need for smart machines
In the same way that machine automation, using PLCs and PACs, revolutionized equipment design a generation ago, Smart Factories and Smart Manufacturing are evolving the industrial world again. Applying digital transformation concepts enables companies to operate at peak performance by leveraging the data collected by control systems and sensors. This approach provides real-time analytics to operators, inputs that help improve Overall Equipment Effectiveness (OEE), and data that can be collected and analyzed locally. Data can also be sent to the Cloud for deeper analysis in a variety of applications. Applications can include optimized maintenance, supply chain agility, and increasing yield across multiple facilities engaged in similar production. Machines and OEM equipment are fundamental elements of production. Smart Factories and Smart Manufacturing can only be achieved by Smart Machines operating as designed by their OEM or specified by the end user. Embracing digital transformation enables machine and equipment builders to enhance design and supports the development of new services around equipment performance and
in d u s t r ial et h er ne t b o o k
04.2021
SOURCE: STRATUS
IT-OT Convergence
maintenance. An important question is what is the best way to develop new generations of smarter equipment, or enable existing designs and deployments to meet the requirements that customers are demanding? Edge Computing is an approach that can solve many of the key design requirements.
What is edge computing?
Edge Computing comes in many forms and can be broadly defined as any computing that takes place outside the data center. With such a broad definition, it is not surprising that there are many confusing definitions that span both the IT industry and the automation and controls industry that we refer to as Operational Technology (OT). Gartner, a major analyst firm, has published a simple topology that enables everyone to understand what type of Edge computing might be useful to them. This topology image demonstrates an infrastructure technology stack that shows the types of servers, devices, or platforms that can be characterized as Edge Computing and where they reside from the physical endpoint “at the edge” of the data source. It alludes to the types of computing power, collection, analysis, and data movement that is available outside the actual data center. Edge provides extremely important benefits for equipment manufacturers and their customers. From this topology, it is easy to see that the Device Edge through the Compute Edge are applicable to machine and equipment builders. In practice, all machines and equipment today have aspects of the Device Edge with PLCs, and some have Gateway Edge devices. In some respects, Industrial and panel PCs used by machines or equipment fall into this Gateway category. They may be the first “point of contact” for a device edge PLC. Yet IPCs have limited computing and analysis power or suffer from common reliability and usability issues that are similar to a typical PC or server that is not manufactured for harsh environments. So, other types of purpose-built Edge compute platforms are required to meet customer needs. When it comes to the modernization of machine control and automation systems, it is the Compute Edge that should be the focus for equipment builders. Edge Computing delivers all the characteristics and capabilities necessary for machine and equipment manufacturers to enhance their current designs and transition to the delivery of even smarter equipment.
Edge computing technology
Edge Computing is a scalable modular technology that supports the development of standardized, modular software components and applications to increase efficiency. 04.202 1
Ruggedization is important along with eliminating downtime, and offering remote monitoring. It can help enable equipment manufacturers’ existing applications—such as monitoring and control software—to be consolidated onto a single platform while enabling other critical applications to run on the same platform. This makes it easier to develop smart, Industrial Internet of Things (IIoT)-enabled machines and equipment, and to easily add future applications that support customers’ evolving Industry 4.0 and Smart Manufacturing demands. When implemented, Edge Computing can facilitate a smooth evolution from the equipment of today to digitalized machines that are integrated into a customer’s operations. This includes multiple capabilities such as on-prem or Cloud connectivity to fully-integrated Smart Machines that embrace Digital Transformation without the need for complex retrofitting and redesign.
Edge computing for OEMs
While the Compute Edge category is quite broad, there are a number of key characteristics that narrow the types of functions that apply specifically to a machine or equipment manufacturer. Edge Computing platforms must be capable of running existing control and automation
i n d u str i a l e th e r n e t b o o k
applications in a modular way, without the need for significant re-architecting. Virtualization is a key technology. Essentially, each existing IPC can run as a “virtual IPC” or virtual machine on the Edge Computing platform. This enables consolidation of all existing IPCs onto a single platform. None of these virtual machines are aware of the other virtual machines so all applications continue to run independently.
Modifying existing controls
With a virtualized Edge Computing platform, it is now possible to add much bigger degrees of flexibility to machine and equipment operation and monitoring. For example, with the traditional IPC/HMI approach, each station on a machine or piece of equipment is limited to what that IPC/HMI can display so only a single operator can monitor complex equipment from a single location. This is not an ideal model. Using virtualization, equipment manufacturers can achieve increased flexibility for their customers by having multiple people remotely access data and applications at different stations. This autonomous model is preferred and provides customers more options.
13
IT-OT Convergence
capital purchase. To successfully implement and maintain such a model, the machine or equipment manufacturer must understand the performance and maintenance profile of equipment, and have the ability to accurately collect and analyze the appropriate data.
Ruggedization
For a machine and equipment manufacturer, this is perhaps obvious, but Edge Computing platforms are supplied by a variety of vendors – not all of which are aware of the equipment’s final operating environment. It is not just about ruggedization, but also about eliminating downtime, making maintenance and support as simple as possible, and offering remote monitoring. Similarly, Edge Computing platforms in the OT world must be prepared to operate autonomously for long periods of time as often there may be no connectivity to the outside world.
Addition of new analytics
With virtualization it can potentially be easier to add new applications simply by creating a new virtual machine on your Edge Computing platform. This paves the way to for companies to add new applications without impact to existing applications. This can include localized data collection, real-time analytics and data filtering before transport to external locations, such as a customer’s data center, or to the Cloud, perhaps your own.This can also depend on such things as data privacy laws and, of course, assumes you have a scalable Edge Computing platform as needs expand.
Support for OT/IT convergence
This is perhaps a topic not often considered by machine and equipment builders, but it is increasingly important in Digital Transformation initiatives. The control and automation applications that operate the machine are well understood by your design engineers and your customers who operate the machines. They are focused
on the Operational Technology (OT). The expertise to develop sophisticated analytic capabilities, perhaps involving machine learning (ML) and artificial intelligence (AI), plus the capability to securely transfer appropriate data to a data center or to the Cloud is often the domain of Information Technology (IT) experts. Edge Computing platforms can support both OT and IT requirements. It is important to ensure that a platform that supports multiple types of operations applications is simple to deploy and maintain, and easy to manage by both OT and IT professionals.
New revenue opportunities
Many of the capabilities and new applications that Edge Computing platforms support create the possibility of new revenue generating services, or the optimization of existing service and support capabilities. For example, a trend that is beginning to emerge is the concept of the “machine as a service” or “equipment as a service”, whereby the end customer pays based on time or usage instead of a traditional
Cost reductions
Ultimately, adopting Edge Computing only becomes viable if it meets the stringent cost constraints of the highly competitive markets in which equipment builders play. When evaluating Edge Computing platforms, it is important to consider capital costs as well as development and ongoing costs. In addition to the savings from things like virtualization, cost is a function of value. More expensive platforms provide a much higher set of value and benefits than simple devices or gateways with lower computing power. This approach will reduce other costs such as manual monitoring, downtime, staff resources, and application efficiency. Some of these costs can be less tangible, but as industries move to automation, Digital Transformation and related initiatives, such as Industry 4.0, Smart Manufacturing and IIoT, equipment manufacturers must consider future applications and changes in the customers’ environment.
Summary
The machine and equipment manufacturer market is entering an evolutionary period that will bring as much change as the introduction of the PLC a generation ago. Edge Computing is a critical technology that will enable and ease this transition. Edge Computing platforms will play in integral role in expanding the traditional control and automation capabilities of equipment. They will also provide the bridge for equipment manufacturers to embrace digital transformation initiatives for themselves and to integrate with initiatives of their customers. Technology article by Stratus. View White Paper
14
in d u s t r ial et h er ne t b o o k
04.2021
IT-OT Convergence
Essentials of IoT device data management SOURCE: ISTOCK
A key to IT-OT convergence is device data management solutions for effectively monitoring, processing and managing large amounts of data from IoT devices. Traditional embedded database solutions fall short in understanding and fulfilling the sophisticated data processing and management requirements of IoT devices.
Intelligent systems in the IoT age collect and analyze mass quantities of data, and findings should be accessible to edge devices and embedded systems. SMART SENSORS AND DEVICES ARE BECOMING an important part of the Internet of Things, IoT, and are continuously changing the way we automate tasks. We employ intelligent systems to improve production in factories, manage smart home energy to monitor and reduce energy costs, build industrial automation systems to replace human assignments, and develop autonomous transportation to improve driver safety. Inside these embedded systems are sensors which rapidly transmit data that must be immediately captured, processed, and acted on. But how are we able to capture, process and manage the flow of massive amount of data continuously coming to the system and empower devices to make decisions or take actions? Traditional embedded database solutions fall short in understanding and fulfilling the sophisticated data processing and management requirements of IoT devices. IoT edge database solutions designed to understand the continuous stream of data produced by sensors enable devices to make important decisions in microseconds. What are the available device data management 04.202 1
solutions for IoT devices to monitor, process, and manage data? In this article, we will review some embedded IoT device data management options available for edge devices and highlight the distinct design primitives’ approach for addressing device data processing.
IoT edge device data challenges
First, what is an embedded system? An embedded system is a device that performs tasks automatically through self-learning and self-management, which often connects to other systems. These systems are starting to use the IoT to improve lives. But as a significant volume of data accumulates on each connected device, a comprehensive approach to data management is required. Across these embedded systems, the primary challenge is to monitor, capture and process the data to intelligently ensure safe behavior and fault‐free operation of the devices. More than simply streaming data and receiving commands, these devices run complex, high level software programs that operate with or without a network and cloud connection.
i n d u str i a l e th e r n e t b o o k
Meanwhile, devices embedded in these systems must handle large transactions for various tasks and need to be able to connect to each other on multiple networks. Therefore, IoT data management needs to be divided into real-time interaction with objects, or things, as well as offline mass storage and long-term trend analysis. In the real world, device manufacturers seek a scalable edge data management solution to deploy hundreds of IoT devices, so each can collect, analyze, and manage the flood of data that IoT sensors produce, without losing performance. These devices do not need to permanently store all real time data but must capture and store critical information. Simultaneously, each IoT node must make independent decisions that trigger appropriate reactions. Database queries enable device applications to gain the intelligence to make informed decisions in real time: efficiently and without delay. Succeeding on the IoT requires both the right data management software and the capability to quickly collect and connect device data at the right throughput rate to achieve low latency.
15
SOURCE: ITTIA
IT-OT Convergence DB SQL data processing capabilities enable edge application to analyze data, perform continuous analysis, and configure data distribution. It is scalable to any number of edge devices, so edge applications can capture data flows from multiple sources, analyze data, and emit valuable findings as events.
Traditional vs. IoT data management
Case Study
volume of data typical of autonomous systems. But how can a device avoid concurrency race conditions and data corruption? Developing a custom solution to index information is not a trivial task and would essentially drive you to become your own database provider! Data management system software updates and new database features each introduce new challenges for application developers.
IoT data processing and management Common data management solutions currently available for devices do not fully address to the complexity of architecting software for IoT data processing on the edge. Sensors are the primary source of data, but they are constrained by their limitations from facilitating sophisticated analysis. The focus of data analysis and management on the Internet of Things is to harvest real time information and make sense of data in a very limited time, even without permanent storage available to save what is important to keep or reject what isn’t. A good solution seamlessly extends technologies familiar to many developers, such as SQL, to the new problem of analyzing IoT sensor on directly on edge devices.
SOURCE: ITTIA
Microchips and devices are now, more than ever, used to build autonomous systems that collect data and gain intelligence. When it comes to data management on these systems, a device application may connect directly to the cloud, buffer data on the device until connectivity is available, or manage data directly on the device. Possible candidates for local device data management include storing arrays and data streams in flat files, creating tables with data management software, or developing a custom data storage solution. Traditional storage-centric embedded databases generally manage the storing, retrieving, and updating IoT data with records, tables, and files. In the context of IoT, data management software must start by monitoring data in real time, while also providing storage and logging options for future analysis. Datacentric solutions expand the role of data management from simple data storage and analysis to include online data monitoring, continuous query processing, intelligent filtering, and automatic data distribution, in real time. For example, flat file formats such as CSV, JSON and XML are notoriously difficult to update and search, especially for the large
DB SQL advantages include device data processing and management for edge applications to filter and process data originating from a complex flow of data events.
16
Animal healthcare is an interesting sector that can employ and benefit from IoT systems. Devices now monitor feeding activities for cattle, horses, and other livestock to determine the most efficient milk yield, weight control, and best health care practices. In addition, a prediction algorithm can continuously collect sensor readings to monitor animals’ temperatures 7x24. These goals require processing time series data in real time. Animal healthcare systems, embedded with IoT devices, are expected to search for feeding instructions, capture temperature for different animals, and track feeding history. Data may be communicated with a central cloud data management location, but it also must be indexed on each animal and system individually. The instructions given by the RFID sensors on animal body (i.e. ears) issues immediate alert and communicates with other part of the system within. After obtaining the data by an RFID reader, the database must be robust enough to continuously monitor, catalog and retrieve data and prepare the system locally for feeding information/health, as well as temperature data management challenges, even when there is failure such as power loss. The main goal is to monitor the real food consumption of animals as well as abnormal temperature behavior to monitor and guarantee health and prosperity. High performance concurrent read and write is a common characteristic of such an IoT data management. For this scenario, animals need to be monitored in real time, communications must be handled with a monitoring module and obtained information must yield to immediate act or storage of the data locally on the device. In addition, as explained earlier, each animal is connected via RFID to a sensor, and
in d u s t r ial et h er ne t b o o k
04.2021
IoT data management framework
Process time series data in real time: Through years of engagement with customers, we recognized special needs for IoT edge data management. Therefore, we decided to implement a new generation of embedded database software with a special focus on IoT data streaming, processing, and management. We recognized that sensors need to collect and aggregate a massive amount of data, which must be queried in real time, and for which results must be shared with other embedded systems. We decided to handle this common scenario by implementing two engines: one for data management and the other for IoT stream processing. In our framework, we paid close attention to the benefits and tradeoffs between the cost of data processing and storage on the edge in comparison to fundamentally utilizing entire cloud data processing and management approach. High performance concurrent read/ write on flash media: Due to the nature of IoT systems interaction with real time data, which is generally fresh information that requires deterministic online analysis, as well as collection for future analysis requirements, both processing and management tiers made sense. Therefore, we designed our database to be an optimization engine to aggregate a large volume of data collected from the sensors, find its detailed value, and save important information while other garbage data is automatically discarded. This aggregation is aimed toward minimizing data storage and maintenance cost for embedded systems. We also added a data distribution layer that enables other parts of the system to receive information from both the data processing and data management engines. We also developed a union layer which is enables different sensors’ streams to join from various desired sources and direct them for further processing. Distribute embedded data across heterogeneous devices: We designed our 04.202 1
SOURCE: ITTIA
IT-OT Convergence
sensors are connected to an edge device to be remotely, on premises, for data analysis. In these embedded systems, low power consumption must work with various desired microcontrollers or applications processors, and the system must be programmed for data acquisition according to the integration period desired, and the data is sent wirelessly to the central device in each facility. Therefore, a database for distributed embedded data across heterogeneous devices is required. As a company often has different global sites, the number of edge devices that are connected to each network is practically unlimited, and each new site can add new device modules as required. Additionally, each device can be programmed to manage the data per each location’s particular needs. ITTIA DB SQL is a relational database management software library for embedded systems and intelligent IoT devices. SQL features not typically available on an embedded device greatly simplify data management for software developers. solution to be highly available, so when there is a failure due to any reason, a single crash cannot stop the system and real time data management operation can continue. Although archiving and backing up data is an option, that approach is not fully capable for mission critical systems. High availability makes the data record recoverable and provides peace of mind against general failures. Security is another important factor that we had to pay close attention to. It has been estimated that device application security concerns are expected to remain an important and frequent cause of confirmed breaches. With devices increasingly collecting, storing, and connecting critical data, the risk of a breach grows continuously. Meanwhile, manufacturers building embedded systems for various markets— including industrial automation, medical devices, power grid, or transportation— all largely face the same device security challenges, as they need to harness the incredible power of data management and connected computing. SQL injection is one of the most common database attacks. This involves the injection of malicious code into a device to execute malicious queries. Another vulnerability that accelerates injection attacks is when the database is not adequately isolated from the running code. Though isolation may reduce some of these attacks, it is better to look for alternatives. While the ITTIA DB SQL cockpit tool, know as Web Console, allows developers and end-users to monitor database activities, ITTIA DB Security Agent, DB SEAL, monitors those database activities automatically, isolates databases stored on devices, decides between alternatives to mitigate an attack, and keeps
i n d u str i a l e th e r n e t b o o k
the database contents always available. This proactive monitoring of the data and database by our agent will allow the device to issue an alert, block access, or shut down when data management metrics fall out of the expected range. This is a virtual agent that monitors database responsibilities and metrics in real time, and responds when there is an outage, or other security concern.
Conclusion
At every stage in building a device application, developers grapple with decisions around the best data management software for a successful development and launch of their edge centric IoT system. Such decisions as selection of database software consume significant development and validation time and cost. Intelligent systems in the IoT age collect and analyze mass quantities of data throughout the system. These valuable findings should become accessible to other edge devices and embedded systems. As the most common IoT data management use case, devices produce a large quantity of continuous raw data which must be collected, analyzed, and distributed. Devices require a data management framework that empowers local applications to process data, capture events and share important information to many other devices deployed on the IoT edge network. ITTIA DB SQL© is an ideal data management product for manufacturers developing Internet of Things solutions that must collect data from sensors nodes for real time and historical data management. Sasan Montaseri, Founder, ITTIA. Visit Website
17
| PC11-53E |
Maximum performance with minimum footprint
The C6030/C6032 ultra-compact Industrial PCs: with reduced space requirements for highly complex and demanding applications Intel® Core™-i processors of the highest performance class up to the ninth generation maximum computing power up to Intel® Core™ i7 with 8 cores at 2.6 GHz each almost any automation and visualization task can be realized the C6032 variant offers modular interface extension for increased flexibility
Ultra-compact
C6015
C6017
Ultra-powerful
C6025
C6027
Scan and get all the performance learn details
C6030
C6032