Industrial automation software management on AWS: End-to-end DevOps for factory automation coding to commissioning

The traditional approach to programming industrial automation systems, such as programmable logic controllers (PLCs) and robots, has been devoid of modern software development best practices. Meanwhile, as the industrial landscape evolves towards greater factory automation, the management of automation software becomes crucial. Implementing DevOps methodology on cloud platforms like Amazon Web Services (AWS) can significantly improve automation software development efficiencies and save lifecycle costs.

Unlike IT, the operational technology (OT) domain suffers from organizational as well as proprietary PLC vendor technology stack silos—for example, mechanical, electrical, pneumatic, and control engineering teams as well as proprietary PLC vendor technology stacks—necessitating the centralization and abstraction of automation development across the silos to bring full DevOps functionality to users. While the DevOps cycle in IT is defined through stages like code, build, test, release, deploy, operate, monitor, and implement feedback, the challenges in OT are weighted by the link between coding and their associated physical assets—such as automated machines on a factory floor—that are remote and often inaccessible in real time to the automation coders. AWS and its system of partners provide a centralized platform and DevOps tools to overcome these challenges.

Additionally, simple aspects like role-based access control for operations, especially across PLC vendor silos, have not been the norm in OT so far. The lack of an effective collaboration platform across PLC vendors, plus external machine builders and system integrators (SIs), increases the complexity of integration, restricting a manufacturer’s efficiency in automation development, commissioning, and lifecycle management.

Each branded PLC supplier offers an independent and rich integrated development environment (IDE) with its own added layer of programming methods on top of the IEC 61131/3 PLC coding standard. This complexity is multiplied by the different PLC programming styles, such as ladder logic, function blocks, and structured text. Moreover, the PLC code often resides within proprietary project files on individual laptops, creating inefficient and error-prone version management, code integration, and testing.

An industrial DevOps solution needs to break these barriers to traditional PLCs, coding, and commissioning. This article presents the application of end-to-end DevOps, from PLC code development to commissioning and beyond, based on a solution by Software Defined Automation (SDA), an AWS Partner. It delves into how DevOps, traditionally not synonymous with PLC or robot programming, can revolutionize these domains and how SDA’s solution built on AWS storage and compute services provides a reliable, scalable, and secure platform for automation engineers to collaborate remotely and increase their productivity. This blog post elucidates the advantages of cloud-based DevOps using a customer case, particularly focusing on agile project management aspects; collaboration tools for industrial automation SIs; and a platform for code backups, version management, and reusable automation code standards.

A related blog on the topic of AWS-based best practices for automation code operational excellence can be found here.

PLC code development with IDEaaS

PLC and robot programming is traditionally developed in siloed environments without modern software engineering best practices. Engineers write PLC code line by line in the PLC vendor–specific IDE in their computers and download the code to physical PLCs for testing. Typically, the PLCs are connected to the hardware equipment that they are controlling on the plant floor. This requires engineers to be on site to transfer the updated control logic to the PLCs whenever changes are made. While remote VPNs exist, factory IT often does not use them because the endpoints—the PCs of engineers from the machine builder or system integrator teams—are not protected and can potentially expose the OT network to cyber risks. SDA’s IDE-as-a-Service (IDEaaS) solution, which uses AWS, provides secure remote connectivity from the cloud to the PLC, letting users generate a digital twin of the PLC and run the IDEs using any web browser. This removes the vulnerability of unmanaged endpoints while facilitating collaboration across company boundaries. The PLC programs can be downloaded securely to the PLCs through SDA’s secure remote connectivity.

In real life, almost every factory has PLCs from multiple vendors across the fleet of production machines. Traditional automation engineers often need to install non-value add tasks as well as maintain and update multiple versions of vendor-specific IDEs on their local machines to develop code or maintain automation projects. SDA’s IDEaaS manages vendor-specific versions, such as Siemens TIA Portal and Rockwell Studio 5000, in the cloud, freeing up the engineers to focus on their core task of developing, integrating, testing, and commissioning automation code. As an additional benefit, the IDEs start in seconds instead of minutes, saving automation engineers valuable time each workday. In addition, SDA offers the following features that complement their IDEaaS:

• Rendering of PLC code, including graphical programming languages and hardware configurations in the browser. For example, Rockwell ladder logic is rendered as in its native IDE to increase odds of adoption by automation engineers.
• Near real-time monitoring of systems through IDEaaS over secure connectivity. Directly connect your IDE to your PLC from IDEaaS and upload, download, and monitor tags, input/output (I/O), and other output, depending on your specific vendor IDE.

Customer case study—Henkel Consumer Brands

Henkel Consumer Brands—a core business of Henkel AG in Germany—manufacturers a wide range of products in hair, laundry, and home care. Its well-known brands include Schwarzkopf (hair care) and Persil (laundry products). Henkel Consumer Brands uses SDA’s web browser–based IDEaaS for simplifying PLC code development across different PLC brands, facilitating direct hosting of development environments to streamline the development process and enhancing collaboration among control engineers.

Marcel Welz MSc., digital engineer at Düsseldorf Holthausen at Henkel Consumer Brands, attested to the ease of use of and the cost savings in PLC code development from SDA’s AWS-hosted IDEaaS: “The ability to run the PLC programming environment in any browser lets us maximize the use of our licenses, and every PLC programmer can access it with any device.”

End-to-end DevOps methodology for automation

Automation DevOps on the cloud offers a paradigm shift by providing a centralized, scalable, and accessible platform for PLC and robot code development. The integrated DevOps, IDE, and cloud storage becomes a cornerstone, letting developers collaborate seamlessly from anywhere and fostering near real-time code sharing. Automation machines and their PLC programs can be assembled from standard reusable code, much like building software from components or modules. An automation engineer can work on a team and be responsible for one or more of the machines and its PLC programs, dividing development and testing work very much akin to a sprint team. SDA offers manufacturers an overarching platform to implement DevOps methodology and enforce their enterprise-wide standards across various PLC coding or integrating vendors. SDA offers the following advantages:

• Remote collaboration: Geographically dispersed teams can connect through SDA’s web-based platform to collaborate in near real time on an AWS-centralized infrastructure.
• Check-in and checkout: Control engineers can securely check out, modify, or further develop code in the IDEaaS and submit it for testing and supervisor approval. Tested code can then be checked in. Pull requests to debug and manage versions using this process is explained later in this blog.
• Scalability: Cloud resources for IDEaaS can be scaled up or down based on project requirements, verifying optimal performance and cost efficiency.
• Accessibility: This helps developers to access PLC programs and vendor-specific IDEs to edit from any device with an internet connection, promoting flexibility.
• Project management: Control supervisors, typically owners from original equipment manufacturers (OEMs), can oversee code development and testing over multiple control-programming vendors or SIs toward continuous integration and testing. This functionality is further detailed in the following section.

During the on-site commissioning of new manufacturing lines, the PLC programs are added into an integrated production system automation software stack, tested in the pipeline, and deployed continuously against a preproduction system. Users need to connect to the running preproduction system to perform system-wide functional testing to validate how the system performs against operation, performance, and safety requirements. Automation engineers can then do a Factory Acceptance Test (FAT) after the preproduction system is validated and signed off by the factory operators. In addition, SDA offers the following features:

• Role-based access controls and the authority to change and commit PLC code that is integrated into enterprise access control systems, such as Active Directory.
• Traceability of who accessed which PLC code as well as what changes were made and when (also highlighted by the version comparison tool).
• Manual and automated code deployments for various vendors over the air.
• Ability to quickly undo rollback changes, if needed.

Customer case study—Grass Movement Systems

GRASS Movement Systems (GRASS) is a leading manufacturer of furniture hinges, slides, drawer systems and flaps—called movement systems—for furniture such as kitchen cabinets and office shelves for industrial manufacturers and fittings retailers. GRASS implemented SDA’s DevOps and PLC project management tools for end-to-end automation development and commissioning. The company can now conveniently open and manage PLC projects in any browser at any time, check in or out PLC code during development, and compare versions quickly and effectively without long loading times.

Collaborative code development and project management with multiple system integrators

OEMs typically manage the complex process of code development and plant commissioning by dividing the problem to various PLC engineering service providers or industrial automation SIs. These service providers work in silos, so any faults or unexpected behaviors observed during this phase require more on-site debugging and retesting across all service providers before the system can be validated for production. Thus, the OEMs have a complex management task in supervising and coordinating multiple service providers and code modules for eventual factory automation success. The OEMs also own the maintenance and lifecycle of the code for its changes, updates, retooling, and modernization. DevOps practices streamline project management, offering a cohesive environment for diverse teams. SDA’s cloud-based repositories provide remote access and collaboration between vendors and geographically distributed PLC programmers. The SDA solution based on AWS also provides a centralized repository to oversee and manage development, integration, and testing.

SDA’s industrial DevOps solution does not only provide fine-grain access control on the user and asset level; it also provides complete tracking capabilities for OEMs to monitor and steer the commissioning process as well as time-based access controls to limit a resource, such as a supplier, to a predefined access to the system, easing administrative burden.

Automated backups, version management, testing, integration, and commissioning

PLC code goes through many iterations, logical changes, and versions during its development, integration, testing, and commissioning. AWS facilitates collaboration and provides a centralized location for code reviews. Comparison of versions is also easier because the code is available at the central repository for various developers and experts, who can review and compare the code either synchronously or asynchronously.

When developers complete a new feature or fix a bug, they create a pull request using the checkout/check-in process. The request contains the changes they made to the PLC code. Other team members can review the changes, add comments, provide feedback, and suggest improvements before a code supervisor can approve and integrate the code into the main branch.

After the production system is commissioned and brought online, the OEMs have a challenge to continually maintain the system, which is often in service for 20–30 years, while minimizing downtime to avoid significant financial losses. The version management system’s ability to track every code iteration and facilitate easy rollbacks to stable versions verifies operational continuity and minimized downtime as well as a fast and full system recovery in case of a disaster.

Once commissioned, it is not uncommon for changes to the PLC code to be made. In fact, in many production facilities, micro-adaptations are happening daily to optimize production systems. These changes often require the automation engineer to manually save or share these changes with others on the team, a process that is prone to human error. Manual processes like this often lead to lack of traceability and a lack of clarity surrounding deployments. Automated PLC code backups from the PLCs directly to the AWS Cloud resolve this challenge by providing engineers with the latest running versions from an easily accessible location to debug or compare with previous versions. In such a scenario, SDA provides a centralized, automated backup system that captures changes made to PLC devices on the factory floor. A PLC backup schedule can be defined, and if a change has been detected on a PLC program, the system automatically initiates a backup, verifying that no modifications are lost. This process not only safeguards against data loss due to unforeseen events but also provides a comprehensive change log, enhancing traceability and accountability in PLC program management.

SDA’s automated backup system is configurable and runs on a schedule that can be centrally managed. By automating the backup process and centralizing control, it drastically reduces the operational risk and administrative burden associated with manual backup processes.

SDA offers the following advantages:

• provides automated backups to capture any changes made to the PLCs on the factory floor after the system is commissioned to augment industrial DevOps.
• detects if changes are made to the PLC and only backs up the PLC if changes are detected.
• centralizes management of PLC backup scheduling.
• represents each backup as a new version in the SDA repository.
• keeps detailed logs of the backup to provide full traceability.
• offers notifications of various backup statuses (this feature can be turned on or off).

Customer case study—Henkel Consumer Brands

To improve management and development of production machines, Henkel Consumer Brands implemented SDA’s Version Control for transparent code change management and web browser–based engineering. SDA also incorporated automated backup version control capabilities with a revision history tracking system that captures the “four Ws” of project management: what changed, when it changed, who changed it, and why it changed (see figure 1).

 

“With the new system, we can increase productivity for automation engineering and, at the same time, manage our machine suppliers’ and our own PLC codebase safely through fine-grained control rules,” says Welz.

SDA offers the following further advantages:

• Unified repository: A centralized code repository on the cloud verifies that all vendors work with the latest version of the code, minimizing integration challenges. SDA offers project file version control (who, what, when, and why), and code is committed under supervisory oversight.
• IDEaaS integration: SDA browser-based engineering, IDEaaS, provides for version changes within the managed SDA environment.
• Rendering of differences: The solution offers rendering of differences or changes in the code, including graphical programming languages.
• Near real-time communication: Cloud-based collaboration tools enhance communication across PLC vendor stacks, fostering an agile development environment.
• Traceability: All changes are logged, and through the version management, every change can be rolled back within minutes.
• Version comparison: The advanced version comparison and diffing solutions let multiple developers collaborate efficiently on the same file for code reviews.

Project independent controls standards

Manufacturers want PLC code to conform to certain standards. These standards can be regulatory, such as the FDA’s Good Manufacturing Practices (GMP) in the healthcare industry; enterprise-wide, such as standard tag names or error codes across repeated automation and across plants; or security-based, such as rules for data protection, cybersecurity, or code storage times for error traceability. Manufacturers also want to reuse repeatable code snippets—for example, conveyor-turntable controls with certain parameters, such as degree of turn and speed with standardized tags, error codes, and parameter names, to be reused across factory lines and geographies.

In all above cases, standardized PLC libraries, specifications, and historical code need to be stored independently of factory automation projects and made available for various purposes. SDA offers such a repository that can be securely accessed with authorized approvals. Such libraries can be integrated with SDA DevOps tools, distributed to control systems integrators as part of the coding specification, and supervised for compliance in code review and approvals.

SDA offers the following advantages:

• A global repository of standard hardware libraries, such as General Station Description (GSD) files for Siemens.
• Standards change management access to only authorized product managers.
• High availability and fault tolerance, verifying that the PLC standard libraries are accessible even in the event of failures or outages.
• Implementation of data security features, such as encryption of data files in transit and at rest and advanced user rights management up to a single PLC and its program file level.

Customer case study—GRASS

GRASS maintains standard hardware configurations (for example, GSD files) and reusable PLC standards centrally on SDA for their engineering teams. The customer can update configurations over the air from AWS through the SDA interface instead of manual work.

Implementation of SDA’s DevOps and its solution architecture on AWS

The AWS solution architecture for SDA as an IDEaaS is shown in figure 2. In this type of deployment model, SDA is responsible for managing the cloud infrastructure and the required software modules that run on it. Only a simple local connection agent needs to be installed, which most customers are able to do on their own within minutes. Because SDA does all the heavy lifting and shifting for automation engineers and IT on the customer side, the total cost of ownership is lower than traditional on-premises backup solutions.

 

The core features provided by SDA are Backup, Version Control, Browser-Based Engineering, and Secure Remote Access with Role Based Access Control. Version Control provides secure storage and traceability of PLC source code versions and changes backed by Amazon Simple Storage Service (Amazon S3), an object storage built to retrieve virtually any amount of data from anywhere. Version Pro is also central for collaboration, project management, the checkout/check-in process, and version comparisons. SDA Browser-Based Engineering uses AWS-hosted IDEs on Amazon Elastic Compute Cloud (Amazon EC2), which provides secure and resizable compute capacity for virtually any workload. These IDEs are streamed to web browsers using NICE DCV, a high-performance remote display protocol. SDA PLC Ops provides API-driven capabilities for vendor IDE interaction. It can be used for code-integrity checks and on-demand or scheduled backups of PLCs. This service is backed by Amazon EC2 for vendor-specific installations and Amazon Dynamo DB—a serverless, NoSQL, fully managed database—for metadata storage.

Users can access these applications through a web-based console. They have full control over setting project-level permissions per user. They can also enable temporary access for third parties, such as automation vendors for collaboration on a specific project. Customers can securely connect their factory floor to the SDA application by installing the gateway server provided by SDA in their factory network. This gateway server contains an SDA agent VPN client, which can establish secure and short-lived VPN connection between the SDA applications in the cloud and the factory floor on demand through a message request in AWS IoT Core, which helps users easily and securely connect devices to the cloud.

This secure connectivity facilitates deployment of projects to PLCs in the factory network from the SDA application in the cloud. SDA also provides a local client to be used with local installations of vendor IDEs. The local client provides near real-time code check-in, checkout, and synchronization, establishing cloud repository as the single source of truth. All the actions in SDA are also available through a REST API layer in Amazon API Gateway—a fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale—helping users to develop their own custom applications and integrations. In all interactions with the cloud, the data in transit and at rest is always encrypted to provide secure communication and storage.

Conclusions

Implementing DevOps methodologies and using the scalability and accessibility of AWS can revolutionize industrial automation software development and management. As manufacturers embrace cloud-based automation DevOps solutions like the one provided by SDA, they can centralize code repositories, enable remote collaboration, enforce their enterprise and regulatory standards, and streamline testing and deployment. With these features, manufacturers can increase efficiency while reducing costs and downtime. The SDA’s ability to automate backups, compare versions, and roll back changes provides enhanced traceability and control over the entire automation lifecycle.

DevOps methods can be further extended by simulation with virtualized I/Os in the cloud. AWS-based digital twin capabilities can offer virtual commissioning, testing, and validation of the PLC and robot code in the cloud prior to its physical deployment in the plant. Integrating machine learning (ML) algorithms could facilitate predictive maintenance and automated code optimization based on operational data. Additionally, the emergence of generative artificial intelligence (AI) can help in automating or assisting PLC and robot code documentation, explaining legacy code explanations, comparing versions, and suggesting code improvements. AWS and its partners are continually innovating with modern AI/ML technologies and bringing joint solutions in automation software management.

This post originally appeared in the AWS for Industries blog. It is written by Software Defined Automation’s Josef Waltl and Leo Kilfoy and AWS’s Srinivas Nidamarthi and Nishant Saini.