Change Window Engineering in OT: Testing, Rollback and Approval

How an OT change is carried out on site: risk class, testing and pilot, PLC and switch backups, downtime plan, measurable rollback criteria and a go/no-go list.
Technical summary
Technology
IEC 62443-2-3, NIST SP 800-82, PLC firmware, industrial switches, MRP/REP, PLCSIM
Topic
Carrying out approved OT changes without putting production at risk
Level
Intermediate
Prerequisites
  • Basic knowledge of PLC and industrial switch management
  • Access to the plant's production and maintenance planning processes
In this article8 sections

A change started as “a five-minute firmware job” can turn into downtime on the production network lasting until Monday morning. In IT, a change that goes wrong leaves a service unreachable for a while; in OT, it stops the line, scraps a half-finished batch or causes a safety function to behave unexpectedly. Below, we describe step by step how an approved patch, firmware update or network change is carried out on site: risk class, pre-validation, backups, implementation plan, go/no-go and rollback. Which patch to apply is a separate decision; the subject here is the engineering of work that has already been decided.

Constraints that make change in OT different from IT

What makes OT change difficult is not the technology but the operating conditions. Before the plan is written, these are put on the table:

  • Availability first: Keeping the system running usually comes before both confidentiality and being up to date. Even a simple restart needs planned downtime.
  • Vendor approval: SCADA, HMI and engineering software vendors usually list which operating system patches they have tested with. A patch not on the list may affect the support contract.
  • Processes that carry state: Batch production has batches; continuous processes have temperature and pressure balance. The change is timed to a moment when the process can be safely stopped and restarted.
  • Dependency chain: A single switch restarting means ring reconvergence, connection drops on PROFINET devices and a flood of alarms on the HMI.
  • Legacy systems: End-of-support operating systems and workstations whose reinstallation has never been documented make rollback harder.

That is why copying the ITIL flow to OT as it is does not work. The approval mechanism can stay similar; the difference lies in the depth of technical preparation and coordination on site. Audit evidence is a separate job; here we look at the engineering side.

Types of change and risk classification

If you apply a process of the same weight to every change, one of two things happens: the team gets tired, or the process gets bypassed through the back door. The method that works in the field is to weigh the change on two axes: impact on production (none, one cell, one line, plant-wide) and reversibility (immediate, planned, difficult). The table below is a starting template; adapt it to your plant.

Example change Domain Rollback Risk class
Adding a syslog/SNMP target to a switch None Instant Low
New allow rule on a cell firewall One cell Instant Intermediate
Operating system patch on an HMI workstation One line Planned (image) Intermediate
Logic change in the PLC program One line Scheduled High
PLC or switch firmware update Line or plant Difficult High
Change to the ring/VLAN structure Plant-wide Scheduled High

Low-risk work can be defined as pre-approved “standard changes”: the procedure is fixed, it is done during production and only a record is kept. At medium risk, a pre-test and informing the shift supervisor are enough. At high risk, all the steps below are applied. An emergency change (an active fault or an exploited vulnerability) does not skip the process; it only changes the order: it is applied first, and the record and review are completed immediately afterwards. The “emergency” label must not become the process’s escape hatch.

Test environment and pre-validation methods

The ideal is a test environment that is a copy of production. Most plants do not have one. Even so, step-by-step validation is possible:

  1. Bench validation: The firmware update is tried on the same model PLC and switch from the spare parts stock. Update time, restart time and whether the configuration is preserved are measured. The times used in the window come from here, not from guesswork.
  2. Simulation: PLC logic changes are tested in the manufacturer’s simulation tool (for example PLCSIM in a Siemens environment). It catches logic errors; it does not fully reflect timing and I/O behaviour.
  3. Virtual copy: If HMI and SCADA servers are virtualised, they are cloned on a network isolated from production and the patch is applied there. The clone must run on a separate virtual network; a clone connected to the production network with the same IP and name causes bigger problems than the system you want to test.
  4. Pilot: If there are similar lines, the change is applied to one of them first and monitored for at least one full production cycle (shift, batch or recipe change).

The output of pre-validation is written down: what was tested, with which versions, what the result was, and what could not be tested. The last item is the most valuable. Every part that could not be tested goes into the implementation plan as an extra observation step.

PLC project, firmware and switch configuration backups

The precondition for rollback is a verified backup taken immediately before the change. “There’s a project file somewhere on the server” is not a backup. A common situation: the program running in the PLC and the project in the archive are not the same, because a small fix made during the last commissioning never made it into the archive.

  • PLC: The current project is uploaded from the device with the engineering software and an online/offline comparison is made with the archived project. If there is a difference, that difference is explained first; no change is made on top of an unknown difference. The firmware version, hardware order number and, if there is one, a copy of the memory card are recorded.
  • HMI/SCADA workstations: A system image is taken and it is verified once that the image can be opened. Licence files and hardware-bound licences are noted separately; licensing is usually the first thing that gets stuck in a restore.
  • Switches, routers and firewalls: The running configuration and firmware version are recorded; it is checked whether the configuration has been written to persistent memory. An unsaved running-config is lost at the first restart.

On Cisco IOS-based industrial switches, a minimum pre-change record can be taken as follows:

show version | include Version|uptime
show running-config
show interfaces status
show spanning-tree summary
show rep topology            ! if REP is used
copy running-config scp://[email protected]/cfg/cell3-sw1-before.cfg

The same outputs are taken after the change too; the difference between the two outputs is the basis of post-change verification. If MRP or a vendor-specific protocol is used as the ring protocol, the relevant status command is added to the list. The name of the ring’s “healthy” state varies by protocol (closed on the ring manager in MRP, segment complete in Cisco REP); the plan is written using the terminology of the protocol in use.

Implementation plan and downtime window coordination

The implementation plan is concrete enough to be read and followed on site. The owner, expected duration and success criterion of each step are written down. The template below is for a high-risk switch firmware update:

Change    : Line 3 cell switches firmware update (4 devices)
Window    : Sunday 06:00-10:00, line 3 planned maintenance downtime
Roles     : Window lead (maintenance), implementer (OT network), observer (process)
Precond.  : Line in safe state, batches completed, backups verified

T+00  Confirm line status, screenshot of HMI alarm list      -> process
T+10  Pre-change record commands, ring state closed/complete -> OT network
T+20  SW-4 (ring end) update and restart                     -> OT network
T+35  SW-4 verification: version, ports, ring closed again   -> OT network
T+40  GO/NO-GO 1: is SW-4 OK?                                -> window lead
T+45  SW-3, SW-2, SW-1 in turn, verification after each
T+120 PROFINET device states, I/O test, HMI communication    -> process
T+150 GO/NO-GO 2: release the line to production or roll back
Latest rollback start time: T+180

There are two points most often left out of plans. The first is the “latest rollback start time”: the latest time rollback must start for it to finish within the window. Without this line, the team keeps pushing forward until the window ends, and the rollback spills into overtime. The second is the observer role. The person typing commands cannot watch the process’s reaction at the same time. Someone from the process side must follow alarms and critical values on the HMI.

Production planning, shift supervisors, maintenance and, if necessary, the machine builder are informed before the window. If a manufacturer is going to provide remote support, the connection is tested before the window and opened for a limited time. Finding out on the morning of the window that the VPN client does not work is a common waste of time.

Rollback criteria and the go/no-go checklist

The rollback criterion must be measurable enough not to cause arguments on site. “If there’s a problem, we’ll roll back” is not a criterion. Examples: if the ring does not return to the closed state within 5 minutes; if any PROFINET device remains in a connection fault; if a communication alarm on the HMI that was not there before the change persists for 10 minutes; if the latest rollback start time has been reached.

Go/no-go list to use before the window starts:

  • The change approval and risk class are recorded.
  • The pre-validation result is written down; the points that could not be tested are known.
  • PLC projects, images and switch configurations have been taken and are stored in a second location.
  • The integrity of firmware files has been verified against the hash published by the manufacturer.
  • The rollback method and its duration are known; if downgrading is not supported, a spare device is ready.
  • The window lead, implementer and observer are on site; the communication channel is defined.
  • The process is in a safe state; operators have been informed.
  • The firewall rules needed between the engineering workstation and the target devices are open for a limited time.

If a single item on the list is not met, the change is postponed. Under pressure, this rule bends; to keep it from bending, the list must have been shared with managers before the window. A postponed window is cheaper than a half-finished one.

Post-change verification

The job does not end when the device boots up. Verification is done at three layers. At the network layer, the pre-change record commands are repeated and the outputs compared; port error counters are reset and read again a few hours later, because some problems only show up under load. At the control layer, the PLC diagnostic buffer is reviewed, the online/offline comparison is repeated, and I/O and safety functions are tested. At the process layer, you check whether the quality and speed values of the first batch or first shift are in the normal range, and whether gaps have appeared in the data flow in the historian.

For quick diagnosis of communication problems, the PROFINET fault diagnosis steps can be used. Secure access to network devices during a change requires a management path separated from production traffic; its design is in the article OT management networks and out-of-band access. In segmentation projects, every wave is in effect a change window; the plan for segmenting a flat OT network is based on the method here. For organisation-wide management of records, owners and approvals, see the IT change and configuration management process.

The final step is closing the record: new versions, a new configuration backup, observed deviations and lessons for the next window. Which vulnerability to patch and which to manage with a compensating measure falls under OT vulnerability management and patch decisions. For patch management, the IEC 62443-2-3 technical report and NIST SP 800-82 Rev. 3 can be used as a framework. If you would like to review your change process and OT asset inventory together, see our factory OT infrastructure survey and OT cybersecurity and segmentation services.

Frequently asked questions

Does every OT change need a downtime window?

No. Work that does not touch the process, such as adding a syslog target to a switch or changing the description of an unused port, can be done during production. Changes that touch firmware updates, downloads to the PLC, or the ring or VLAN structure, however, are done during a controlled shutdown or after the relevant part of the line has been brought to a safe state. The change’s risk class decides, not a sense of urgency.

How is a change validated if there is no test environment?

A bench trial with the same model PLC and switch from the spare parts stock, a logic test in PLC simulation software, and first a pilot on a single line or cell. The change is not rolled out to other lines until the pilot has been monitored for at least one full production cycle.

Who makes the rollback decision?

A single person named in the plan and present on site at the time; usually the window lead from production or maintenance. Because the criteria are written in advance, the decision is not debated: if the measured result does not meet the criterion, or the latest rollback start time has arrived, the change is rolled back.

Can you go back to the old version after updating PLC firmware?

It depends on the manufacturer and model. Some models allow a downgrade, some do not, or require the project file to be recompiled for the old version too. Before the rollback plan is written, the downgrade conditions in the release notes are read; if downgrading is not supported, the rollback method is a physical swap with a spare CPU kept ready.

Support

Contact us for support or a discovery session on this topic.

Contact form
[email protected]

Eretek Industrial

PLC, PROFINET/PROFIBUS and SCADA communication, IT/OT network segmentation and industrial field connectivity. We install and monitor without stopping the production line.

Knowledge Center

Other articles