ITS Solving Customer Problems in the Field

We have received a customer call regarding client/server connection problems and error messages occurring in their power plant process-LAN.

The problem was: 

Some of the Mark V alarms/events weren’t written into both of the redundant SQL-Servers. The database content of the used tables was different without any understandable reason and the in-house IT technician wasn’t able to find the problem. 

Our technician was able to find the cause quite fast because the extended log-files created from every TMOS component, shows it in clear words. A timeout of the TCP/IP protocol occurred during establishing the connection between client and server. The timeout was reached after a delay of 3 seconds where the connection couldn’t be established in their modern gigabit-network. After some tests a needed connection time of about 4200 ms was measured.

The network connects two power plants and uses a great amount of switches and media converters between copper and fibre cables that there are a vast amount of opportunities for an error to occur. It took more than a day of digging into the Ethernet network structure and its components to finally find the problem.

The results from the availability-test, bandwidth-test and free system resources etc., were negative. Also the software configurations of the network interfaces of the server and switches were all well executed except one management interface. The SNMP-based TMOS network infrastructure overview display shows one interface which could not be accessed and matched to its component.

It is only after a walk through of one of the plants, checking all the hardware, that a possible issue was spotted on a dedicated server that is used for the ITS Online Combustion Monitoring System. The shared port for management and regular Ethernet was used to connect the process-LAN.

Our technician realised, as this is the only server with a shared ethernet port used in this network,  a detailed study of this server seems to be necessary. After remembering the perils of an unconfigured management interface and the process LAN on one single ethernet port he quickly found through his experience that this could be a reason for the problem.

After configuring the management port and modifying the second interface as the port used for the process LAN, the problem was solved. The connection time was back on about 3 ms, which is typical for these power plants.