Nuisance Fault 13012 "Error During ECat stack Initialization"

So, I got contracted to do a quick bit of support for an integrator that just installed a KR10R1000 on a KRC4 Compact (KSS 8.3.33). The robot had already been programmed and integrated successfully, I'm just on site to touch up a few points and be available if anything unexpected happens. The integrator has only basic knowledge of KUKAbots, and their resident KUKA "expert" is unavailable, hence me dropping in for a few days.

Now, the robot runs fine in Teach and Auto, but there's a constant warning message that just won't go away: KSS 13012 Error during ECat Stack Initialization (No network response).

I found that the WV project for this robot included several ECat I/O modules on the KEB, that didn't exist in hardware (I suspect the project was copy&pasted from an earlier, similar, system). There was no I/O mapped to these modules, which made their presence in the hardware tree harmless. So when I had a moment, I removed them from the project and re-deployed, thinking this would get rid of the nagging error.

It didn't.

Now, there's one other "nagging" message on the pendant that won't go away: PMS 18, Battery Defective (Load Test Failed). I have a ticket open with KUKA Service on this one, since the robot is brand new. I'm still waiting for a response, but I originally thought that these two errors were unrelated. Now I'm less sure.

So, why would I be getting an ECat error when I've deleted all the deletable ECAT modules from the KEB? This would imply it's an internal ECat error, but in that case, why does the robot run? I don't think I've ever seen an internal network error that wasn't fatal, or generated a safety fault of some kind. And could it be related to the Battery error?

As this is my first time hands-on with a Compact KRC4, is there anything different about the Compact (as compared to full-size KRC4s) that could be related to this?

If troubleshooting this involves opening up the controller, it's going to be a real pain -- the controller is placed deep in the guts of the machine, and getting to it will involve some major frame disassembly. Even the cabinet USB ports are unavailable -- the only way I can make backups is to use the SmartPad USB port.

So, it sounds like something disconnected in the controller? Bleah....

Diagnostic monitor for Network Interface SYS-X44 shows no errors at all.
However, the monitor for KEB SYS-X44 MasterStack shows an ERROR state, with a Lost Frames count equal to the Sent Frames count, and a Received Frames count of 0. The Master Requested State, Slave Requested State, and Master OK are all ERROR'd, and Master Status is "0 - Invalid". But the Send-to-Send and Receive-to-Receive times keep updating. Even with 0 slaves configured or found.

The Diag monitor for ECatIO shows no errors except for SYS-X44 (Code 3).

Whatever the problem is, it certainly seems to be part of the KEB, but there's no smoking gun to explain what. Or, for that matter, why the robot keeps running with what I would normally expect to be a fatal error.

Pulling a KRCDiag shows... weirdness. There are "spurts" of error 12051 and 12052 (KPP Fan E1 or E2 Revolution too low), where the error occurs every 8-20 minutes, but only for a second or so then clears itself. This can last for an hour or two, but then stop entirely for several hours. It doesn't seem to matter if the robot is running, idling, or sitting in Teach mode.

There's also some occasional 14004 errors (No communication to Power Management Service), that seem pretty random. They're not happening during E-Stops, mode changes, power cycles, just... randomly.

There's an occaisional numberless error "SmartPad Service Received message length 0", but that only seems to pop up every couple hours.

Just BoardPackage 1.4.0, EIP 2.0.3, and MultiSubmit 1.0.1, and there's only the primary SPS doing anything. Heck, even the production program is tiny -- about 10 pages, total? It's a really small, simple process -- this robot is essentially replacing a 30-year-old SCARA robot, but the process is barely changed at all.

Malware? No idea. Nothing obvious -- the SmartHMI is snappy and responsive at all times, never feels bogged down. And the HMI is usually where I see resource issue symptoms first, on KRC4s. No strange programs show up in the Windows install list. I can't get at the Windows Services list right now, but I'll check it when I can.
The KRCDiag does show a periodic string of "Microsoft Security Audit" messages, but they seem like just logged events, like a regular file scan.

Quote from mookie

Im working on the compact controller and all of my profisafe stuff works properly but nothing is coming across the wire for standard io.

That sounds like a completely different issue than the thread is about, but... hm? ProfiSafe works, but ProfiNet doesn't? That's a weird combo -- I've only ever seen it the other way around, generally due to a Safety ID being set wrong.

Normally, if PS works, PN should come along for the ride, since they're essentially different packets on the same channel. If there aren't any PN error messages, the only thing that leaps to mind is the mapping in WoV.

Quote from mookie

SkyeFire Fun Fact, One of the robots we have here has your name in the SPS.DAT. just thought it was funny.

Huh! In production, or recycled?