Strange IRC5 failure -- lost pendant and EIP, but RS connection fine

IRC5, RW 6.12

This one was a bolt from the blue -- literally. This robot has been in service for a few months without any issues. I was connected to the service port with RS and examining some I/O settings when suddenly my connection dropped. I checked the FlexPendant, and saw that the controller was rebooting. No one was near the cabinet, and I hadn't hit the Reboot button in RS, so I assumed there had been a mains power hiccup or something.

However, the FP hung at "waiting for connection to robot controller," and couldn't get past it even after 20min. I started digging, and found that the robot was still apparently communicating with one EIP Slave module, but the EIP Masters connected to the LAN and AnyBus ports were reporting lost communications. I rebooted the robot again using the cabinet power switch, and was able to get RS communications back, but the FP still hangs at the same message.

There were no strange messages in the event log when I looked at it using RS. Just a "Shutdown initiated" followed by a normal stream of restart messages when the "spontaneous" reboot happened.

If I try to activate the FP monitor from inside RS, it times out. I've never tried using RS on an IRC5 that didn't have an FP connected, so I'm not sure if this is normal or odd.

So, the controller itself seems fine? But I've lost my FP connection, and 2 out of 3 of my Ethernet/IP connections. No obvious smoking guns. I was literally standing by the cabinet when this happened, and no one was touching the robot. There was some electrical work going on adding earth grounds to the line equipment, but there were no dramatic events (sparks, smoke, "yow!", etc), and a quick check of the wiring doesn't reveal any shorts or damaged cabling.

Anyone ever seen anything like this? I don't see any place where the AnyBus, WAN, and FP connections have a common failure point, especially when my LAN connection and the service port still work. Normally, I would think anything that could kill 3 of the busses like this would fry the rest of the controller. I'm tempted to just try swapping the FP with another robot, but if there is some "mystery short" in the wiring I haven't found, I'm a bit afraid of killing another FP.

Addendum: the FP cable got pinched badly (someone dropped something heavy on it). Didn't break the outer jacket, but apparently damaged one or more wires inside. We swapped the cable with another FP's cable, and there appears to have been no lingering damage done to the FP itself or to the IRC5.

In retrospect, it seems obvious, but the strange failure of some-but-not-all of the Ethernet/IP interfaces had me looking for a fault inside the cabinet.

Quote from Black

try to change only the cable from flexPendant

That's exactly what we did. Pendant and IRC5 were fine, it was just the cable.

Well, I would try plugging a laptop directly into the ports on the motherboard, just to see if they light up. If not, I would suspect a power or motherboard issue.