EthernetRSI on KSS5.6 - odd 5.6 IPO cycle delay between command and motion

I'm developing a software that controls robots via RSI, and I just noticed a really, really weird bug - RSI's actual motion execution in is ST_ETHERNET/ST_SKIPSENS() mode is delayed by approximately 5.6 IPO cycles(~68ms).

It's running on a 12ms cycle, because EthernetRSIXML 1.2 on KSS5.6.12 cannot run in the 4ms mode(it just times out immediately, because it still sends packets 12ms apart for some reason, not 4).

Now, here's the weird part: it perfectly executes the motions. There's no lost packets, the delay marker never indicates anything other than 0. It's just that everything sent to the robot happens several cycles after it's been sent, as if there's a small buffer of motion commands before they get executed.

Is this normal for RSI? Judging by documentation and every piece of code I've seen online, people are treating it as if it's cycle-instant - you send it a correction in IPOC=1, and you see it returned in the next packet, IPOC=2. This is just not what's happening for me.

The robot is unloaded, and the corrections are minimal(single digit mm/s^2 accelerations and ultimate speeds), but not insignificant(they're not being ignored as lsb errors/below encoder resolution).

This is obviously not a networking issue, since packet numbering is fully sequential with no delay errors. The robot is connected directly to a dedicated ethernet port, too, so there's no buffering/offloading issues.

Any clue as to what's going on?

For reference, here's what the motion looks like.

ent_joint is commanded position, real_joint is what RSI returns. Y axis is angles, dots are 12ms apart(synced to the IPO cycle)

Quote from SkyeFire

Does your RSI program have any low-pass filtering or PID objects between the inputs and the (I assume) AXISCORR object?

Nope. I'm calling ST_ETHERNET to create the object, ST_SETPARAM(hEthernet,eERXFastCycle,0), then ST_AXISCORR for all 6 axes, then ST_NEWLINK for all 6 axes, ST_SETPARAM for relative positioning, very large upper and lower bounds(+-999), then straight to ST_ON1(#WORLD, 0) and ST_SKIPSENS() to hand over control.

Quote from SkyeFire

I have to wonder if this is a result of adding communication lags into the mix? You're charting commands vs transmitted "actual" position. I could see it potentially being a 3-cycle (36ms) total time -- 12ms for the command to be received and processed, 12ms to execute the RSI program, 12ms to transmit back.

The software receives the RSI packet, decodes it, does all the control math, and sends the response in ~240-300us(captured via Wireshark). It's definitely more than 12ms(since RSI is 12ms), could be 24 if it's executing it within the next cycle(but it shouldn't?), but it's way over 60, which is more than 5 cycles!

Quote from SkyeFire

What I would try, would be to use the RSIMON object, and track everything internally to the robot. Add a time stamp variable to the data being sent to the robot (just an integer that increments with each data exchange), and chart it alongside the input command and AXISCORR. In fact, you could also add that timestamp to the data the robot sends back, and use those stamps to align your computer data plot and the robot's internal RSIMon data chart. The lag has be be happening somewhere, the trick is to eavesdrop on different parts of the chain to narrow down where.

This RSI version is so old that there's no RSIMon. There's ST_MONITOR, which sends data in 12ms cycles to an external IP address, but I never got it to work properly.

On that note, I also can't figure out why the eERXFastCycle parameter(Supposedly #7 in ST_ETHERNET) doesn't actually kick the transmission rate into fast mode when set to 1. It certainly starts expecting packets earlier than 12ms, because it times out almost immediately, but it's not sending anything faster than 12, despite the manual saying it's 4ms, and everywhere else(.chm docs, example src) saying it's 2ms(2???). Maybe it's just not meant to be in KSS5.6.12/ERSIXML 1.2, but people did have success with it here.

Quote from Fubini

You could lookup the value in ms of the filter in one of small eko_dat of $robcor.dat. Maybe index 4 but I am not totally sure about the index

Doesn't look like it, unfortunately. $EKO_MODE=#OFF in the beginning, all $EKO_DAT are 0.0, uninitialized. This is probably because the robot is an older KR3(the CRS F3, not the Agilus) with a KRC3 controller(KRC2ed05 but KPC only), which has internal servo drives, and some of that stuff might be happening in there, but I'm not sure.

Quote from SkyeFire

Unfortunately, I never tried using IPO-Fast mode, so I can't speak to that. I know it had some definite limitations (in KSS 5) that meant it wasn't appropriate for my use cases, so I never dug in.

It'd be nice if it ran at all. When I enable fast mode, it just times out while still sending messages at 12ms intervals.

Quote from SkyeFire

There's also $FILTER, which limits acceleration to a minimum number of IPO cycles.

Tried playing with $FILTER, looks like the range on my robot is 0 to 15. Tried all of them, no change whatsoever.

Quote from SkyeFire

I'm not sure how to go about testing that possibility, though.

Well, I haven't found a cheap desktop KRC4 robot yet

Maybe it's the version of RSI I'm using? I keep seeing people mention RSI2.x, but I'm on EthernetRSIXML 1.2.

I don't expect anyone to actually use what we're building on a CRS KR3, I've barely even seen them in the wild myself, but that's what I have for testing. Makes it a lot easier to have a KRC3(2ed05) robot in an apartment.

Quote from SkyeFire

This is going waaaay back, but ISTR that the earliest versions of RSI did not support TCP/IP natively, only Fieldbus. Then ERX was added as an extra option to add TCP/IP to RSI. The only time I recall seeing RSI with TCP/IP on a KRC2, I think it was under KSS 5.2 or maybe 5.4, and RSI and ERX were separate packages -- we had to install both in order to get RSI working over TCP/IP.

I've now confirmed this with my contacts at KUKA. I was running specifically EthernetRSIXML 1.2, whereas the latest available for KSS[5, 6, 7] is actually RSI 2.3(not EthernetRSIXML), which has all the networking functionality built in.

Interestingly enough though, I have nothing besides EthernetRSIXML installed on the robot - it came as one package and that's all it took. Well, and a cursed network config(the option ate the MFC ethernet port despite my objections, and didn't see the extra 3COM card).

Quote from SkyeFire

Which was the entire point of the KR3, as I understand it. KUKA needed a desktop-sized robot that would run on typical household or classroom wall-socket power, and was small/weak enough to be minimally dangerous (this was before Cobots), for use in classrooms and transportable training cells. And they didn't make anything that small/light, so (apparently) they decided that the quick fix was to hack a KRC2 KPC and re-label some Denso robots.

The Denso robots came after, actually! The CRS F3 is a totally separate frankensteined robot, with harmonic drives connected to bike chains on the output, motors in funny places, drive units in-axis(hey we've come full circle with the Agilus). The Denso mechanical units seem to have been popular with many other manufacturers - Adept rebranded it too and still sells them under their name.

I wouldn't call it minimally dangerous - a few of my own minor collision incidents resulted in some major damage, but to the equipment, not the robot. Lots of torque, and you can comfortably run almost double the rated payload with no issue.

Quote from SkyeFire

Then... did CRS go out of business, or just get bought out by someone?

Their remains were absorbed into the great glob of Thermo Fischer Scientific(read: murdered).