Very peculiar path bug... or possibly haunted robot

Okay, so...

There's a program on our MA3100/DX100 cell that precedes my employment here, and to my knowledge hasn't been modified whatsoever. It's a basic hinge program, and for reference it's like a 90° angle where the program (which I will attach) welds four brackets along the base plane, then the part is removed and flipped and put back on the positioner to do the welds on the other plane, with a jig that allows absolutely zero chance for the operator (who also hasn't changed, by the by) to place the part on it improperly. Plus, he batches these at like 50 per go so it's not like he would suddenly forget how to do it in that run.

Here's the problem:

So there's 16 main welds in the program, 8 for each plane. We've never had a problem with it until about the new year, at which point our operator noticed that seemingly at random, the 9th weld (which is the first one in the second orientation of the part, the 2nd half of the program) will just, malfunction and attempt to start at an almost 2 inch offset. Nothing changes in the job, it pauses for the flip, he flips it, and resumes welding. It does a quick tac weld for each, which it does fine for this weld, but after the tac when it goes to COMARC the joint it just, starts way off. They're all comarc joints, so that part isn't out of the ordinary, just the haunted weld.

Some other things to note:

- The dude who programmed it no longer works here,

- It has never, ever malfunctioned before

- It is 100% always the same weld

- There is no consistency to when it will happen though the welder says it generally will be either at the beginning of a run or toward the end

- If the first one malfunctions, the rest of the 7 for that half will too by the same amount (there's no SFT instance though)

- No positional or major parameter data has changed on the robot\

- Nobody has proven that robot ghosts aren't real

Some oddities in the program, though. Like there are two questionable external axis command points right before the weld that screws up but the weld start point has the correct one. I'm giving you the original unedited program without the fixes I've tried so that it's not further corrupted by my 'helping.' You'll notice negative values for the comarc amplitude for some of the calls. This, too, baffles me as I'm not a physicist or mathematician and as far as I know for a symmetric waveform you pretty much always deal in absolute values, unless you're working with current which, I don't think applies here? I could be wrong. There's also a P variable defined in the header data which I'd never seen before and could very well be confusing the controller. It gets called in some weird ways, too.

Related followup question: Anybody know if there is a way to look at the code that COMARC uses? I couldn't find it, and my hex reader doesn't like yaskawa's code and doesn't tell me much.

If anyone likes mysteries or ghobots by all means, please check out the code. If you're using a text editor that has a line counter, the error is around the literal line 537, which is commented rather clearly as Weld 9.

Thanks!

randombugjob.txt

Quote from Boring_robot_guy

Are you able to send a backup and cmos?

I could, if you think the problem is in there; but would it be if it's just that literal one job?

Quote from Motouser

Is it possible that the workpiece after the flip is in a wrong position?

It is possible I suppose, but I should note that the welder/operator is rather meticulous and very good at what he does, and we also have a jig/fixture that pretty much ensures that it'll always be in the same spot every time, give or take. I wish I had video of the error, it's surreal how far off it goes.

I apologize for the late response, it's been a bit of a week.

I did some deep diving into the steps themselves and discovered that there was redundant movement between the start of that part of the program and it's respective first weld - it has a lightning-fast MOVL to the part and then a slower approach MOVJ that, for an almost non-discernible amount, the external pulse values change before going back to the proper one. I'm wondering it hits that part of its ladder so fast that maybe sometimes it does this tiny turn, and sometimes does not based on... who even knows, something in the air? Something executed prior in the program?

I replaced them with a singular, slower move statement and added a wait statement so it can catch up with itself but we haven't had a batch to test it on yet. So I'll keep you posted.

Quote from Motouser

Just to give my opinion: I think that the problem is the welder instead of the robot's moves. Maybe there is something that delays the start of your welding, maybe the cables of the welder are in a strange position around the 9th welding line or the cables have a little damage that is forced during those moves?

Absolutely within the realm of possibility. If the new program doesn't fix it we will start troubleshooting on the welder.

Still though - so strange how it's random and infrequent; you would think with such precise movement it would replicate the fault more consistently. It uses COMARC as well which is completely unecessary for these tiny welds so the fitter making a mistake such that it caused an initial path deviation could be the cause as well.