axis X and axis Y discorrect move

Is this an Absolute Accuracy robot? If so, that may be part of the problem -- AA robots only apply the AA calibration corrections when the robot is running a program, not when the robot is being jogged manually.

Is the payload Mass and CG correct? Is the mass at least 30% of the robot's rated payload?

Just because you program a perfect square does not mean the robot moves that perfectly. For one thing, milling operations mean that the robot encounters resistance, which causes the robot arm to flex slightly, with an end result that the robot thinks it is making a perfect square, but is not in reality. How much resistance force is the robot encountering?

Finally, robot backlash makes it nearly impossible to perform a "closed" shape like a square or circle accurately. The nature of a circular milling motion means that various axes of the robot are reversing direction at least 4 times around the circumference of the circle, which drives 100% of the backlash inaccuracy directly into the shape you are milling. The very same effect is seen in classical 3-axis CNC machines, but robots are much worse due to having less rigidity and more backlash than CNC machines.