Till now we did standard ABB robot arm joint calibration and TCP position calibration as described in RoboDK docs. TCP orientation is just rotated 90 degrees relative to robot flange.
Seems like angle errors comes from joint inaccuracies, as angle error changes when moving tool along single axis.
We also did some custom calibration where we used laser distance sensor and collected real distances to the blade surface and theoretical distances using virtual environment. Did some optimization on that and got orientation offset which improved angles but not as much as we would like. Initially angle error could go up to 1.3 degree, now getting up to 0.7 degree error depending on joint configuration and where tool is pointing to. Preferred error is up to 0.2 degree (based on saw blade diameter).
Accidentally missed SkyeFire first comment. Thanks for the idea, will give that a try.