Sorry to ask ( i don't know how much you know) but the robot software will calculate it for you.
Do yo want to know the formulas the software uses ? Where would you like to apply your formula ?
My total guess is that (geometrically speaking). You can define 3 spheres were the center point is one of the 3 points you taught and then find the intersection. The four point will be use to find a line (defined by the intersection and the fourth point. the result will be a direction of your tool