I haven't tried Roboguide with Modbus but I have done Ethernet/IP and OPCUA in Roboguide.
1. I found wireshark to be a helpful tool. I was able to find some error message in one of the Ethernet packets that helped me figure out my problem.
2. In Windows you can open a Command Prompt as Administrator. Run the command: netstat -ab
It will list all of the ports. This will help you to see what ports Roboguide has opened. For example, I have FRVIRT~1.exe (roboguide) listening on Port 4880 which is used for OPCUA. If port 502 is not opened by Roboguide then you won't be able to connect.