304.8 = 304.799 ???

  • AD
  • Numeric registers are 32 Bit Floating Point numbers. In that data type there is no exact representation for 304.8, so it sets the register to the closest value 304.79998779296875. The screen only displays the last 3 decimal points so you see 304.799, but in reality the register is the longer number above.


    If you want to know more google "single precision floating point"

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account
Sign up for a new account in our community. It's easy!
Register a new account
Sign in
Already have an account? Sign in here.
Sign in Now