what is meant by instrument resolution and precision and how they are calculated
+2
A:
Resolution determines whether your instrument can measure a difference between a 19.3mm length and a 19.4mm length (this example would require a resolution of at least 1mm).
Precision determines how far the instrument-provided value will be from the actual value, such as a 19.3mm length being observed as anywhere between 19.1mm and 19.5mm (this example would be a precision of 0.2mm).
It would be best, of course, to read the documentation for your instrument (or your program requirements) to determine what conventions they use for representing and computing resolution and precision.
Victor Nicollet
2010-01-08 14:49:41