Decimal Places and Precision

Beginners in the field often ignore some simple facts concerning the number of significant digits of a result. The use of pocket calculators usually results in values with too many decimal places, regardless of whether this is meaningful. In general, one should use only as many decimal places as are compatible with the precision of your experiments. It may therefore be necessary to determine the precision of a measurement by repeating it several times and calculating the standard deviation.

Example:

Suppose the outcomes of several repetitions of an experiment have been recorded as follows 12.3075 12.3351 11.9949 12.2722 12.3117 12.0766 The four decimal places used to denote the results are meaningless, since the repeated measurements indicate that at most the first decimal place is valid (a closer analysis shows that the average of the result is 12.21, with a standard deviation of 0.14).

In addition to the signal variation the accuracy of measurements may become even worse for quantities which are determined by the calculation of differences or ratios.

Example:

The calculation of the electric resistance according to Ohm's law: R = U/I. In order to determine the resistance we measure the voltages U1 and U2 on both ends of the resistor. Let's assume that U1 is 8.97+/-0.01 V, U2 is 8.71+/-0.01 V, and the current I through the resistor is 0.015+/-0.001 A. From Ohm's law the resistance calculates as follows: R = (8.97-8.71)/0.015 = 17.33 Ohm. If we consider the measuring uncertainty, the calculated resistor may take the following extreme values: Rmax = (8.98-8.70)/0.014 = 20 Ohm Rmin = (8.96-8.72)/0.016 = 15 Ohm Thus the resistance is known with low precision - the accuray is around +/- 15%. Thus the resistance should be rounded to the nearest integer number, decimal places are meaningless and superfluous.