| precedence | Index Level |
| Syntax | PRECISION num.constant |
| Category | BASIC |
| Type | Statement |
| Description |
defines the number of fractional decimal places that a numerical value can hold.
The default precision is four. "precision" may be set in the range of 0 to 9. Numbers with more decimal places than the precision are truncated on R83 and D3. Only one "precision" statement is allowed in a program and the "precision" statement must precede the use of any numeric data. Programs calling subroutines or entering other programs must have the same precision. (See "call" and "enter"). If the precisions do not match, the program aborts into the debugger. This restriction also applies to main programs which share data in named common space. However, violations in this case are not reported, and values of named common variables will be incorrect. For certain "arithmetic functions", higher "precision" limits the magnitude of the values returned. In R83, where the maximum precision is 6, exceeding the limits of "precision" results in a negative number and invalidates the results of the calculation. In D3, most functions have no limitation on their numeric range. "precision" in the range of 1 to 9 handles numbers as 48-bit scaled binary numbers. If the result of an "arithmetic.expression" which divides (/), multiplies (*), takes a remainder (\), or produces an exponent (^) exceeds the maximum magnitude of the 48-bit representation, the system creates an internal variable type which uses a precision of 18 digits to the right of the decimal point, and unlimited to the left. This feature assures maximum accuracy. Operations which exceed the limits of the current "precision", round the results rather than truncate; (99999/100000 with "precision 4" results in a 1). A "precision" of 0 forces 48-bit integer arithmetic for all operations. (99999/100000 results in a zero). |
| Options | |
| See Also | Boolean Evaluation operators statements & functions LN ENTER INT \= CALL CHAIN SUBROUTINE arithmetic operators RND - *= += -= /= := COMMON + * |
| Example |
precision 2
This statement changes this program's arithmetic precision to 2. All subroutines called from this program must have the same "precision" statement. precision 6 print 9999999/10000000 The result of the statement is rounded to 1 since the limits of precision have been exceeded. The expression (9999999/10000000) when converted for print is rounded when it is translated to a string. |
| Warnings | |
| Compatibility | D3 7.0 AP R83 |
| precedence | Index Level |