Rational orthogonal approximations to orthogonal matrices

Several algorithms are presented for approximating an orthogonal rotation matrix M in three dimensions by an orthogonal matrix with rational entries. The first algorithm generates an approximation M₂(M, ε) with accuracy ε and (2b + 2)-bit numerators and a common (2b + 2)-bit denominator (bit-size 2b + 2), where b = ⌋-1g ε⌉ (ε ≈ 2^-b). The second algorithm uses basis reduction to generate an approximation M_v(M, ε) with accuracy ε^v/^1.5 and bit-size vb for some 1.5 ≤ v ≥ 6 (but v cannot be controlled except by trial and error). A third algorithm, based on integer programming, generates optimal M_opt(M, ε) with accuracy ε and bit-size proven to be no more than 1.5b. In practice, the second algorithm generates an approximation with v ≈ 1.5 and is much faster than the third algorithm. The best bit-sizes which one could obtain using previously known results in two dimensions (Canny et al., 1992) are more than 3b bits for numerator and denominator. Applications are described for the approximation functions in the area of solid modeling.