Evaluating general purpose automated theorem proving systems

A key concern of ATP research is the development of more powerful systems, capable of solving more difficult problems within the same resource limits. In order to build more powerful systems, it is important to understand which systems, and hence which techniques, work well for what types of problems. This paper deals with the empirical evaluation of general purpose ATP systems, to determine which systems work well for what types of problems. This requires also dealing with the issues of assigning ATP problems into classes that are reasonably homogeneous with respect to the ATP systems that (attempt to) solve the problems, and assigning ratings to problems based on their difficulty.