Search All of the Math Forum:
Views expressed in these public forums are not endorsed by
NCTM or The Math Forum.



Charlwood Fifty test results
Posted:
Jul 5, 2013 3:14 PM


Following is a table comparing the results produced by 6 symbolic integrators on the Charlwood Fifty testsuite:
Charlwood Fifty Test Results # 1 2 3 4 5 6 7 1 0 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 0 0 2 1 0 2 1 4 1 1 1 1 1 1 0 5 1 1 0 1 0 2 0 6 2 2 2 0 1 2 0 7 2 2 2 1 2 2 2 8 1 1 0 0 0 1 0 9 2 2 2 1 2 0 2 10 2 2 2 1 0 2 0 11 0 2 2 2 2 0 0 12 1 1 2 1 0 1 0 13 1 1 2 1 0 0 0 14 0 2 1 2 2 0 0 15 2 2 2 1 2 0 0 16 0 2 1 2 2 0 2 17 2 2 1 1 2 0 0 18 2 2 2 2 2 1 2 19 2 2 2 2 2 2 2 20 2 2 2 2 2 0 2 21 0 2 1 1 0 0 0 22 0 1 2 1 1 0 0 23 0 2 2 2 2 0 0 24 2 2 2 2 2 2 2 25 2 2 1 2 2 0 2 26 2 2 1 2 2 0 0 27 2 2 1 2 2 2 2 28 2 2 1 2 2 0 2 29 2 2 2 2 2 0 0 30 2 2 2 2 2 0 0 31 2 2 1 2 2 0 2 32 2 2 2 2 2 2 2 33 2 2 1 2 2 1 2 34 2 2 2 2 2 2 2 35 0 2 2 2 2 0 0 36 2 2 1 2 2 2 2 37 2 2 1 1 2 2 2 38 1 2 2 1 0 2 0 39 1 2 2 1 0 2 0 40 2 2 1 2 0 2 1 41 0 0 1 1 0 0 0 42 1 1 1 1 1 1 0 43 2 2 1 1 2 2 1 44 1 1 1 0 2 1 0 45 0 0 1 1 0 0 0 46 0 2 2 2 2 1 2 47 1 1 1 1 1 1 0 48 1 2 2 1 2 2 1 49 2 2 1 2 2 0 0 50 2 2 2 2 2 2 0 Tot 65 85 75 73 71 49 40
In summary, the column labels and total scores for the 7 systems tested are as follows:
1. 66 Rubi 4.0 2. 85 Rubi 4.1 3. 75 FriCAS 1.2.1 4. 73 Mathematica 9 5. 71 Derive 6.1 6. 49 Maple 17 7. 40 Maxima 5.28.02
The table in slightly more readable format is available as a pdf file at http://www.apmaths.uwo.ca/~arich/CharlwoodFiftyTestResults.pdf The 50 integration problems and optimal antiderivatives making up the testsuite are available at http://www.apmaths.uwo.ca/~arich/CharlwoodIntegrationProblems.pdf The table uses a numeric grading system to compare the result produced by the builtin integrate function provided by each system with the optimal antiderivative as shown in the above file. Simplification of results using functions and/or domain restrictions on variables is not allowed. In other words, the results a novice user would see are graded as is. For each problem, if the system gives a result as good as the optimal antiderivative, 2 points are earned. If the system gives a mathematically correct result (i.e. its derivative equals the integrand) but is substantially more complex than the optimal antiderivative (e.g. significantly larger or involves the imaginary unit), 1 point is earned. If the system is unable to integrate the expression in closedform or timesout after 30 seconds or so, no points are earned. If the system returns a mathematically incorrect result, 1 point is decremented. So a perfect score on the Charlwood Fifty testsuite is 100. I would be happy to include other systems in the table if a knowledgeable user sends me its version number and numeric grades and/or raw test results for each of the 50 problems using the scoring system described above. Also if you think I have unfairly graded a system, please provide evidence and I will revise the table accordingly. Of course, the improvement of Rubi from a score of 65 to 85 was due to version 4.1 having been written in light of the Charlwood Fifty testsuite. Hopefully the other systems will quickly improve their scores as well... Albert



