Drexel dragonThe Math ForumDonate to the Math Forum



Search All of the Math Forum:

Views expressed in these public forums are not endorsed by Drexel University or The Math Forum.


Math Forum » Discussions » sci.math.* » sci.math.symbolic.independent

Topic: Charlwood Fifty test results
Replies: 16   Last Post: Sep 19, 2013 10:09 PM

Advanced Search

Back to Topic List Back to Topic List Jump to Tree View Jump to Tree View   Messages: [ Previous | Next ]
Albert D. Rich

Posts: 220
From: Hawaii Island
Registered: 5/30/09
Charlwood Fifty test results
Posted: Jul 5, 2013 3:14 PM
  Click to see the message monospaced in plain text Plain Text   Click to reply to this topic Reply

Following is a table comparing the results produced by 6 symbolic integrators on the Charlwood Fifty test-suite:

Charlwood Fifty Test Results
# 1 2 3 4 5 6 7
1 0 2 2 2 2 2 2
2 2 2 2 2 2 2 2
3 0 0 2 1 0 2 -1
4 1 1 1 1 1 1 0
5 1 1 0 1 0 2 0
6 2 2 2 0 1 2 0
7 2 2 2 1 2 2 2
8 1 1 0 0 0 1 0
9 2 2 2 1 2 0 2
10 2 2 2 1 0 2 0
11 0 2 2 2 2 0 0
12 1 1 2 1 0 1 0
13 1 1 2 1 0 0 0
14 0 2 1 2 2 0 0
15 2 2 2 1 2 0 0
16 0 2 1 2 2 0 2
17 2 2 1 1 2 0 0
18 2 2 2 2 2 1 2
19 2 2 2 2 2 2 2
20 2 2 2 2 2 0 2
21 0 2 1 1 0 0 0
22 0 1 2 1 1 0 0
23 0 2 2 2 2 0 0
24 2 2 2 2 2 2 2
25 2 2 1 2 2 0 2
26 2 2 1 2 2 0 0
27 2 2 1 2 2 2 2
28 2 2 1 2 2 0 2
29 2 2 2 2 2 0 0
30 2 2 2 2 2 0 0
31 2 2 1 2 2 0 2
32 2 2 2 2 2 2 2
33 2 2 1 2 2 1 2
34 2 2 2 2 2 2 2
35 0 2 2 2 2 0 0
36 2 2 1 2 2 2 2
37 2 2 1 1 2 2 2
38 1 2 2 1 0 2 0
39 1 2 2 1 0 2 0
40 2 2 1 2 0 2 1
41 0 0 1 1 0 0 0
42 1 1 1 1 1 1 0
43 2 2 1 1 2 2 1
44 1 1 1 0 2 1 0
45 0 0 1 1 0 0 0
46 0 2 2 2 2 1 2
47 1 1 1 1 1 1 0
48 1 2 2 1 2 2 1
49 2 2 1 2 2 0 0
50 2 2 2 2 2 2 0
Tot 65 85 75 73 71 49 40

In summary, the column labels and total scores for the 7 systems tested are as follows:

1. 66 Rubi 4.0
2. 85 Rubi 4.1
3. 75 FriCAS 1.2.1
4. 73 Mathematica 9
5. 71 Derive 6.1
6. 49 Maple 17
7. 40 Maxima 5.28.0-2

The table in slightly more readable format is available as a pdf file at

http://www.apmaths.uwo.ca/~arich/CharlwoodFiftyTestResults.pdf

The 50 integration problems and optimal antiderivatives making up the test-suite are available at

http://www.apmaths.uwo.ca/~arich/CharlwoodIntegrationProblems.pdf

The table uses a numeric grading system to compare the result produced by the built-in integrate function provided by each system with the optimal antiderivative as shown in the above file. Simplification of results using functions and/or domain restrictions on variables is not allowed. In other words, the results a novice user would see are graded as is.

For each problem, if the system gives a result as good as the optimal antiderivative, 2 points are earned. If the system gives a mathematically correct result (i.e. its derivative equals the integrand) but is substantially more complex than the optimal antiderivative (e.g. significantly larger or involves the imaginary unit), 1 point is earned. If the system is unable to integrate the expression in closed-form or times-out after 30 seconds or so, no points are earned. If the system returns a mathematically incorrect result, 1 point is decremented. So a perfect score on the Charlwood Fifty test-suite is 100.

I would be happy to include other systems in the table if a knowledgeable user sends me its version number and numeric grades and/or raw test results for each of the 50 problems using the scoring system described above. Also if you think I have unfairly graded a system, please provide evidence and I will revise the table accordingly.

Of course, the improvement of Rubi from a score of 65 to 85 was due to version 4.1 having been written in light of the Charlwood Fifty test-suite. Hopefully the other systems will quickly improve their scores as well...

Albert



Point your RSS reader here for a feed of the latest messages in this topic.

[Privacy Policy] [Terms of Use]

© Drexel University 1994-2014. All Rights Reserved.
The Math Forum is a research and educational enterprise of the Drexel University School of Education.