The authors of the current test implementations do not see themselves as experts for SQL, JDO or Hibernate or for any of the database engines used. If you think that you can provide a faster implementation for a circuit, please supply it to the PolePosition project instead of criticising the authors for ignorance or the test for lacking objectivity.
This is a community project. It is for you! Please think positive, help us and be nice to us for starting the initiative. If you have a strong ego, good! Please write a better Circuit, Team, Car or Driver than we did instead of laughing about us.
Objectivity must be the most important goal of the benchmark tests. The results show very clearly that there can not be one single "best product". Depending on your application, some circuit results may be very important for you and some may be completely irrelevant. If you consider basing your choice of database on this benchmark, bring along lots of time to look at what every test does. You will not be able to judge the results without looking at the source code that produces them.
Original PolePosition test suite doesn't include Perst port. So I have created Perst team for PolePosition myself. Unfortunately I have to copy some files from PolePos framework and make my local copies of them (because Perst requires all persistent capable classes to be derived from Persistent base class). It is possible to avoid this requirement, using AspectJ or JAssist reflection packages but it will significantly complicate integration of Perst in this test suite. The sources of Perst team can be taken here. I have performed competition only between Perst and db4o because it seems to be the most close competitor for Perst and because all other databases requires some kind of installation and launching server. Please look at the results. I have to notice that advantage of db4o in write and update operations can be explained by lack of transaction support in db4o (at least in configuration used for testing - it doesn't perform synchronous write to the disk during transaction commit and so power or system failure can cause loosing of committed changes and even corruption of the whole database file. In Perst it is also possible to avoid synchronous writes, but I think that it will be not honest to switch on this option in the test measuring performance of transaction commits.
Patched version of OO7 benchmark for Ozone is available here: oo7.zip
Results of the benchmark (in milliseconds) are as follows:
Product | create small | query traversal | query match |
---|---|---|---|
Perst | 1 041 | 1 352 | 0 431 |
Ozone | 112 732 | 13 680 | 0 591 |
The version of TestIndex project for ObjectStore is available here: testindex.zip.
Another object oriented database for Java/C# is db4o.C# and Java version of TestIndex port for db4o is avilable here: db4o-testindex.zip.
One of the most popular databases for embedded applications is BerkeleyDB by Sleepycat Software, including a Java version of BerkeleyDB. It doesn't provide transparent integration with Java - it is not possible to transparently store/fetch instances of Java classes. Instead of it database is represented as set of <key,value> pairs and application classes can be stored using a serialization mechanism. The port of the TestIndex.java example for BerkeleyDB is here: je-testindex.zip.
And yet another competitor is JISP (Java Indexed Serialization Package). Jisp provides implementation of B-Tree and is able to store objects in the database using serialization mechanism. It seems to be all it can do. Jisp is written in pure Java and is very small. The version of TestIndex for JISP is here: jisp-testindex.zip.
We should also consider JDO compliant databases, such as ObjectDB, but its evaluation version does not allow storage of more than 5000 objects in a file and also does not support collections, so a meaningful comparison is not possible. A popular personal JDO-compatible database is FastObjects from Versant. It fully supports JDO and provides many of its own extensions. Unlike ObjectDB which enhances byte-code at runtime using a special class loader, FastObjects requires you to explicitly run a byte-code preprocessor after compilation. Unlike all other competitors, FastObjects is not written in pure Java - it requires loading of native code. The port of TestIndex for FastObjects is here: poet-testindex.zip.
Time of each step execution in milliseconds is measured. Number of records in each case is the same - 100000. Page pool size for Perst in is 32Mb, which is enough to hold all records in memory. All test were executed at the same computer: AMD Athlon 64 (3200+), 1.5Gb RAM, WinXP. I am using Sun Java JDK 1.4.2_04.
Product | Language | Create | Search | Remove |
---|---|---|---|---|
Perst | Java | 3 775 | 1 683 | 3 275 |
Perst | CSharp | 4 446 | 2 403 | 3 975 |
ObjectStore PSE Pro | Java | 8 272 | 9 413 | 3 104 |
FastObjects J2 | Java | 13 399 | 10 856 | 38 435 |
FastObjects.Net | CSharp | 43 012 | 2 714 | 7 461 |
db4o-4.0 | Java | 18 457 | 6 279 | 38 886 |
db4o-4.0 | CSharp | 31 725 | 41 099 | 88 517 |
Berkeley DB JE | Java(*) | 15 513 | 10 755 | 12 758 |
JISP | Java | 350 674 | 343 063 | 527 248 |
I didn't perform this test for other DBMSes, because it requires a lot of time and free space on disk. But if you think that some other pure Java or C# DBMS can offer better results, I will be glad to perform competition.