Our methodology is based on XMach-1 (Böhme and Rahm 2001). That benchmark requires a Web-based application to model a general XML data management system. Based on the Web application, a number of performance measurements are taken: database population and scaling, document generation by queries, data manipulation operations, and operation mix. The methodology has two phases. The first phase is to explore the functionality of an XML-enabled database in handling XML documents. The second phase is to evaluate the performance of the XML-related libraries or gateway supported by the XML-enabled database.
Our methodology for measuring the performance of the two databases being tested (the XML-enabled database and the native XML database) is similar. The main difference is in the database design. We apply the method developed by K. Williams et al. (Williams et al. 2001) to convert a relational schema into a DTD. Then we use this DTD to generate an XML schema and store it in a database. A fixed number of records in text format are assigned to be imported into a database. The measurements are repeated from 100, 1,000, 10,000 and 50,000 sales records. Each measurement is repeated three times, and the average value is taken.
First, the database size is measured for both schema mappings. The objective is to investigate how efficient the database server stores XML documents. The size for building indexes is measured in megabytes.
We then examine the efficiency of the mapping schemes for the two database products. The time for XML document reconstruction from the database is measured in seconds. This tests the performance of the database engine. The time required to import from external XML sources into the databases is also recorded in seconds.
In our experiments, a sales order application was implemented for evaluating performance. The implementation involves using an XML gateway or XML libraries for mapping or data conversion. To cope with the performance measurements, the application has the following functions:
GUI interfaces to perform insert, update, delete, and search functions for single records
GUI interfaces to perform transactions (insert and delete) for bulk size records
A number of reports and queries to evaluate query optimizations
In developing the application, Java DOM APIs were used to directly access the native XML database server. The entire application was written using Java servlets. X-Query was used instead of SQL for reporting because X-Query was required to access the server using the HTTP protocol.
Since two versions of the application were implemented in this project, a number of APIs from the two database products were used. Our focus will also be on API usability comparisons, ease of programming, program size, program readability, compatibility, supported features, debugging tools, and functions related to their development environments.
After collecting data from the above-mentioned experiments, we were able to perform a detailed analysis. From the study of XML-related features of both databases, we were able to draw some conclusions that we present below.