WebBench - A Tool for Exploring JDataStore Performance

by the JBuilder Team

Abstract:  WebBench is a multi-threaded application that shows how to establish concurrent connections to a transactional datastore. It demonstrates JDataStore's ability to support large amounts of data and many concurrent transactions. You choose the number of concurrent users and other configuration options, set WebBench running, and see how many transactions were executed. In this way, you can use WebBench as a benchmarking tool.

Who can run WebBench?
To run WebBench, you must have JBuilder 3.5 Pro (with JDataStore installed), JBuilder 3.5 Enterprise, or the standalone JDataStore 3.5.  You can download JDataStore 3.5, with a license for development use only, from JDataStore 3.5 Downloads.  All these products include WebBench as a sample, but you can get a slightly newer version from [link].  Then follow the instructions for running WebBench.

Introduction
WebBench generates sample data for an order-entry application and runs sample transactions. Through the application's UI or code you can set parameters such as table size, number of connections, type of transactions (read-write, read-only or both), and the duration of the test. When you select Run, the requested connections to the datastore are established and begin running transactions. After it has run for the requested time, WebBench shuts down the connections and reports the number of transactions executed.

Unlike most sample applications, WebBench features industrial-strength - and industrial-sized - tables. Generating the datastore takes five to ten minutes, and the store and its log files together occupy about 150 megabytes. The data itself isn't very interesting - sometimes just strings of random characters. However, the data is realistic where it counts: in the mix of datatypes used, the relationships between tables, the index defined, and the values in key columns. As a result, this data can be used to simulate realistic production transactions and to investigate how throughput varies with table size, number of concurrent users, transaction type, and other variables.

WebBench's UI isn't dramatic - just the necessary menus and status reporting. You may want to use the DataStore Explorer along with WebBench to see what's happening inside the datastore. If you do, remember that WebBench and DataStore Explorer can't open a datastore at the same time. Only open the store in the DataStore Explorer between WebBench operations, and close it again before the next WebBench run. If you run read/write transactions, you'll see that new records are added at the end of the ORDERS and ORDER_LINE tables (check the values in the timestamp column to see when they were added). Several other tables are updated. Study the SQL statements in ReadWriteTx.java for details.

Running WebBench
Follow these steps to create the database and populate it with data

1. Select Bench | Options. Use theFilename and File Path options to specify the name and location of the sample datastore that will be created.  If you accept the defaults, webbench.jds will be created in the .../samples/JDataStore/WebBench subdirectory of your JBuilder directory.  By default, the datastore's log files will go in the same directory.  You can direct them to another location by setting the LogA Directory option.  For now, accept the default values for the other options.
2. Chose Bench | Create Database. The datastore is created, but so far there's nothing in it.
3. Bench | Load Data is now available. Select it. WebBench will create the sample table and fill them with data. The tables are large, so this may take ten minutes or more. When the status-line message reports "Data load complete", you can explore the newly-created tables using DataStore Explorer if you want. Close the datastore in the Explorer when you're done.

1. Optionally, you can select Bench | Options and set the following options.  To simplify your first run, you can let them take their default values.  Later, you can return to this dialog and see how changing the values affects the datastore's performance.
• Min Cache: sets the DataStore's minCacheSize property, which specifies the minimum number of blocks in the cache.  WebBench's default value of 512 blocks is fairly large.  On a machine with ample memory, a larger cache will improve performance, especially if there are many concurrent users.  JDataStore will allocate more cache blocks than the minimum if necessary.
• ReadOnly Delay:  sets the datastore's readOnlyTxDelay property. DataStore inherits this property from its parent, DataStoreConnection. A connection executing read-only transactions gets a snapshot of the datastore's data when it opens (or refreshes). This property specifies how long, in milliseconds, to hold onto a given snapshot so that it can be shared by other read-only connections.  WebBench's default is 6000 milliseconds.  Because read-only transactions sometimes have to "back out" work done by read-write transactions in order to see a consistent snapshot, large values of ReadOnlyDelay improve efficiency.
• Soft Commit:  sets the softCommit property of the TxManager object that WebBench will create to true or false.  By default, softCommit is false.  When it is true, JDataStore doesn't force every transaction to be written to disk.  This can improve performance.  Crash recovery is still guaranteed because the log file is still forced to disk.  However, transactions completed less than about a second before a crash may be lost.
2. Select Bench | Run and set these parameters:
• Transaction types: check or uncheck the ReadWrite and ReadOnly checkboxes to reflect the kind of transactions you want to execute. You can run only read/write transaction, only read-only transactions, or both kinds.
• Threads: for each transaction type you select, enter the number of connections that should execute these transactions.
• Time: enter the number of minutes the test should run. If you run both types of transactions, enter the same number for each.
• Backup: check this option to run a database backup along with the transactions.  Besides making a copy of the database, this has the effect of adding a thread that's performing long-running read transactions to the mix.
3. Click the Run button. The threads you requested are created, each with its own connection to the datastore, and begin running transactions. When the time you selected elapses, WebBench reports the number of transactions executed and the number of transactions per second.
4. Optionally, you can use DataStore Explorer to verify the integrity of the datastore after running the benchmark.

• WebBench can be extended to run against database servers other than JDataStore. If you've done this, enter the name of your WebBench "driver" class in the Driver entry on the Options dialog to run benchmarks against that database.
• To create a backup copy of a datastore, select File | Backup.
• To delete the datastore or other database currently named on the Options dialog, select Bench | Drop Database.
• WebBench displays a cumulative log of what it's done. If you want to clear it, right-click in the text and select Clear.

• How costly write transactions are compared to read transaction. Of course this depends on the complexity of the transaction, but in general a write transaction, which has to be logged in case the user decides to rollback, makes the database do more work than a read transaction does.
• How well the database accommodates a mix of read and write transactions. A write transaction often locates and update a small number of records. A read transaction, on the other hand, sometimes selects thousands of records - think of a report generator, for example. In WebBench, the ReadOnly transaction threads execute short transactions while running a backup during a benchmark test provides long-running read-only transactions.  JDataStore is designed to handle a mix of short write transaction and long-running read transactions efficiently.
• How many concurrent database users can you add before throughput begins to fall off? At first, additional users just take processing time that would be unused otherwise, so the total number of transactions goes up. Eventually, though, adding users causes so much competition for resources that throughput falls off. It's important to realize, however, that WebBench threads hammer on the JDataStore steadily; this is very different from having human users who many spend a long time working on their selected data before saving any changes back to the database. So four or five WebBench threads might simulate the activity of dozens or hundreds of actual users.

Keep in mind when comparing benchmark results that there is a element of randomness in the tests. Random numbers are used to determine which customer's record is selected, how many line items are inserted, whether or not a transaction is rolled back, and so on. As a result, one benchmark run might have to do more work than a second run with the same parameters. Also note that read/write transactions add records to several table, which might perturb the situation slightly.

Organization of WebBench's code
Application structure and UI: WebBenchApp and WebBenchFrame are the default JBuilder application files. WebBenchApp is the main executable file and WebBenchFrame is the UI. WebBenchFrame is designable, but because the UI is one big jTextArea, there's not much to see in the UI designer. However, you can open the menu designer and examine the menu's organization, properties, and event handlers. OptionsDialog is opened from Bench | Options and RunDialog is opened from Bench | Run.

The AppProperties class is responsible for storing and managing settings from the options dialogs. AppProperties knows how to save its data to disk (as WebBench.properties, in the java user directory) and reload it. At startup, WebBenchFrame calls AppProperties' static getAppProperties() method. This reloads previous settings, if any, and provides WebBenchFrame with any information about options that it needs. Other classes use AppProperties in the same way.

Event handlers and backup: WebBenchFrame's event handlers are where the application's UI and its database manipulation meet. The major menu commands are File | Backup, Bench | Load Data, and Bench | Run. The code in all three is organized in the same way. As an example, let's look at BackupDataItem_actionPerformed, the event handler for File | Backup. Associated with it is the class BackupThread, defined in WebBenchFrame.java. The event handler creates an instance of BackupThread and sets it running. BackupThread in turn creates a BackupData object which does the necessary work with datastores while BackupThread handles the status reporting in the WebBench UI.

The BackupData class is fairly simple. It gets the current datastore name from AppProperties, prepends "backup" to it to name the backup datastore, creates the backup datastore in the same directory (deleting the existing datastore of that name, if there is one), and uses DataStore's copyStreams() method to copy everything in the original datastore to the backup.

One other small class comes into play here. A transactional datastore consists of a single file for data, one or more log files, and two status files. FileList is a utility class that simplifies working with this set of files. BackupData and JDataStoreDriver use FileList.

Server-specific work: WebBench does most of its database work through SQL statements that should run on any database server. However, some operations, such as establishing the connection to the server and creating and dropping databases, are done differently by each server. Driver is an abstract class that encapsulates the server-specific work that WebBench needs to do. JDataStoreDriver is a concrete extension of Driver for JDataStore. Almost everything in WebBench that is specific to JDataStore is in JDataStoreDriver. With extensions of Driver for other database servers, WebBench could be used to compare their performance to JDataStore's.

In a few cases code outside of Driver and its subclasses is not generic:

1. Most of the options set on the Bench | Options dialog are datastore-specific.
2. File | Backup only works for JDataStore.
3. WebBench assumes that all servers support the fairly standard SQL that it uses. The SQL feature that's most likely to be problematic is the CURRENT_TIMESTAMP keyword, which not every server supports.

Loading tables: We've seen how WebBenchFrame's event handler for the File | Backup menu command works. Bench | Load Data is similar. The event handler creates and runs an instance of DataLoadThread, defined in WebBenchFrame.java, and sets it running. DataLoadThread handles the UI updating and creates a LoadData object to do the database work. The novelty here is that LoadData in turn instantiates another class of interest, Tx. Tx and LoadData both execute SQL statements. Tx's statements, which create tables and indexes, are hard-coded. LoadData generates data and inserts it into the tables using prepared statements and parameterized queries. LoadData intermixes calls to Tx with its own work. It stops running when it's done and thus returns control to the UI.

Running transactions: Bench | Run present the Run Options dialog. Assuming you close the dialog by clicking the Run button, the menu command's event handler starts a TxRunnerThread (as usual, this class is defined in WebBenchFrame.java), which creates and runs a TxRunner. Parameters to TxRunner's constructor tell it the number of connections to create for each type of transaction. It creates the appropriate ReadOnlyTx and/or ReadWriteTx objects, sets each as the target of a new thread, and starts the threads. When every transaction objects is ready to run transactions, TxRunner signals them all to start, waits for the specified time, stops the threads, closes the transaction objects, and reports the results.

ReadOnlyTx and ReadWriteTx extend Tx, which contains the methods for synchronizing transaction runners that they both need. Their work is in two parts: first, they create prepared statements for all the SQL statements they will run; then, when TxRunner tells them to start, they run transactions until told to stop. Like Tx and LoadData, ReadOnlyTx and ReadWriteTx do all their work using classes from the java.sql package such as Connection, PreparedStatement, and ResultSet. Because the focus here is on database operations rather than data manipulation in the application, the DataExpress component set is not used, except for isolated use of a few classes such as DataSetException.

ReadWriteTx inserts a new order record and a random number of line items for a randomly-chosen customer. The transaction runs a total of nine different SQL statements, including queries against customer, stock, and item tables among others and updates to stock and next-ID information. Not all transactions succeed; on a random basis, some are rolled back. ReadOnlyTx executes the same queries but omits the inserts and updates.
 

Download the WebBench source code (zip file format). To uncompress this file properly, make sure you preserve folder names.