The Express Way to the Internet, Part 2

Last time, in The Express Way to the Internet, Part 1, I gave a quick overview of InternetExpress while en route from Houston, Texas back to Scotts Valley, California. This time, I'm on the Amtrak 6 am Metroliner from Washington DC's Union Station to New York City's Penn Station to show Delphi 5 in the Borland booth at InternetWorld in the Jacob K. Javits Convention Center.

Even as I write this introduction, I gladly interrupt myself for spectacular views of the sun rising over the mist hovering over the meadows and waterways I'm passing on the East Coast. If you have an opportunity to do so, I recommend the 6 am train on a fall day; for some parts of the trip, there is nothing but train tracks between you and the water, and you can't see the train tracks. It looks like you're riding on the surface of the water. (I'll try to keep myself focused enough on my notebook to get this article done before I arrive in NY.)

It's in the script

This time, I'm going to describe bit more of our JavaScript support for XML datapackets from MIDAS. Kurt Hansen, one of our R&D architects, wrote the original data packet technology for MIDAS, which is a very compact binary format. For Delphi 5, he added support for XML-formatted data packets in MIDAS. When we started working on InternetExpress, he wasn't happy with requiring Internet Explorer 5 just to get support for XML data packets, so he implemented XML data packet support with JavaScript. Because of this JavaScript, any browser that supports JavaScript and HTML 4 should work with our XML implementation. (If this wasn't enough extollation of Kurt's virtues, he's also a good volleyball player! I hope we won our match last night.)

As mentioned in the previous article, there are actually a few JavaScript files that implement our XML support. The one I'm going to focus on for this article is xmldb.js, which is specifically for handling MIDAS data packets and delta packets.

A package deal

Since coming to Borland, one of my recurring tasks has been explaining how the MIDAS datapacket works. Because XML is a verbose, text-based data packet, reading XML versions of MIDAS data packets is much easier than reading the binary version of the package. When Kurt was developing our XML support, he wrote some JavaScript routines that would format and display our XML data packets in an indented structure that is ideal for figuring out exactly what is in a data packet.

Although Kurt wrote it for debugging and testing purposes during the development of the JavaScript, it's actually a great tool to use while developing your own InternetExpress applications. Jim Tierney (also a good volleyball player -- I wonder if that's a prerequisite for MIDAS R&D work?) even created web components for producing two buttons you can attach to a data navigator you may embed in an HTML client. One displays the original XML data packet for a dataset, and the other shows the delta packet that will be sent back to the server when updates are applied from the client. If you install the InetXCustom components as I described last time, you will find these two buttons available for any data navigator you put into an HTML client.

Several of the sample HTML clients in InetXCenter use these buttons. The one I usually pick for discussing the data packets is the CustOrders Master Detail client. When you click the Show XML button, another browser window is opened, and there's a slight delay as the javascript formats the data packet for human-friendly display. When complete, it is properly indented with syntax highlighting for the XML tags. The sample output below only shows the formatting, not the syntax highlighting:

<DATAPACKET Version="2.0" > 
    <METADATA>
        <FIELDS>
            <FIELD attrname="CustNo" fieldtype="r8" />
            <FIELD attrname="Company" fieldtype="string" WIDTH="30" />
            <FIELD attrname="Addr1" fieldtype="string" WIDTH="30" />
            <FIELD attrname="Addr2" fieldtype="string" WIDTH="30" />
            <FIELD attrname="City" fieldtype="string" WIDTH="15" />
            <FIELD attrname="State" fieldtype="string" WIDTH="20" />
            <FIELD attrname="Zip" fieldtype="string" WIDTH="10" />
            <FIELD attrname="Country" fieldtype="string" WIDTH="20" />
            <FIELD attrname="Phone" fieldtype="string" WIDTH="15" />
            <FIELD attrname="FAX" fieldtype="string" WIDTH="15" />
            <FIELD attrname="TaxRate" fieldtype="r8" />
            <FIELD attrname="Contact" fieldtype="string" WIDTH="20" />
            <FIELD attrname="LastInvoiceDate" fieldtype="dateTime" />
            <FIELD attrname="CustOrderTable" fieldtype="nested" > 
                <FIELDS> 
                    <FIELD attrname="OrderNo" fieldtype="r8" />
                    <FIELD attrname="CustNo" fieldtype="r8" required="true" />
                    <FIELD attrname="SaleDate" fieldtype="dateTime" />
                    <FIELD attrname="ShipDate" fieldtype="dateTime" />
                    <FIELD attrname="EmpNo" fieldtype="i4" required="true" />
                    <FIELD attrname="ShipToContact" fieldtype="string" WIDTH="20" />
                    <FIELD attrname="ShipToAddr1" fieldtype="string" WIDTH="30" />
                    <FIELD attrname="ShipToAddr2" fieldtype="string" WIDTH="30" />
                    <FIELD attrname="ShipToCity" fieldtype="string" WIDTH="15" />
                    <FIELD attrname="ShipToState" fieldtype="string" WIDTH="20" />
                    <FIELD attrname="ShipToZip" fieldtype="string" WIDTH="10" />
                    <FIELD attrname="ShipToCountry" fieldtype="string" WIDTH="20" />
                    <FIELD attrname="ShipToPhone" fieldtype="string" WIDTH="15" />
                    <FIELD attrname="ShipVIA" fieldtype="string" WIDTH="7" />
                    <FIELD attrname="PO" fieldtype="string" WIDTH="15" />
                    <FIELD attrname="Terms" fieldtype="string" WIDTH="6" />
                    <FIELD attrname="PaymentMethod" fieldtype="string" WIDTH="7" />
                    <FIELD attrname="ItemsTotal" fieldtype="r8" SUBTYPE="Money" />
                    <FIELD attrname="TaxRate" fieldtype="r8" />
                    <FIELD attrname="Freight" fieldtype="r8" SUBTYPE="Money" />
                    <FIELD attrname="AmountPaid" fieldtype="r8" SUBTYPE="Money" />
                </FIELDS> 
                <PARAMS DEFAULT_ORDER="2" PRIMARY_KEY="1" LCID="1033" />
            </FIELD> 
        </FIELDS> 
        <PARAMS MD_FIELDLINKS="14 1 2" LCID="1033" />
    </METADATA> 
    <ROWDATA> 
        <ROW CustNo="1221" Company="Kauai Dive Shoppe" Addr1="4-976 Sugarloaf Hwy"
        Addr2="Suite 103" City="Kapaa Kauai" State="HI" Zip="94766-1234" Country="US"
        Phone="808-555-0269" FAX="808-555-0278" TaxRate="8.5" Contact="Erica Norman"
        LastInvoiceDate="19950202T01:05:03000" > 
            <CustOrderTable> 
                <ROWCustOrderTable OrderNo="1023" CustNo="1221" SaleDate="19880701"
                ShipDate="19880702" EmpNo="5" ShipVIA="UPS" Terms="Net 30"
                PaymentMethod="Check" ItemsTotal="4674" TaxRate="0" Freight="0"
                AmountPaid="4674" />
                <ROWCustOrderTable OrderNo="1076" CustNo="1221" SaleDate="19941216"
                ShipDate="19890426" EmpNo="9" ShipVIA="UPS" Terms="FOB"
                PaymentMethod="Visa" ItemsTotal="17781" TaxRate="0" Freight="0"
                AmountPaid="17781" />
                <ROWCustOrderTable OrderNo="1123" CustNo="1221" SaleDate="19930824"
                ShipDate="19930824" EmpNo="121" ShipVIA="UPS" Terms="Net 30"
                PaymentMethod="Check" ItemsTotal="13945" TaxRate="0" Freight="0"
                AmountPaid="13945" />
                <ROWCustOrderTable OrderNo="1169" CustNo="1221" SaleDate="19940706"
                ShipDate="19940706" EmpNo="12" ShipVIA="UPS" Terms="FOB"
                PaymentMethod="Credit" ItemsTotal="9471.950000000001" TaxRate="0"
                Freight="0" AmountPaid="9471.950000000001" />
                <ROWCustOrderTable OrderNo="1176" CustNo="1221" SaleDate="19940726"
                ShipDate="19940726" EmpNo="52" ShipVIA="UPS" Terms="FOB"
                PaymentMethod="Visa" ItemsTotal="4178.85" TaxRate="0" Freight="0"
                AmountPaid="4178.85" />
                <ROWCustOrderTable OrderNo="1269" CustNo="1221" SaleDate="19941216"
                ShipDate="19941216" EmpNo="28" ShipVIA="UPS" Terms="FOB"
                PaymentMethod="Credit" ItemsTotal="1400" TaxRate="0" Freight="0"
                AmountPaid="1400" />
            </CustOrderTable> 
        </ROW>
        ... (more customer rows, with embedded CustOrderTable rows)
    </ROWDATA>
</DATAPACKE>	

In the METADATA declaration of the data packet, all the field definitions for the data packet are defined. You will notice there is actually a nested FIELDS declaration that contains an additional set of field definitions. This is the detail table of the master/detail relationship, contained in a nested field type. In this example, the nested field is called CustOrderTable. This is important because of the actual row data that gets transferred with the data packet.

Immediately following the definition of the data packet structure is the actual data from the dataset. The declaration begins with ROWDATA, then immediately jumps into the definition of the data for each row. You can see that the nested dataset can contain multiple rows in its own section. Because CustOrderTable is defined in the data structure for the data packet, the JavaScript knows what fields to expect from its nested rows.

You may be wondering why it was necessary to actually declare a field for the nested data rows. If you are, good question! The answer is quite reasonable; you need to name the nested structure because you may have more than one type of nested dataset in your data packet. For example, you may have items contained inside the orders dataset (perhaps named OrderItemsTable) and a history list of communications with that customer that are another nested dataset for the customer table itself (perhaps named CustHistTable).

As you can readily see, you can quickly produce a very large datapacket by having multiple levels of Master/Detail relationships. If you design your client interface so only one customer record is retrieved at a time, having multiple nested datasets may result in a highly usable interface. If your client interface allows multiple customer records to be retrieved at once, and they contain a number of orders, and these orders contain a number of items, your data packet is going to get large and the person using the client application may have to wait a little while for the data packet to load and get rendered.

InternetExpress gives you the power to create nested datasets for good or evil. Please use it for good.

The flow of the delta

The entire time the end user is making changes to the data, the JavaScript is keeping track of those changes automatically. In fact, the user can even undo the changes to the active row since they last applied updates to the server just by clicking the Undo button (if you make that button available to them in their client). Undoing their changes will set the data back to the values originally retrieved from the server.

When changes are made to the data packet, they are tracked in a delta packet that can subsequently be sent to the web server, which in turns uses the XML broker to send that data packet back to the MIDAS server, which will resolve those changes back into the persistent store from which the data originally came (or somewhere else, if you so desire). As you would expect, this data packet looks much like the original data packet, but it's usually a fair amount smaller, since only changed data is sent back.

Here is a sample delta packet ready to be sent back to the server. It's been displayed by pressing the Show Delta button.

<DATAPACKET Version="2.0" > 
    <METADATA>
        <FIELDS>
            <FIELD attrname="CustNo" fieldtype="r8" />
            <FIELD attrname="Company" fieldtype="string" WIDTH="30" />
            <FIELD attrname="Addr1" fieldtype="string" WIDTH="30" />
            <FIELD attrname="Addr2" fieldtype="string" WIDTH="30" />
            <FIELD attrname="City" fieldtype="string" WIDTH="15" />
            <FIELD attrname="State" fieldtype="string" WIDTH="20" />
            <FIELD attrname="Zip" fieldtype="string" WIDTH="10" />
            <FIELD attrname="Country" fieldtype="string" WIDTH="20" />
            <FIELD attrname="Phone" fieldtype="string" WIDTH="15" />
            <FIELD attrname="FAX" fieldtype="string" WIDTH="15" />
            <FIELD attrname="TaxRate" fieldtype="r8" />
            <FIELD attrname="Contact" fieldtype="string" WIDTH="20" />
            <FIELD attrname="LastInvoiceDate" fieldtype="dateTime" />
            <FIELD attrname="CustOrderTable" fieldtype="nested" > 
                <FIELDS> 
                    <FIELD attrname="OrderNo" fieldtype="r8" />
                    <FIELD attrname="CustNo" fieldtype="r8" required="true" />
                    <FIELD attrname="SaleDate" fieldtype="dateTime" />
                    <FIELD attrname="ShipDate" fieldtype="dateTime" />
                    <FIELD attrname="EmpNo" fieldtype="i4" required="true" />
                    <FIELD attrname="ShipToContact" fieldtype="string" WIDTH="20" />
                    <FIELD attrname="ShipToAddr1" fieldtype="string" WIDTH="30" />
                    <FIELD attrname="ShipToAddr2" fieldtype="string" WIDTH="30" />
                    <FIELD attrname="ShipToCity" fieldtype="string" WIDTH="15" />
                    <FIELD attrname="ShipToState" fieldtype="string" WIDTH="20" />
                    <FIELD attrname="ShipToZip" fieldtype="string" WIDTH="10" />
                    <FIELD attrname="ShipToCountry" fieldtype="string" WIDTH="20" />
                    <FIELD attrname="ShipToPhone" fieldtype="string" WIDTH="15" />
                    <FIELD attrname="ShipVIA" fieldtype="string" WIDTH="7" />
                    <FIELD attrname="PO" fieldtype="string" WIDTH="15" />
                    <FIELD attrname="Terms" fieldtype="string" WIDTH="6" />
                    <FIELD attrname="PaymentMethod" fieldtype="string" WIDTH="7" />
                    <FIELD attrname="ItemsTotal" fieldtype="r8" SUBTYPE="Money" />
                    <FIELD attrname="TaxRate" fieldtype="r8" />
                    <FIELD attrname="Freight" fieldtype="r8" SUBTYPE="Money" />
                    <FIELD attrname="AmountPaid" fieldtype="r8" SUBTYPE="Money" />
                </FIELDS> 
                <PARAMS DEFAULT_ORDER="2" PRIMARY_KEY="1" LCID="1033" />
            </FIELD> 
        </FIELDS> 
        <PARAMS MD_FIELDLINKS="14 1 2" LCID="1033" DATASET_DELTA="1"/>
    </METADATA> 
    <ROWDATA> 
        <ROW CustNo="9841" Company="Neptune's Trident Supply" Addr1="PO Box 129"
        City="Negril" State="Jamaica" Country="West Indies" Phone="778-897-3546"
        FAX="778-897-6643" TaxRate="0" Contact="Louise Franks"
        LastInvoiceDate="19951101T06:32:05000" RowState="1" > 
            <CustOrderTable/>
        </ROW> 
        <ROW Addr1="PO Box 129-A" RowState="8" > 
            <CustOrderTable/>
        </ROW> 
        <ROW CustNo="9841" Company="Neptune's Trident Supply" Addr1="PO Box 129-A"
        City="Negril" State="Jamaica" Country="West Indies" Phone="778-897-3546"
        FAX="778-897-6643" TaxRate="0" Contact="Louise Franks"
        LastInvoiceDate="19951101T06:32:05000" RowState="64" > 
            <CustOrderTable> 
                <ROWCustOrderTable OrderNo="1045" CustNo="9841" SaleDate="19881016"
                ShipDate="19881017" EmpNo="36" ShipVIA="FedEx" Terms="FOB"
                PaymentMethod="Credit" ItemsTotal="787.8" TaxRate="0" Freight="0"
                AmountPaid="787.8" RowState="1" />
                <ROWCustOrderTable AmountPaid="799.9" RowState="8" />
            </CustOrderTable> 
        </ROW> 
        <ROW CustNo="9841" Company="Neptune's Trident Supply" Addr1="PO Box 129-A"
        City="Negril" State="Jamaica" Country="West Indies" Phone="778-897-3546"
        FAX="778-897-6643" TaxRate="0" Contact="Louise Franks"
        LastInvoiceDate="19951101T06:32:05000" RowState="64" > 
            <CustOrderTable> 
                <ROWCustOrderTable OrderNo="1145" CustNo="9841" SaleDate="19940117"
                ShipDate="19940117" EmpNo="144" ShipVIA="FedEx" Terms="FOB"
                PaymentMethod="Credit" ItemsTotal="4229.8" TaxRate="0" Freight="0"
                AmountPaid="4229.8" RowState="1" />
                <ROWCustOrderTable AmountPaid="4229.84" RowState="8" />
            </CustOrderTable> 
        </ROW> 
    </ROWDATA> 
</DATAPACKET> 

The METADATA section is actually identical to the previous data packet, except for the additional parameter DATASET_DELTA = "1" in the second PARAMS section of the metadata. This value indicates that this is a delta packet. There are some additional rows of data that follow the metadata. These rows identify the changes to the data since it was sent by the server. The following table describes the changes I made to the data so you can see how the delta packet retains only the information needed for the MIDAS server to correctly handle conflict resolutions while still providing the information needed to make the data changes.

For every change to a data row, the original contents of the row is passed back to the server for two reasons. First, if a primary key is not available for that row, the row will be matched based on all these values. Second, it makes graceful conflict resolution possible. For example, you may have grabbed a set of data from the server in the morning, and made changes to it over the course of a day. While you were making the changes, someone else also updated one of the same records on the server, so your original record is out of date. If your original row is not passed back to the server, it is difficult to determine the origin of the conflicts in the data.

Return to sender

By passing back the original contents of the row you received to the server, an intelligent dialog can automatically be provided to the user by hooking up a TReconcilePageProducer component to your XML Broker. This dialog shows:

• The data currently on the server
• The data you originally retrieved
• The changes you made to your original data

You can then choose what to do with each specific conflict, either leaving it alone, applying your modifications, editing it, and so forth.

The second part of the packet for the data row is the actual change to the data. In this section, only the changed fields are listed, along with a RowState flag indicating what to do with the values in that row. If multiple fields are changed for the same record, they will be listed in this section of the data packet.

The following table lists the RowState values and what they mean.

Also, notice that the value of the data packet for the customer row changes after the first delta data row, since the data will have been changed by then.

Ready for the big time?

The train just passed through Newark, so I've got about 10 minutes to before the train reaches Penn Station. I'll wrap it up here, and next time we'll talk a bit about how the web components work. Until next time, you can ask questions in the MIDAS newsgroup. I hope to see you at InternetWorld!