Borland publishes Plum Hall conformance test results for Borland C++Builder 4

By: John Ray Thomas

Abstract: This paper describes the importance of published conformance test results for the Borland C++ compiler.

This paper describes the importance of published test results for the Borland C++ compiler. According to the Plum Hall Validation Suite, Suite++(R)* "xvs99a", Borland C++Builder 4, Update1 is 93% compliant across chapters 2-16 of the ISO/IEC 14882:1998 specification.

    Why is this significant?

Followers of their respective religions look to their scriptures for guidance and understanding in their lives. The Catholic/Christians have their Bibles, the Muslims have their Koran, and the Buddhists have the Tripitaka. We, the C++ developers of the world, have the ISO/IEC 14882:1998. We turn to the C++ specification when we are confused by a C++ feature that won't compile. Often, however, we end up even more confused than if we just blindly trusted the compiler vendor. "Is the compiler right or am I misinterpreting the spec?", you may have heard yourself saying more than once. At which point you might use another C++ compiler. Then after hours of insecurity you may post the question to your favorite C++ language newsgroup. "bcc32.exe v4.5 gives me error XXX when I try to implement feature YYZ from Section 12.1.2. Visual C 6 and egcs 2.1.5 let me do it. Who's right? Help ME!!!".

Now that the ANSI/ISO C++ specification has been approved it is natural to think that we will not be plagued with misunderstandings and confusion again. Like the "Tao te Ching" we can consult the great ISO/IEC 14882:1998 for our answers. But like any great work it is subject to personal interpretation. And as any religion has its spiritual leaders to interpret and explain their "word" we should turn to the technical leaders in our field for this guidance. Now compiler vendors (often the technical leaders in that field) have to interpret this spec and implement it. They really need a sanity check. For a C++ compiler vendor this is where the Plum Hall test suite steps in.

Borland has been using the Plum Hall Validation Suites ( http://www.plumhall.com) for many years to test the compliance of our C and C++ compilers. There are three tests involved in using the Plum Hall C++ Validation Suite, Suite++(R). They are "conform", which test conformance to the ISO/IEC 14882:1998 (Chapters 2-16);"negtests", which test the compiler's ability to catch malformed constructs; and the "C-tests", which test the C++ compiler for allowing certain C-style constructs. The results we are publishing are from the "conform" test only. We believe that the "conform" test on it's own is sufficient to detail our conformance to the spec.

    The results are in

Go to http://ww6.borland.com/plumhall/phq.exe/query to view the results of the test suite. It is a CGI application completely written with Borland C++Builder 6. The form is pretty straightforward. First, select a major section of the ISO/IEC 14882:1998 specification, then choose the compiler (there are three available at this time) and lastly choose the results you would like to see (PASSED, FAILED, SKIPPED or All). Then click the Submit button.

You will be returned a table that displays the section, paragraph, sentence and sub-sentence of that feature of the spec that was tested. PASSED tests will be color-coded green, FAILED tests will be color-coded red and SKIPPED tests will be color-coded yellow . You need to own a copy of the ISO/IEC 14882:1998 specification (www.ansi.org) in order to look up the feature that the test refers to.

    How the tests were run

In the interest of scientific validity we encourage users to test our compiler against the Plum Hall test suite themselves. If there is any discrepancy we would be happy to be made aware of it. We tested Borland C++Builder 4 (Update1) with Plum Hall Suite++(R) version "xvs99a". The tests were run on a Dell Precision 410 (Pentium II-266, 128M RAM). The operating system was Windows NT 4.00.1381 (Service Pack 5). The test suite batch files were modified slightly (to point to our binaries and to set up environment variables) and were run in a 4NT shell. Some tests were skipped if they wouldn't compile and should be considered a failed test (an example of this is in Section 2). Some of the tests would not compile because we do not support alternate keywords (AND, OR,...) in this compiler version. These tests had to be skipped in order to batch process the test. These show up as a SKIPPED test in the results.

The results were output by the PlumHall test into a .out file and test numbers were marked as PASSED or FAILED. SKIPPED tests write a .clg file of the same name as the test. A C++Builder application was written to import this data into an InterBase table. Lastly, a CGI application was written with C++Builder 4 using WebBroker technology to access the database and to create the HTML table with the results.

    What's next?

We decided to make these results public because we believe a developer (especially a C++ developer) needs to know exactly what level of compliance their compiler vendor provides. This is crucial information when it is necessary to port code to other platforms or other compilers. In the near future we will provide the results for some of our older versions and of course our newer versions as we are committed to providing the most compliant compiler avaliable. We have no plans to publish the results from other vendors but we highly encourage them to do this themselves. I sincerely hope that you find this a useful service and I would love to get your comments/suggestions. Enjoy.

//jt
Borland C++Builder
Developer Relations Mgr.
jt@borland.com

*Suite++(R) is a trademark of Plum Hall Inc.

Server Response from: ETNASC04