InterBase 6 Certification and Multi-Platform Test Automation

By: Conference Speaker

Abstract: This paper discusses the structure and ideology of InterBase QA and how our open source community can participate in development testing using TCS, the InterBase Test Control System.

This is the technical paper from a talk given at the 12th Annual Borland Developer's Conference
By Aaron Ruddick – Borland Software Corp.

Aaron Ruddick has worked on both the JBuilder and InterBase projects. Currently, he is a quality assurance engineer for the InterBase product group. One of his latest projects is refining our cross-platform testing and performance tools. Aarons hobbies are fishing and programming.

Note: No portion of this paper may be distributed or reproduced by any means, or in any form, without the author's and/or Borland's prior written permission.

Introduction

The InterBase QA strategy has been developed and passed on for over 15 years.  Today, with our open source community and various InterBase projects, we are now able to include the entire community in our process.  In this session, I will show you not only how our system currently works, but how you, as our InterBase partners, can participate in the process.

Overview of InterBase Development Cycle

We break our development cycle into five phases:
    Phase_0
             The purpose of Phase 0 is to evaluate product opportunities in a structured manner.
             It explains the product concept and assess the concept's fit with the company product line strategies.
    Phase_1
            Phase 1 is the fundamental building block in a product development process. The product is fully defined
            and the program is planned. It defines WHAT the product is.
    Phase_2
            Phase 2 focuses on refining and finalizing the definition of the product. This requires the completion of
            high-level and detailed designs and functional planning for Development, QA, Publications, Product Marketing,
            Marketing Communications, and Support.
    Phase_3
            All implementation work is completed, with the exception of documentation refinements, and bug fixing.
            At this time Product Marketing and Marketing Communications are finalizing the product introduction plans.
            The product is tested to verify it performs in accordance with the Product Specification (Phase 1).
    Phase_4
            Phase 4 validates the products readiness for release. At this time Media Control produces the pre-master
            and master media, for final testing. RTM marks the initiation of associated launch, Marketing
            Communications, and Support activities.
 

How the community participates in our Development Cycle

Our community is involved in every phase of development:
    Phase_0
            At the start of every development cycle we filter through hundreds of customer feature requests and bug reports.
            Currently, bugs and feature requests can be entered by going to: http://www.borland.com/devsupport/
 
    Phase_1
            This is where we finalize our plan.  Your proposed features, that have made it through our Bug Counsel and
            and Core Team, get written up in our "Product Definition".
    Phase_2
            In phase 2 we will take the "Product Definition" and create both Design, Functional, and Testing Specifications.
            In addition to QA's testing specs. is our Beta testing plan.  This is where you, as InterBase community, will get
            to see the product as it progresses.  Field testers have the advantage of using our new cutting edge technologies
            before the rest of the world.  Without your help QA would never find the hidden bugs that only "real world"
            programs and users will find.
    Phase_3
            Testing, testing, and more testing.  In phase 3, QA and Field testers do their best to break the product.  In addition to
            our Test Control System (TCS), we do rigorous manual and load testing.  With the help of our field testers we find,
            and R&D fixes, every bug we can get our hands on.  This process can be a long cycle because every time R&D fixes
            a new bug QA must fully retest the entire product.  And, as all developers know, no product is without bugs, so this
            QA cycle usually goes on until our director chains us in our offices and runs away to put the cd to market.
   Phase_4
            In truth, no product leaves Borland without the signature of every Core Team member.  This final phase is our
            "Development Hand-off".  The final cd is made and our developers are let out of their chains for one last round of testing.
            In the frenzy of final testing, every engineer we have tries his very best to, once again, break the product.  This is
            a dangerous time for our team.  At any moment we could be maimed by flying InterBase cd's and Microwave pizza box
            explosions.  However, after our director again intervenes, we take the final cd, thoroughly tested and fought over,
            and ship it off (RTM).
 

TCS for field testers

    What is tcs...
            TCS is InterBase's Test Control System.  TCS started some time before 1988 and has evolved for over 12 years.
            TCS is a test automation system that can be compiled and ran on virtually any operating system that can run shell scripts
            and InterBase.  All tests, and their results, are stored as BLOBs in InterBase.  When ever a test is run, TCS will compare
            the current result with the expected result and tell you if it passed or failed.

    How do I get tcs...
            0.  Download and install cvs:  http://www.cvshome.org/dev/codewindow.html
            1.  Set your system path to the InterBase/bin and your /cvs directory.
            2.  cvs -d:pserver:anonymous@cvs.ib-tests.sourceforge.net:/cvsroot/ib-tests login
            3.  cvs -z3 -d:pserver:anonymous@cvs.ib-tests.sourceforge.net:/cvsroot/ib-tests co [modulename]
            4.  To make your binary:  Go to the /tcs/tcs directory and follow the directions in the README file.
 

    How do I use tcs...
            When you pull down the "tcs" module, you will have a /doc directory.  Inside are six documents:
                0.  PORTABLE_TESTS:  Tips on Portable test writing in TCS
                1.  QA_NON_C_LANGUAGES:  Guidelines for Writing Portable TCS Language Tests
                2.  QA_TCS_QUICK_GUIDE:  TCS quick setup guide and TCS notes
                3.  QA_WRITING_TCS_TESTS:  Conventions for test writing in TCS
                4.  TCS_Q_AND_A:  Only answers two questions:
                    a.  Q: How does "Test Versioning" in TCS work?
                    b.  Q: So, which init record version do we use for a given TCS test?
                5.  VECTOR_RUN_DOC:  Explains the VECTORED TCS run method
            The 6 documents above do not provide a complete manual for TCS.  Currently the only real way to
            learn our system is to work with it.  I have provided a brief tutorial to get you started.
 

Summary

Now that our QA team consists of not only IB employees but the entire IB community, we are open and ready for your input.  Any ideas or thoughts on the structure of the QA cycle are very welcome.  We are ready and willing to add your ideas to our model.  You are always welcome to email me your on ideas or post to our opensource newsgroups.
 

Server Response from: ETNASC03