What We Expect From Our Field Testers

By: Nick Hodges

Abstract: This article covers what it takes to be a good and valuable field tester for RAD Studio

We are almost always running a field test. During the development process for a new release, we are running a field test and updating the field test every couple of weeks with a new build. We start out with basically a rerun of the previous version and then slowly add the new features in as we go along. We solicit feedback from our testers and endeavor to listen to what they say. We fix bugs, make adjustments and refine features based on their feedback. Once the product is shipped, we almost always move into Update mode where we continue to improve quality. (Quality is the watchword around here. We’ve learned that lesson, believe me.)

A good field tester is like gold to us. Some of our field testers do amazing work. Some of them actually become de facto members of the team. Good testers are hard to find, and when we find one, we do our best to try to keep them happy. In the end, we wouldn’t be able to release a product without our field testers.

We have a lot of field testers, but there are certainly some that rise to the top like cream in milk. We usually try to reward these folks with free product and other goodies. We actually analyze data about who does what, and often give away free product to those who are very helpful.

So what makes a good field tester? What is it that we’d like to see from them? What do our top field testers do? Below are the things we like to see out of a field test. If you are or become a field tester, you can use this as a guide to become a good one, and maybe even earn a free copy of the product you test.

    Good field testers are our customers

Generally speaking, we want people who are committed to the product to field test it. We want people that have purchased and used the product in real world situations. We want them to try out their existing code and let us know how it works with the new version. We want field testers that have made a financial commitment to the product.

    Good field testers honor their Non-Disclosure Agreement

In order to participate on our field test, one has to sign a Non-Disclosure agreement. It almost goes without saying that a good field tester will honor that agreement. He won’t divulge the fact that he’s a field tester and he won’t talk publicly about it at all. He won’t share field test information with his friends. And he absolutely won’t post a field test build on a public website. And if he does break his NDA, he won’t be surprised to hear about it from us – maybe even from our legal department.

    Good field testers read all the information we give them

We work pretty hard to communicate with field testers. We provide introduction information, and all kinds of information about the field test, what is available, where it is, what needs testing etc. A good field tester will read all of that before asking questions. He will seek out answers and search the newsgroups for the information he’s looking for. He’ll ask only as a last resort, realizing that administering a field test is a busy job, and that he doesn’t want to burden the system unnecessarily.

    Good field testers submit bug reports

This is pretty obvious and naturally gets to the core of the issue. If field testers aren’t telling us what is wrong with the product, then why are they even field testers? A good field tester knows that a good bug report is clear and concise, and contains either a good code example or clean, clear, thorough steps that will reproduce a problem. A good, solid bug report is like a treasured jewel – you can’t have enough of them. A good field tester will also submit Automated Incident Reports to our bug database at every opportunity, filling out the form as completely as possible to give us every clue possible for tracking down those access violations. When you get right down to it, reporting bugs is the main job of a field tester.

    Good field testers communicate with us and other field testers

One of the most fun parts of being a field tester (I was a field tester for many years before joining the team) is discussions on the private forums that we run for our field testers. Here, field testers can directly interact with the developers, QA folks, and management team for the product. Bugs are discussed, feature implementations hashed over, and ideas are exchanged. It’s a lot of fun. A valuable field tester will be an active participant in these discussions. He will share ideas and fixes and thoughts with us and with other field testers.

    Good field testers take field test surveys

One of the things that we routinely do for each new field test build is to survey our field testers to see what they thought of the previous build. This allows us to build a trend line to see if things are improving. If the field tester view of the field test isn’t steadily improving, we start worrying. Therefore, field test build surveys are a critical part of our field test process. We use field test survey results as a key indicator that the product is ready to ship. A good field tester carefully and diligently fills out every field test survey as each new field test build is released.

    Good field testers volunteer for leadership roles

As part of the field test, we organize a group of folks that we call “Field Test Marshalls”. Every major area of the product gets a Field Test Marshall. The job of the Field Test Marshall is to scrub through all bugs that come in for a particular area and ensure that they are reproducible and that they are properly promoted from QualityCentral into our internal bug tracking system. This ensures that the best reports are seen and worked on and that our team doesn’t have to waste time on reports that aren’t valid. The best field testers volunteer for this job and work hard at it.

    Good field testers review, provide feedback on, and even write documentation

We need our field testers to be supporting the cause of good documentation. A good field tester will review the documentation on new features to ensure that it is correct and complete. Many field testers are given write permission on our Documentation Wiki, and a good field tester will take the time to help with the writing documentation by correcting and adding to what has already been written.

    Good field testers get on our case

The very best field testers are the guys who relentlessly follow up on their bugs and bother us when we let something slip through the cracks. We don’t mind – in fact we really like it – when a field tester pesters us in email or the newsgroups about something that needs to be done. We work hard to cover all the bases, but occasionally something does get missed, and field testers who pesters us in those cases are highly appreciated.

    Conclusion

So, if you want to be a field tester for RAD Studio, that is what you should be prepared to do. It’s not tough – just a touch of dedication and some hard – but fun – work. Field testers get an early look at new features and technology, and thus get a jump on learning what’s new. They get to input their thoughts and opinions directly to the development team. They get to have real influence over the future of the product. They get to help improve the quality of their favorite development tool. And finally, they get the undying gratitude of the entire RAD Studio team. All in all, not a bad deal.

Server Response from: ETNASC04