A Little History on the Interbase Security Hole

In the midst of the furor over the discovery and subsequent announcement by CERT of the Interbase backdoor, the feeding frenzy at slashdot and various newsgroups, (including our own), and given the frigid relationship between Borland and the people asserting that Borland did not respond, I have published here a message posted to the newsgroup news://news.mers.com/mers.interbase.list by Charlie Caro, of Interbase R&D here at Borland. While both he and I are Borland employees, I think he gives the most accurate and even handed assessment of the situation and history I have seen to date.

He also provides technical information that to this point has not been available.

Download the fix right now

Before going any further, I think it's important to let you know that you can find further information and download the patch right now.

The Message

Here is his message. You can read news://news.mers.com/mers.interbase.list for replies.

The following response represents my personal opinions and not necessarily those of my employer, Borland/Inprise.

I would like to respond to the account of the security back door and doomsday function that has been described by senior members of the IBPhoenix/Firebird team. In particular, Jim Starkey's account and its posting to the IBDI Web sites has painted an inaccurate, irresponsible, and frankly slanderous, characterization of the current Borland RD team. He could have stuck to the technical aspects of the security breach but made a conscious decision to personally attack me and members of the Borland/InterBase RD team.

1) It is stated that:

"In July of 2000, Borland published the source code. It apparently didn't occur to the Borland engineers that inserted the back door in the first place and used it for other purposes during version 6 development that a security back door was sufficiently dangerous to call it to the attention of their management or a responsible adult."

A) None of the current Borland engineers inserted that back door. The security database was introduced in IBV3.3 (not IBV4 as reported by Starkey). We checked the IBV3.3 source code and there was no backdoor security access. This means that the blatant security hole was not part of the original design.

B) None of the current Borland engineers, except me, worked on IBV3.3. Leave them out of it; they didn't participate in the security design or implementation. I participated in design reviews in a peripheral capacity around 1992. The Borland RD team was introducing InterBase to the Windows platform as a Borland strategic initiative. The existing InterBase security at the time relied on trusted clients and domains but Windows 3.1 was naked; no host login, no Active Directory, Kerboros or NT Domains ... nothing. Like the original security scheme it replaced, it was not designed for wide open Internet database access by anarchic Windows clients.

C) At some point in IBV4.x, Borland splintered our development team to work on Windows NT SuperServer. I refered to this division as "SplinterBase"; it divided the team and took architectural control of the server design away from my responsibility. When we shipped IB4.0 on UNIX, the SplinterBase code was merged back to the main development tree. At that time, our internal source code tool did not mail, to members of the RD team, the source code change "diffs". If it did, the changes were so numerous, that it was checked in undetected. It's possible that the backdoor could have been introduced during this merge. There was no formal or informal source code review at Borland/InterBase or, for that matter, at ISC in the 1980's before that. These best practices weren't introduced until the late 1990's.

D) I can't be sure but I suspect the thread re-entrancy complexities with the SuperServer architecture led to the introduction of this security hole because it didn't exist before this. I can assure you that I didn't, nor did any current team member, introduce it. Nor was it mentioned in any design review that I recall.

2) From the IBDI Web site:

"Inexplicably, the Inprise R & D engineers who worked alongside Ann Harrison from February to July last year, as she managed the preparation of the IB 6 beta source code for open release, had never seen fit to document the exploits or even to alert Ann to their existence. At least one of those engineers was well aware of one of the vulnerabilities, since he used it to implement a new feature in version 6."

E) I don't know what V6 feature Jim claims uses the security back door. I suspect he is confusing the V6 garbage collector with the V3.3 sweep thread to the Apollo/DomainOS multi-client server. The V6 garbage collector doesn't use any security backdoor whatsoever. Someone modified the V3 sweep thread, which attaches a database, to use the LOCKSMITH username/password circa 1993-4. It doesn't have my coding fingerprint with respect to indentation (look at the while loops, Jim). I was in that area of the code to address thread safety issues using static buffers but I wasn't consciously focusing on the semantics of the code itself -- just the thread unsafety. It was similar to the attention paid to surrounding code when replacing character-by-character moves with memcpy(). If this is the basis of his argument, it doesn't hold water.

F) This leads to the last accusation that the current Borland RD team was aware of the security hole. We weren't. If we were, why would we have released the code? We did discuss with the July 2000 active management member and responsible adult about whether we should withold the password encryption and jrd/pwd.h file that held sensitive password salts. Neither the responsible adult nor the team was aware of the security backdoor it posed. We agreed together that we wouldn't replace that module with a stub file because we wanted the executables to match the sources.

G) Jim isn't privy to the organizational structure, dynamics and behavior at Borland. I have raised many protestations over technical decisions over the years. I don't always win and I have shared some of those defeats with the community. The Borland/InterBase QA raised the quality of InterBase far beyond where it stood a decade ago. In my opinion, it's a fair criticism to say that some of their procedures were inefficient and delayed releases in the 1990s. However, pre-Borland InterBase pretty much relied on localization of memory leaks and errors in the Classic architecture to bail them out. Today InterBase is a far more robust product because of Borland QA, notwithstanding that ill-advised UDF.

H) We put a quick fix to change the LOCKSMITH username/password because SourceForge would not take down the tree. This was to protect the installed customer base. It was never intended as the final fix; we followed up with a posted fix which eliminates any backdoor LOCKSMITH username/password.

Jim has impugned my reputation and the professionalism of my collegues. He has done it thru this public announcement based on heresay, innuendo and groundless facts. He has done it privately behind my back to my managers, to members of this community and customers that I have served. He called me last month to get the phone numbers of my director and Borland Sr VP to relay the CERT information. I gave him the phone numbers and later discovered that he called those individuals to denigrate and slander me. When I called to confront him about it, he said nothing to indicate that he was contrite or regretted the incident.

I suspect that Jim resents that I am still working for Borland -- that I should be a conscientious objector. I can't put my kids thru college or pay the mortgage with the source code accolades I would be paid with if I worked for IBPhoenix. If the community can't scrape together the money to release Jim's ODBC driver, how would it pay me?

To the community members, you will have to ask yourself whether his description is consistent with your first hand experience of my behavior and service. Each of us occupies a place in this community: to the party right is Borland, to the party left is IBPhoenix and in the middle are the majority who just want to use InterBase as a product. I, and the current team, have contributed much to the InterBase project over the years. I, and some others, insured its survival by moving to California to be part of Borland. I've argued passionately in Borland boardrooms with several chairmen/CEOs to keep InterBase alive. And I don't deserve this treatment from a community member.

I demand a public apology from Jim and a retraction, or a fair and balanced account, on the IBDI Web site. Otherwise, it's not reporting but self-serving propoganda.

You have everything to gain by making the apology. If not for yourself, then think of your partner -- there's no reason to tarnish her good reputation, by association, because of your bad behavior. I can forgive you and this community can forgive you. But if you can't do this, you will have severely diminished your leadership credibility within this community.

Regards,
Charlie Caro

Jim Starkey's response

On Saturday, Jan 13, 2001 this message was posted on news://news.mers.com/mers.interbase.list, with a title of "Retraction".

My statement that current members of Borland technical staff were aware of the back door when it was implemented was based on a conversation with an engineer employed by Borland at that time. He has since clarified that his comments were based on how he thought decision was reached and not how the decision was actually reached. Furthermore, Charlie Caro is correct that the sweep thread, which exploits the back door, is a separate mechanism from the V6 garbage collect thread, which does not.

There is no reason that I am aware of to believe that any of the present members of the Interbase development team had any involvement or knowledge of the back door mechanism.

I would like to appologize formally to Charlie Caro for a misunderstanding that I mistakingly propogated. I tend to think of Charlie as the person entrusted with the care and feeding of my baby; it is unfortunate that Borland has not delegated that responsibility and authority to Charlie or to any other individual.

The security back door was implemented long ago in a different era. Other than serving as an historical example of the dangers of security by obscurity, nothing useful can come from further investigation of who did what when. It happened, it came to light, and with coordination by CERT, both IBPhoenix and Borland have been able to offer fixes to the Interbase community.

Jim Starkey

I hope this helps to give you a little more background on the situation. Again, the important thing to note is that the backdoor has been closed in the patches already made available.

John Kaster, Borland Developer Relations