This Old Pipe

By: Randall Nagy

Abstract: Few appreciate how much modern computing relies upon pipes and streams. This article reviews how to apply these elegant IPC mechanisms to your console and GUI applications!

This Old Pipe

Forward - Part 1

As the first in a two part series, Randall Nagy examines the history and use of Inter Process Communications (IPC) using Pipes. Designed to increase the understanding of pipe and stream usage for the beginning, intermediate, and advanced developer alike, this series reviews pipe usage in the console (command line), graphical (VCL/CLX), networked (named pipes), and a mixed networked / console / GUI programming environments.

Pipes: An Introduction

Sooner or latter, every software developer encounters the same question: How to link two applications together? While answer to this question over the last three decades has spawned just about as many solutions as there are brands of breakfast cereal available at your local grocery store, over the years several alternatives have clearly left their mark on the software developer community. Various operating system conventions demonstrate the elegant power and infinite extensibility behind using pipes and streams.

The Pipe Usage Paradigm

Upon closer inspection, we discover that the "pipe" name has been well applied to this communication paradigm. Pipes can either be written to, read from, or both read and written to, much like pouring water into it's namesake, or "flowing" data into a file.

Figure 1 - Dual Process Write / Read Concept

Unlike when using a file however, when one is putting something into one side of a pipe (write), something else will need to be available to receive the contents of the pipe (read), at the other end. The inability to immediately connect a pipe to another process spawns the concept of a "blocking" (or blocked) pipe.

Other than the need to have a cooperating process, we now understand how simple using any piping paradigm can be.

Streams

Whenever a pipe is opened, lots of little pieces of data can be transmitted. Moreover, while some data might be of a known size and structure (often referred to as "messages" in the Microsoft world), others can be sent as a series of octets (bytes). Because the concept of a "transaction" would be difficult to apply to such a democratic flow of content, the term "stream" is often used to describe what a pipe can contain (years ago RJE called this "Class A Data").

Pipes Everywhere: DOS, WIN16, WIN32, POSIX

Unbeknownst to many new arrivals to the software development community, pipes and streams are available on just about every popular desktop operating system created. From Microsoft DOS and Windows 3.x, to WIN32 and POSIX (Linux / Unix) enabled systems, and beyond.

Not only are streams and pipes a ubiquitous fixture in most operating systems, but because C is the language that the most popular operating systems were written in, there are usually three types of streams available to console applications: standard input, standard output, and standard error.

As their names imply, the standard input stream "flows into" a stream-capable process. Conversely, both the standard output and error streams (also referred to as "devices") are data streams that can be produced by a pipe-enable application.

Figure 2 - Our Notational Convention

By way of illustration, the following can be used to route the standard output from a command to a file.

DOS / Windows

 DIR > DIRLST.TXT

POSIX

 ls -al > dirlist.asc

In a like manner you can use the "pipe fitting" to link the output of one command, to the input of another.

DOS / Windows

 DIR | MORE 

POSIX

 ls -al | more 

Some Notational Differences

-- So much for the similarities. Aside from re-directing standard output on Linux, Windows, and DOS, the notation for working with other types of streams have been know to diverge somewhat.

Win32 Console -v- Bourne Shell?

For example, given the following platform independent C++ Program. 

#include <iostream>
int main(int argc, char *argv[])
{
cout << "Program Name: " argv[0] << endl;
cerr << "And so says all of us..." << endl;
}
 

Tip: When using C, these three streams are called stdin, stdout, and stderr. They are included in <stdio.h>. In C++, these three streams are known as cin, cout, and cerr. They and are brought to you by <iostream.h> or <iostream>.

If you wanted the output capture the output for the standard error stream (or device), as well as the standard output, you would have to resort to.

On DOS / Windows

 DIR 1> DIRLST.TXT 2>&1

On POSIX

 ls -al > dirlist.asc 2>> dirlist.asc

Of course there are many variations of this theme of command-line re-direction -- Topics which we do not need to discuss at this point. Suffice us to mention here that when connecting to system utilities on DOS / Windows and POSIX Systems that there are very subtle differences between the notation required to re-direct console input, output, and errors. A fact that will become important when we discuss the use of system() and popen() in a GUI application in Part II of this series.

A Word on API Choices: While WIN32 has it's own native API to support pipes, Windows also has a good many POSIX system call wrappers, as well. Because WIN32 has both WIN32 and a subset of POSIX function calls available, when writing portable applications (for example, when using CLX) it is usually better to check to see if there is a POSIX equivalent before opting to use the native WIN32 API. We will be using both WIN32 and POSIX function calls in this article series.

Piping Streams to a Windows Application

Now for the fun part -- Lets demonstrate how pipes and streams can be used to "glue" console and windows applications together.

A Simple Pipefitting

By way of review, we have included a sample batch file and C++ application that can be used to demonstrate how to use a pipe to tie two applications together. Specifically, the task was to accept the recursive output from the DOS attrib command, pipe it to an application, and then use the application to extract the valid file names and display them on (you guessed it) the standard output.

Figure 3 - A More Demonstrative Diagram

The project "1_The3Amigos" contains the C++ code and batch file that demonstrates how to process a simple standard input over to a standard output stream. For those who are more interest in reading, than in compiling at the moment.

 
 //--------------------------------------------------------
#include <string>
#include <iostream>
.
.
.
//---------------------------------------------------------
using namespace std;
#pragma argsused
int main(int argc, char* argv[])
{
.
.
.
char buf[MAX_PATH]; 
string str;
while(cin)
   {
   cin.getline(&buf[0], MAX_PATH);
   if(!cin)
   break;
   str = buf;
   // Use the drive specifier to detect file names:
   size_t whence = str.find(':');
   if(whence != -1)
      cout << "File [" << &str[whence+1] << "]" << endl;
   }
.
.
.
}
//------------------------------------------------------ 
 

We can see that the stream cin is being used to read the pipe-fitted input --- Like writing to cout and cerr, leveraging cin is very simple -- Just read from the device like you would any other stream!

Once we are inside of the C++ realm of operations, using the standard input, output, and error devices are very much platform independent. This independence means that while written on WIN32, this particular console application (above) will work the same anywhere ANSI C++ and the Standard Template Library (STL) are sold.

Pipes or Lego's?

Of course the benefit of linking together several applications by using standard input and standard output itself presents us with a higher degree of tool integration and elegance. For example, the Common Gateway Interface (CGI) uses standard input and output mechanism to pass data to and from stream-enabled applications. Indeed, for Internet content providers who cannot afford their own dedicated host to manage their site (and whose ISP will not host the Java Virtual Machine (JVM)), CGI remains the only viable way to integrate rich, customized site content into most popular web servers. We will provide an example of portable, server-side CGI programming in Part II.

When the concept of pipes, data streams, and object factories are combined together, very little imagination is required to understand how, by simply interconnecting the input, output, and error streams of several applications, that both graphical and non-graphical tool applications can be more easily inter-connected. Indeed, as we hope to demonstrate in the second part of this article series, it is this very flexibility that drives the Internet and Web Services.

Figure 4 - Pipe Dreams!

The next demonstration will illustrate how pipes can be used to create software that may be more elegantly blended together.

A More Interesting Example

When it came time to implement a streamable connection into Microsoft Windows Applications, the concept of creating a flexible interface seems to have been forgotten by our friends in Redmond for a while. So much so that many of us had to resort to creating our own pipe-fittings to "glue" our standard input creating, and our graphical standard output consuming, applications together.

While somewhat convoluted, the process of overcoming the limitations of "pipe challenged" applications is not very different than what many operating systems do (and did) to implement basic pipelining capabilities.

Figure 5 - Tool-Enabling Pipe-Challenged Applications

The project "2_The4Amigos" contains the C++ code and batch file that demonstrates how a application can submit data to a new instance of a classical Microsoft Windows (WinMain) Application. As a bonus, the program also demonstrates how an application can start up another, as well.

Note: While the VCL framework is used throughout this article series, the techniques and IPC source code classes have been designed to be framework independent. They should work wherever ANSI C++, STL, and WIN32 are available.

Console -to- Windows Piping Explained

While the results of piping the output from a console application into the graphical world might be somewhat awe-inspiring, the underlying mechanisms that we use to make it all happen are certainly not. Just as we did in our first example, we first collected our data from stdin. Once collected, we allowed main() to place the data into a class (IpcClient) that was a descendant of a class called IpcData.

class IpcData
{
protected:
   vector<string> aData;
   ostream& write(ostream& os);
   istream& read(istream& is);
public:
   IpcData(size_t max = 100);
};

As the ultimate ancestor of both IpcClient and IpcServer, IpcData merely manages persistent data --- Data that all descendant classes will be interested in. Once the data are collected, we used some form of IpcData to abstract (generalize) and encapsulate (hide) what the client or server IPC mechanisms are going to do.

class IpcClient : public IpcData
{
public:
   IpcClient(void) {}
   bool AddParam(string& str);
   bool Submit(void);
};

After the data was collected by main() in Console2.EXE, the Submit() cheats a little by "batching" the data to the Windows GUI Application. In short, we use IpcClient to collect the data, rather than simply stream it to the standard output.

bool IpcClient::Submit(void)
{
ofstream ofs;
ofs.open("params.dat");
write(ofs);
ofs.close();
string sUrl = "GuiConsole.EXE";
if(int(ShellExecute(NULL, "open", sUrl.c_str(), "params.dat", NULL, SW_SHOWNORMAL)) <= 32)
   return false;
return true;
}
 

After the data are saved to a file (params.dat), we use WIN32's ShellExecute to start the reading process. We pass the name of the file as part of the command line to let the receiving Windows Application (GuiConsole.EXE) know where the data are.

.
.
.
// We included this form so we could access the member
// function in charge of managing the parameters:
#include "frmMain.h"
.
.
.
WINAPI WinMain(HINSTANCE, HINSTANCE, LPSTR psz, int)
{
string str = psz;
try
   {
   Application->Initialize();
   // ((Note that OnCreate happens below))
   Application->CreateForm(__classid(TForm1), &Form1);
   // Here is where we pass the command line ...
   Form1->Params(str);
   Application->Run();
   }
.
.
.
}
 

Then, once safely inside of the GUI application (After the framework's OnCreate() member function has both come and gone), we stream-in the data using the server side instance of our IPC encapsulation.

void __fastcall TForm1::Params(string& str)
{
if(server.LoadParams(str) == true)
   {
   size_t whence = 0L;
   do
      {
      whence = server.EnumParam(str, whence);
      if(whence)
          ListBox1->Items->Add(str.c_str());
      } while (whence);
   }
}

size_t IpcServer::EnumParam(string& str, size_t next)
{
// aData is simply a vector containing strings;
if(next >= aData.size())
   {
   str.erase();
   return 0L;
   }
str = aData[next];
return ++next;
}
 

While the implementation might make Rube Goldberg proud, it is a common exercise --- Very handy when wanting to perform one-shot (initialization) communication operations with another Windows application.

Here is what it looks like to the user.

Figure 6 - Linking Console Output to a Windows Application

So why all the fuss creating IpcData, IpcClient, and IpcServer classes? To foster a higher degree of encapsulation and re-use. Not only do these three classes simplify and document what we are going to use the class for, but they isolate and hide the implementation details of how data are saved and communicated.

This hiding provides us some latitude. By not disclosing how we are going to do the "grunt work" (i.e. store and send the data) we are free to switch to other IPC mechanisms at some future point in time. Because will be modifying how we communicate our data at several points in this article series, the IpcData, IpcClient, and IpcServer family tree will let you know where to look to understand what is going on behind the scenes. Of course, placing them all into a single file also allows you to more easily re-use my code (1).

So much for our single "smoke and mirrors" demonstration --- Let's move along to discussing how the pipe paradigm applies to robust and net-workable inter-process communications.

Attaching and Networking Pipes

When piping data to a pre-existing instance of an application, the data exchange problem becomes a little more complicated.

Figure 7- A Concurrent, Multi-Pipe Relationship

Because you will probably want to allow pipes to be attached and removed from your application at different times, we need to use a more provident piping scheme.

Tip: Prior to the advent of the WIN32 API, inter process communications in windows was a very messy affair. Check out an article that I wrote for BYTE in 1992 to see what creating even a simple "piping" ability used to be like before WIN32!

Enumerating Pipe Resources

While our task remains the same (i.e. communicate the information from the attrib command), our mandate is now to add the contents of several command sessions to our graphical application. In order to better support multiple, concurrent communication operations, we will update our IpcData, IpcClient, and IpcServer classes to support Microsoft's Named Pipes.

 
.
.
.
#include <windows.h>
#define PIPE_NAME "MajorAmigosAll"
class IpcData
{
protected:
   HANDLE hPipe;
   string _Qualify(const string& sPipeName);
public:
   IpcData(void);
   virtual ~IpcData(void);
};

class IpcServer : public IpcData
{
public:
   IpcServer(void);
   bool CreatePipe(const string& sPipeName = PIPE_NAME);
   bool WaitParam(string& str, size_t timeout = 0L);
};

class IpcClient : public IpcData
{
public:
   IpcClient(void) {}
   bool SendParam(const string& sData, const string& sPipeName = PIPE_NAME, size_t nTimeOut = 3000L);
};

A few things to note about our new class definition: First, the paradigm has been changed. As a true "server" application, GuiConsole.EXE is now expected to be up and running so the named pipe can be created.

bool IpcServer::CreatePipe(const string& sPipeName)
{
if(!hPipe)
   {
   string sFullPipeName = _Qualify(sPipeName);
   hPipe = CreateNamedPipe(
   sFullPipeName.c_str(), // pipe name
   PIPE_ACCESS_DUPLEX, // read/write access
   PIPE_TYPE_MESSAGE | // message type pipe
   PIPE_READMODE_MESSAGE | // message-read mode
   PIPE_WAIT, // blocking mode
   PIPE_UNLIMITED_INSTANCES, // max. instances
   BUFSIZE, // output buffer size
   BUFSIZE, // input buffer size
   PIPE_WAIT, // time-out
   NULL); // no security attribute
   }
if (hPipe == INVALID_HANDLE_VALUE)
   {
   hPipe = 0L;
   return false;
   }
return true;
}
 

Once the pipe is created, the server will listen for data on the pipe.

bool IpcServer::WaitParam(string& str, size_t timeout)
{
if(!hPipe)
   {
   str = "Error: Pipe not created.";
   return false;
   }
bool br = true;
// Wait for the client to connect.
if(!ConnectNamedPipe(hPipe, NULL))
   {
   br = false;
   str = "Error: No client.";
   CloseHandle(hPipe);
   hPipe = 0L;
   return br;
   }
CHAR chRequest[BUFSIZE+2];
::memset(chRequest, 0, BUFSIZE+2);
DWORD cbBytesRead, cbReplyBytes, cbWritten;
BOOL fSuccess;
// Read client requests from the pipe.
fSuccess = ReadFile(
   hPipe, // handle to pipe
   chRequest, // buffer to receive data
   BUFSIZE, // size of buffer
   &cbBytesRead, // number of bytes read
   NULL); // not overlapped I/O
if (! fSuccess || cbBytesRead == 0)
   {
   br = false;
   str = "Error: Pipe read error.";
   }
else
   {
   str = chRequest;
   // Write the reply to the pipe.
   fSuccess = WriteFile(
   hPipe, // handle to pipe
   "Okay!", // buffer to write from
   5, // number of bytes to write
   &cbWritten, // number of bytes written
   NULL); // not overlapped I/O
if (!fSuccess)
   {
   str = "Error: Pipe write error.";
   br = false;
   }
}
FlushFileBuffers(hPipe); // Flush to client
DisconnectNamedPipe(hPipe); // disconnect client
return br;
}
 

Next, as a true client application, Console2.EXE can now be run many times to pipe it's data from any folder to the GUI Application. As far as the client is concerned, all he (or she) needs to do is to check out the pipe and write the "streamed" parameter to it.

bool IpcClient::SendParam(const string& sData, const string& sPipeName, size_t nTimeOut)
{
DWORD cbBytesRead = 0L;
CHAR chResult[BUFSIZE+2];
::memset(chResult, 0, BUFSIZE+2);
string sFullPipeName = _Qualify(sPipeName);
return CallNamedPipe(
   sFullPipeName.c_str(),	// pointer to pipe name
   LPVOID(sData.c_str()),	// pointer to read buffer
   DWORD(sData.length()),	// size, in bytes, of read
   LPVOID(chResult), // pointer to write buffer
   DWORD(BUFSIZE),	 // size, in bytes, of write
   &cbBytesRead, // pointer to #bytes written
   DWORD(nTimeOut) // time-out in milliseconds
   );
}
 

Finally, note that I left the pipe-name user-specifiable, so you can more easily re-use these classes.

Tip: In order to increase the dramatic presentation of the data to GuiConsole.EXE, the rendition of IpcClient has a 1 second delay before it sends data to IpcServer. In a like manner, the RunMe.bat for the MajorAmigos project not only starts the GuiConsole.EXE for you, but also launches multiple instances of the client program, as well.

One more thing: Because we chose to allow the server application to "listen" (block) on the pipe, GuiConsole.EXE executes WaitParam() inside of a thread. While we could have chose to allow the server application to manage the pipe asynchronously (i.e. not block (wait) on the pipe), allowing a thread to block on a pipe provides better service to client applications.

void __fastcall ThisOldPipe_Thread::Execute()
{
string str;
if(server.CreatePipe() == false)
   {
   str = "Error: Unable to create named pipe "";
   str += PIPE_NAME;
   str += """;
   ListboxMessage(str);
   return;
   }
while(!Terminated)
   {
   if(server.WaitParam(str) == false)
      {
      str = "Error: Unable to read data on named pipe "";
      str + PIPE_NAME;
      str + """;
      ListboxMessage(str);
      return;
      }
   ListboxMessage(str); // What we have...
   }
}
//---------------------------------------------------------------------------
void ThisOldPipe_Thread::ListboxMessage(const string& str) 
{
while(Form1->pString)
   ;
Form1->pString = new string;
*(Form1->pString) = str;
Synchronize(Form1->DisplayParams);
}

For those who are used to the operation of Borland's TThread component, much of this probably looks familiar. While the interface could have been implemented in many other ways (using a WIN32 PostMessage, etc), the way we chose is easier for those new to the Windows API to understand. The thread allocates the string, while the main process consumes, frees, and resets the pointer to same whenever TThread::Syncronsze() calls the main form's DisplayParams member function. The result is that the input from a much larger number of processes (aka: "Amigos" herein) can be concurrently intercepted and displayed ... all by using a single thread.

Figure 8- Supporting the Concurrent, Multi-Pipe Relationship

The source code for 3_MajorAmigos is yours to use and experiment with. It cleanly illustrates how several applications can chat-it-up using Microsoft's unique Named Pipe API. Feel free to use (and re-use) it in good code!

In the next article we will discuss how to combine standard, anonymous, as well as review the more "net-workable" pipes. 

Tip: For those of you who have managed to read this far, you probably deserve Sainthood. Since canonization is somewhat beyond the mandate of Borland PSO (outside of granting Borland Product Certification, of course <g>), here is a little tip: Because the attrib command will accept sub-parameters, you can use it to search a directory tree. For example:

ATTRIB *.OBJ /S | MORE

This little known fact is useful when you need to search for files of a specific type... But here is the kicker --- When hooked up to our little tool here, you can quickly create command-line pipefittings to your own set of networkable utilities. Creations that can (for example) help you manage all of those .DOC, .OBJ, .BAK, and .~* files that we all know and love. While there are more efficient ways to search and manage the contents of your folders, there are few that are more fun --- or impressive!

About The Author

Randall Nagy is the Principal Trainer for Borland's Professional Services Organization. Randall has written for such trade publications as Byte Magazine, Internet Day, Network World, InfoWorld, End User Computing Management, Data Communications Management, Information Management, The Handbook of Local Area Networks, and the DataPro Reference Series. An avid computer enthusiast and professional since 1978, he has authored over six (6) modern commercial software titles, as well as a host of cross-platform and platform specific software applications.

 

(1) The information in this article, related source code and executable programs have been provided for educational purposes only. Use at your own risk.


Server Response from: ETNASC04