Using blocking sockets inside of a thread, and simulating HTTP requests

By: Corbin Dunn

Abstract: How to create a blocking TClientSocket inside of a TThread and get data from an web server

Using blocking sockets inside of a thread, and simulating HTTP requests
By Corbin Dunn
Delphi Developer Support

This article covers how to send an HTTP request to a web server, and work with the response. All of the data transfer is done with a blocking socket inside of a thread.

Download this example from Borland CodeCentral

You may have a couple questions at this point. Such as, what is a blocking socket and why would I use it?

A blocking function call is one that doesn't return until the function is totally complete. In other words, all reading and writing occurs synchronously. For sockets, this means that a call to the recv function will not return until all the data has been received. If a socket is set to non-blocking, a call to recv will return very quickly, returning the data that is ready at that time of the call. Delphi's TClientSocket is by default a non-blocking socket because the ClientType property set to ctNonBlocking. With the non-blocking socket you use the OnRead and OnWrite events to read and write to the socket. Basically, Windows sends your application a message when data is ready to read or write to the socket. If you set the ClientType to ctBlocking, then your application will freeze until the data has been read. The way to prevent this from happening is to put the TClientSocket in a thread.

If you go to https://www.census.gov/ftp/pub/genealogy/names/ you will notice that there are a couple of text files named dist.male.first and dist.female.first. They contain a list of the most frequent names in the 1990 Census. This example will send an HTTP request to www.census.gov and get on of those two text files. It will then process the data and place it into a TListView. You can even search for your name.

Here is a screen shot of the example we will be working on:

If you want to just dig apart the code yourself, then download it now. Otherwise, here are some step by step instructions on how I created the example.

  1. First create a new application in Delphi. Rename the form to "MainForm" and save it in MainFrm.pas. Save the project as "YourName.dpr."
  2. Next, add ScktComp to the uses list of MainFrm.pas.
  3. We need to create a TThread that does the blocking socket work. Under the type keyword, but above the MainForm class, add:
      TSocketThread = class(TThread)
      private
        FClientSocket: TClientSocket;
        FData: string;
        FLastException: Exception;
        FGender: string;
        procedure SetGender(const Value: string);
      protected
        procedure Execute; override;
        procedure HandleThreadException;
        procedure ThreadDone;
      public
        constructor Create;
        destructor Destroy; override;
        property Gender: string read FGender write SetGender;
      end;
    Press Ctrl-Shift-C to complete the class.
  4. In the implementation section, add the following code for your constructor and destructor:
    constructor TSocketThread.Create;
    begin
      inherited Create(True); // Intially suspended
      FreeOnTerminate := True;
      FClientSocket := TClientSocket.Create(nil);
      FClientSocket.Host := 'www.census.gov';
      FClientSocket.Port := 80; // HTTP port
      FClientSocket.ClientType := ctBlocking; // Blocking, so it is in a thread
    end;
    
    destructor TSocketThread.Destroy;
    begin
      FClientSocket.Free;
      inherited;
    end;
    The constructor will simply create the TClientSocket that will connect to www.census.gov on the HTTP port (80), with a blocking socket. The destructor cleans up the allocated memory.
  5. Now, standard TThread programming means that we will put our threaded code inside of the Execute procedure. To use a blocking socket, we have to use the TWinSocketStream class. The TWinSocketStream allows us to easily write to the socket and allow a timeout in case something goes wrong. Our Execute procedure will simply connect to the server, create a TWinSocketStream, and then read data until zero bytes are returned from a read, or the socket is closed. Zero bytes will be returned from a read when there is no more data available to read. Also, the HTTP port will be closed by the server when no more data is available. Here is the code:
    procedure TSocketThread.Execute;
    var
      Buffer: array[0..1024-1] of Char;
      SockStream: TWinSocketStream;
      RequestString: string;
    begin
      try
        FClientSocket.Active := True;
        FData := '';
        FillChar(Buffer, SizeOf(Buffer), #0);
    
        // Create the socket stream with a 1 minute timeout
        SockStream := TWinSocketStream.Create(FClientSocket.Socket, 60000);
        try
          RequestString := Format('GET /ftp/pub/genealogy/names/dist.%s.first HTTP/1.0'#13#10#13#10,
            [FGender]);
          // Send the HTTP request
          SockStream.Write(RequestString[1], Length(RequestString));
          // Read the response in 1024 chunks
          while (SockStream.Read(Buffer, SizeOf(Buffer)) <> 0) do
          begin
            FData := FData + Buffer;
            FillChar(Buffer, SizeOf(Buffer), #0);
            // Check for termination of the thread or closure of the socket
            if Terminated or not FClientSocket.Active then
              Exit;
          end;
        finally
          SockStream.Free;
          if FClientSocket.Active then
            FClientSocket.Active := False;
        end;
      except
        on E: Exception do
        begin
          if not(ExceptObject is EAbort) then
          begin
            FLastException := E;
            Synchronize(HandleThreadException);
          end;
        end;
      end;
      // Tell the main form that we are done
      Synchronize(ThreadDone);
    end;
    
    procedure TSocketThread.HandleThreadException;
    begin
      Application.ShowException(FLastException);  
    end;
    
    First of all, notice how I wrap all the operations in a try..except block. Any exceptions are handled via a call to Synchronize to prevent any multi-threaded problems from happening. For more information about how and why to do this read my article How to handle exceptions in TThread objects.

    You will probably also notice my use of the FGender private data field. It contains a string of either "male" or "female" so that we can access the right file on the server depending on what the user selected. It is set via the Gender property's write procedure:
    procedure TSocketThread.SetGender(const Value: string);
    begin
      if (Value = 'male') or (Value = 'female') then
        FGender := Value
      else
        raise Exception.CreateFmt('Unknown gender (%s) to search for',
          [Value]);
    end;
  6. The last thing that the TSocketThread's Execute procedure does is call ThreadDone inside of Synchronize to make it thread safe. The code is as follows:
    procedure TSocketThread.ThreadDone;
    begin
      // The VCL is not thread safe. Accessing MainForm from inside
      // a thread would be bad, unless if it is inside a procedure
      // that was called via Synchronize.
      MainForm.ProcessData(FData);
    end;
  7. ProcessData is a function that we will add to the MainForm.
  8. Now, the next thing to do is to set up MainForm to use the thread class that we just created. First, lay out some components on the form to make it look like the screen shot above.

    Here are the components and names that I used:
    Component Class Name Caption or Text
    TRadioButton rdbtnMaleNames Male First Names
    TRadioButton rdbtnFemaleNames Female First Names
    TButton btnGetData Get Data From census.gov
    TLabel lblStatus lblStatus (set at runtime)
    TLabel lblSearch Search:
    TEdit edtSearch CORBIN
    TButton btnFindIt Find It
    TListView lstvwNames n/a
    Set the ViewStyle property for lstvwNames to be vsReport.
    Double click on lstvwNames and add four columns: Name, Percent of Population, Total Percent, and Rank.
  9. Double click on btnGetData and add the following code to create an instance of our TSocketThread:
    procedure TMainForm.btnGetDataClick(Sender: TObject);
    begin
      btnGetData.Enabled := False;
      // Create a thread
      with TSocketThread.Create do
      begin
        if rdbtnMaleNames.Checked then
          Gender := 'male'
        else
          Gender := 'female';
        Resume;
      end;
    end;
    Remeber that rdbtnMaleNames is the name of the TRadioButton on the form.
  10. The next thing we need to do is process the data returned. The thread will call MainForm.ProcessData(FData) after it has received all its data. We need to add this procedure to the MainForm's class interface. Since the TSocketThread is declared in the same unit, we can put the ProcessData procedure in the private section of TMainForm. Also add a TStringList to store some extra data later on:
      private
        FNamesList: TStringList;
        procedure ProcessData(Data: string);
    Press Ctrl-Shift-C to complete the class.
  11. HTTP responses will have a header that we don't really care about. The start of data is immediately following two carriage-return newline pairs (ASCII #13#10). I simply use Pos to find this location and start processing the data after it:
    procedure TMainForm.ProcessData(Data: string);
    var
      DataStart: Integer;
      CurrentPos: PChar;
      Name: string;
      ListItem: TListItem;
    
      function ReadNextToken: string;
      begin
        Result := '';
        // All tokens are delimited by whitespace.
        // First, pass over any whitespace at the start.
        while (CurrentPos^ in [#10, #13, ' ']) do
          Inc(CurrentPos);
    
        // Read in characters until we hit some more whitespace
        while not (CurrentPos^ in [' ', #0, #10, #13]) do
        begin
          Result := Result + CurrentPos^;
          Inc(CurrentPos);
        end;
      end;
      
    begin
      // This is a very simple parser.
      
      // First, we must skip the response header (it will have 2 CRLF pairs before
      // the start of the data).
    
      DataStart := Pos(#13#10#13#10, Data);
      if DataStart = 0 then
        raise Exception.Create('Could not find the data in the HTTP response');
    
      // lstvwNames is the name of the TListView for displaying the data.
      lstvwNames.Items.BeginUpdate;
      try
        lstvwNames.Items.Clear;
        // FNamesList is a TStringList used to keep a sorted list of the names.
        // This way, we can do a quick locate on it.
        FNamesList.Clear;
    
        // Start past the 2 CRLF's (4 characters) with a pointer to that character
        CurrentPos := PChar(@Data[DataStart + 4]);
        while (CurrentPos <> nil) and (CurrentPos^ <> #0) do
        begin
          Name := ReadNextToken;
          if Name <> '' then
          begin
            // Create a list item
            ListItem := lstvwNames.Items.Add;
            ListItem.Caption := Name;
            ListItem.SubItems.Add(ReadNextToken);
            ListItem.SubItems.Add(ReadNextToken);
            ListItem.SubItems.Add(ReadNextToken);
            // Add the name to the internal list of names so we can easily
            // search for a name.
            FNamesList.Add(Name);
            // Keep a pointer to the item.
            FNamesList.Objects[FNamesList.Count - 1] := ListItem;        
            lblStatus.Caption := 'Processing the data...';
            Application.ProcessMessages;
          end
          else
            Break;
        end;
      finally
        lstvwNames.Items.EndUpdate;
      end;
      // Sort the internal list of names
      FNamesList.Sort;
      lblStatus.Caption := 'Done';
      btnGetData.Enabled := True;
    end;
  12. You may have noticed that we use FNamesList without ever creating it. So, in the OnCreate and OnDestroy events for TMainForm add the following code:
    procedure TMainForm.FormCreate(Sender: TObject);
    begin
      lblStatus.Caption := '';
      FNamesList := TStringList.Create;
    end;
    
    procedure TMainForm.FormDestroy(Sender: TObject);
    begin
      FNamesList.Free;
    end;
  13. The last thing to do is to allow the user to search for their name. Double click on btnFindIt and add the following code:
    procedure TMainForm.btnFindItClick(Sender: TObject);
    var
      Index: Integer;
    begin
      Index := FNamesList.IndexOf(UpperCase(edtSearch.Text));
      if (Index >= 0) then
      begin
        lstvwNames.Selected := TListItem(FNamesList.Objects[Index]);
        lstvwNames.Selected.MakeVisible(False);
        lstvwNames.SetFocus;
      end
      else
        raise Exception.Create('Could not find ' + edtSearch.Text);
    end;
    Now the reason for having FNamesList becomes apparent!
  14. You should be able to compile the application and have it work. If not, then just download the sample project.

That's it! If I made any errors, please email me.


Server Response from: ETNASC01