Indy and HTTPS Post

I'm back to work on the HTTPS Post issue with DHL's ShipIt XML API. The problem I'm having is when I use Indy to post the xml via an HTTPS post, the response is that the file is malformed. I should mention that I'm using a relatively recent version of Indy 10. If I use FF (if you don't have it, check out the Poster add-on, let's you play with the HTTP functions using FF's libraries) it works fine. The first problem is that it's a little hard to see what FF is actually doing that is different from what Indy is doing. Enter Fiddler, stage right. Fiddler is an http proxy that captures that http information and lets you manipulate it or display the raw headers. So, pointing FF to the localhost proxy on port 8888, the headers I get are:

POST /apilandingtest.asp HTTP/1.1
Host: ecommerce.airborne.com
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.0; en-US; rv:1.9.0.1) Gecko/2008070208 Firefox/3.0.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7
Keep-Alive: 300
Connection: keep-alive
Content-Type: text/xml
Content-Length: 1509
Pragma: no-cache
Cache-Control: no-cache

Applying a TIdLogFile to the Indy HTTP IO Intercept provides Indy's headers of:

POST /apilandingtest.asp HTTP/1.0
Content-Type: multipart/form-data; boundary=--------072108153606968
Content-Length: 1662
Host: ecommerce.airborne.com:443
Accept: text/html, */*
Accept-Encoding: identity
User-Agent: Mozilla/3.0 (compatible; Indy Library)

----------072108153606968
Content-Disposition: form-data; name="Shipment"; filename="dhl.xml"
Content-Type: text/xml

Obviously, not the same. A large part of this is because I'm using the TidMultiPartFormData class to encapsulate the data. Changing over to just a plain TStringList results in headers of :

POST /apilandingtest.asp HTTP/1.0
Content-Type: text/xml
Content-Length: 1471
Host: ecommerce.airborne.com:443
Accept: text/html, */*
Accept-Encoding: identity
User-Agent: Mozilla/3.0 (compatible; Indy Library)

but DHL still won't take the file, responding again that the XML is malformed. Looking into the log a little more, it appears as if the HTTP Post converted all CR/LF pairs into the symbol &. Sure enough, in Indy's code, there is a converter that converted CR/LF to &. Stripping CR/LF pairs out of the raw XML produces a workable solution that passes DHL's IIS server and returns the correct response. Obviously, that's a bit of a poor solution since it leaves the XML mashed together and I tend to use a text editor to review my XML data more often than an XML editor. So, with a short bit of preprocess work that converts CR/LF pairs to spaces, I now have a working Indy SSL HTTPS Post routine that DHL will accept. I've posted the code below so you won't have to go digging all over creation for a sample of Indy 10's HTTPS Post. Enjoy!

const
   CR = #13;
   LF = #10;
   EOL = CR+LF;
   xmlEOL = EOL;

type
  // this is a descendant of TidLogFile, it will create a plain text file with
  // information about the transfer session
  TlogFile = class(TidLogFile)
  protected
    procedure LogReceivedData(const AText, AData: string); override;
    procedure LogSentData(const AText, AData: string); override;
    procedure LogStatus(const AText: string); override;
  public
    procedure LogWriteString(const AText: string); override;
    class function buildLogLine(data, prefix: string) : string;
  end;

// this ensures the output of error and debug logs are in the same format, regardless of source
class function TlogFile.buildLogLine(data, prefix: string) : string;
begin

  data := StringReplace(data, EOL, RSLogEOL, [rfReplaceAll]);
  data := StringReplace(data, CR,  RSLogCR,  [rfReplaceAll]);
  data := StringReplace(data, LF,  RSLogLF,  [rfReplaceAll]);

  result := FormatDateTime('yy/mm/dd hh:nn:ss', now) + ' ';
  if (prefix <> '') then
    result := result + prefix + ' ';
  result := result + data + EOL;
end;

// ---------------------------------------------------------------------------

procedure TlogFile.LogReceivedData(const AText, AData: string);
begin
  // ignore AText as it contains the date/time
  LogWriteString(buildLogLine(Adata, '<<'));
end;

// ---------------------------------------------------------------------------

procedure TlogFile.LogSentData(const AText, AData: string);
begin
  // ignore AText as it contains the date/time
  LogWriteString(buildLogLine(Adata, '>>'));
end;

// ---------------------------------------------------------------------------

procedure TlogFile.LogStatus(const AText: string);
begin
  LogWriteString(buildLogLine(AText, '**'));
end;

// ---------------------------------------------------------------------------

procedure TlogFile.LogWriteString(const AText: string);
begin
  // protected --> public
  inherited;
end;
                                                       
// ---------------------------------------------------------------------------

//this will use an https post to push the data
procedure TDHLShipment.sendFile;
const testURL   = 'https://some url that accepts data'; //replace this with the url of where you will be sending the data or send it as a param
      logFileName = 'j:\winprog\main\http.log';     //the logfile name if you use the logging
var
  plainData : TStringList;                              //this is the unencoded data
//  formData : TIdMultiPartFormDataStream;              //I've left this in just in case you want to see how the multi-part works
  HTTP   : TidHTTP;                                     //going to need the HTTP object
  SSLIO : TIdSSLIOHandlerSocketOpenSSL;                 //going to need the SSL object
  response : TStringList;                               //this will catch the response from the server
  logFile : TLogFile;                                   //this is the logfile and is optional but handy for debuging

  //this is because the plainData TStringList may contain CR/LF pairs. If so, HTTP Post will convert those pairs to & which, in my specific case, kills the data.
  procedure ConvertCRLFToSpace;
  begin
    if plainData.Count > 1 then
      // convert CR/LF pairs to spaces
      plainData.Text := StringReplace(Trim(plainData.Text), sLineBreak, ' ',[rfReplaceAll]);
  end;

begin
//  formData := TIdMultiPartFormDataStream.Create;          //if you wanted to send multi-part data, use this instead
  plainData := TStringList.Create;                          //My specific case wants stripped down headers and no multipart functionality
  try
//     formData.AddFile('Shipment', 'dhl.xml', 'text/xml'); //you can add a lot more detail about the file with the multi part data stream
     plainData.LoadFromFile('dhl.xml');                     //or you can just load up the plain file
     Http:=TidHTTP.Create(nil);                             //create the HTTP instance
     try
        SSLIO:=TIdSSLIOHandlerSocketOpenSSL.Create(nil);    //create the SSL handler
        SSLIO.SSLOptions.Method := sslvSSLv3;               //set the SSL mode
        HTTP.ReadTimeout := 10000;                          //depending on size of upload, you may need to adjust these
        HTTP.ConnectTimeout := 10000;
        HTTP.IOHandler := SSLIO;                            //assign the SSL handler to the HTTP IO Handler or you won't be able to interact with SSL servers
        HTTP.Request.Contenttype := 'text/xml';             //I'm doing XML, set the content type appropriate to your data type. If you used the multi-part, you probably don't need this
        HTTP.HTTPOptions := []; //no options
        logFile := TLogFile.Create(nil);                    //create the log file
        logFile.FileName:=logFileName;                      //set the default log file name - NOTE: you could get fancy with this, but I normally don't use it
        logFile.Active:=true;                               //activate the log file (creates the file and intializes everything)
        HTTP.IOHandler.Intercept := logFile;                //set the logFile to the intercept of the IOHandler to capture the events
        response:=TStringList.Create;                       //create the string list to catch the server's response
        try
           try
              ConvertCRLFToSpace;                           //clean up the CR/LFs so the data doesn't get funky (you may not need this)
              response.Text := HTTP.Post(testURL, plainData); //post the data to the URL and place the response in the response string list
           except on EIdOSSLCouldNotLoadSSLLibrary do         //if we have an SSL problem, this is going to be when it shows
              ShowMessage('An error occurred loading the SSL '+ //provide a little clue although, as I mentioned yesterday, you may need to change the SSL loader to get good error messages
                          'libraries. Please make sure you '+
                          'have installed the OpenSSL '+
                          'libraries from http://www.'+
                          'openssl.org/related/binaries.html.');
           end;
           response.SaveToFile('response.xml');             //save the server's response (or process it or whatever you're going to do)
           ShowMessage('Sent successfully. Response='+#13#10+response.Text); //tell the user it worked
        finally
           response.Free;                                   //free the response
        end;
     finally
        Http.free;                                          //free the http object
     end;
  finally
     plainData.Free;                                        //free the data
     //formData.Free;
  end;

end;