views:

1774

answers:

3

I have an Asp .net page like this simple one http://issamsoft.com/app2/page1.aspx and I want to post to it some data and extract data from the response, by using TIdHttp. I tried to do that in Delphi2009 like this:

Procedure TForm1.Button1Click(Sender: TObject);
Const
VIEWSTATE = '/wEPDwUKMjA3NjE4MDczNmRkSxPt/LdmgqMd+hN+hkbiqIZuGUk=';
EVENTVALIDATION = '/wEWAwL40NXEDALs0bLrBgKM54rGBtmtdOYy+U7IFq8B25bYT1d4o1iK';
FORMPARAMS = 'TextBox1=Issam&Button1=Button';
URL = 'http://issamsoft.com/app2/page1.aspx';
var
http: TIdHttp;
lstParams: TStringList;
begin
 http := TIdHTTP.Create(self);
 lstParams := TStringList.Create;
 try
  lstParams.Add('__VIEWSTATE='+VIEWSTATE);
  lstParams.Add('__EVENTVALIDATION='+EVENTVALIDATION);
  lstParams.Add(FORMPARAMS);
  http.Request.ContentType := 'application/x-www-form-urlencoded';
  Memo1.Lines.Text := http.Post(url,lstParams);
 finally
  http.Free;
  lstParams.Free;
 end;

end;

but TIdhttp always gives an error(HTTP/1.1 500 Internal Server Error.) I read some comments in the idHttp unit talks about problems with http protocol v 1.1 like this one:

Currently when issuing a POST, IdHTTP will automatically set the protocol to version 1.0 independently of the value it had initially, This is because there are some servers that don't respect the RFC to the full extent. In particular, they don't respect sending/not sending the Expect: 100-Continue header. Until we find an optimum solution that does NOT break the RFC, we will restrict POSTS to version 1.0.

is there something wrong with my code or it's TidHttp Bug? and if the problem is in TIdHttp, is there any workaround? or is there other solution using Indy components?

besides. I've made a solution in C# using WebClient and it works very good.

        private void button1_Click(object sender, EventArgs e)
    {
        WebClient myClient = new WebClient();
        string viewstate = HttpUtility.UrlEncodeUnicode(@"/wEPDwUKMjA3NjE4MDczNmRkSxPt/LdmgqMd+hN+hkbiqIZuGUk=");
        string eventvaildation = HttpUtility.UrlEncodeUnicode(@"/wEWAwL40NXEDALs0bLrBgKM54rGBtmtdOYy+U7IFq8B25bYT1d4o1iK");
        string postdata = "__VIEWSTATE=" + viewstate + "&" + 
            "__EVENTVALIDATION=" + eventvaildation + "&TextBox1=Issam&Button1=Button";
        myClient.Headers.Add("Content-Type", "application/x-www-form-urlencoded");
        byte[] responce = myClient.UploadData("http://issamsoft.com/app2/page1.aspx", Encoding.ASCII.GetBytes(postdata));
        txtResponse.Text = Encoding.ASCII.GetString(responce);
    }

where i can find (good/trusted) class like WebClient in Delphi? free preferred :)

Edit: I hope mechanism of VIEWSTATE,EVENTVALIDATION is clear enough for you, they are hash values generated by server, and they may change(already changed), my orginal project has a piece of code just to extract the current VIEWSTATE,EVENTVALIDATION values, but I omit that part just to make my example simple and clear, so when you want to try the above code you must take VIEWSTATE,EVENTVALIDATION values from the current page source.

A: 

If you include the hoKeepOrigProtocol option in the TIdHTTP instance HTTPOptions property, it will not fall back to HTTP 1.0.

This is documented some lines below the comment text which you cited :)

    // If hoKeepOrigProtocol is SET, is possible to assume that the developer
    // is sure in operations of the server
    if not (hoKeepOrigProtocol in FOptions) then begin
      FProtocolVersion := pv1_0;
    end;
mjustin
Yes you are right, I did that also, and it didn't work :(
Issam Ali
Have you tried a HTTP proxy like Don's Proxy (from Sourceforge, http://donsproxy.sourceforge.net/) to record the HTTP request of Delphi and C# client? This tool is very useful to track down the differences between the requests.
mjustin
I use Fiddler (http://www.fiddler2.com)
Issam Ali
So you can save the requests to files and use a diff tool to find the differences (either in the HTTP header or in the payload) between the Delphi and the C# version.
mjustin
A: 

You need to update the VIEWSTATE and EVENTVALIDATION values.

Running that code causes a 500 error, but if I open the page and use the new __VIEWSTATE and __EVENTVALIDATION values from the webpage source, it no longer causes a 500 error.

I don't know why a C# client would have worked when the Delphi one didn't, but maybe it doesn't work anymore?

This looks like an ASP issue, not Delphi. Both Indy (TIdHTTP) and Synapse are good socket libraries for Delphi.

jasonpenny
-1 Actually, you should regard the VIEWSTATE and EVENTVALIDATION as readonly. They are property bags, internally used by ASP.NET to reconstruct the 'before' state when a POST request (with fresh form data) is processed at the server.
Jeroen Pluimers
Right, the values in the Delphi code just needed to be updated, and then the posted code worked as is. I didn't mean to change the ASP.NET page.
jasonpenny
+3  A: 

This is most likely caused by the fact that in C#, you use an ampersand (&) characters to concatenate the VIEWSTATE, EVENTVALIDATION and FORMPARAMS.

But in Delphi, you use a TStringList and Add. Add will not put an ampersand (&), but a CR+LF between the additions.

Hence you are sending different data in Delphi than in C#.

If you change your string concatenation logic in Delphi to match the logic in C#, it should work, not matter what kind of WebClient you use on the Delphi side.

--jeroen

Edit: 20090830

This code works; inspecting the C# and IE7 output using Fiddler2, I found out you also need to escape the '/' and '=' characters.

Edit2: 20090831

Somehow the VIEWSTATE and EVENTVALIDATION on the site have changed since yesterday (though I'm not sure why). I have changed the code, and it works at the time of posting my changes to this answer. This is the new request content as captured per Fiddler when posting the page from within Internet Explorer 7:

__VIEWSTATE=%2FwEPDwUKMjA3NjE4MDczNmRk5%2FC2iWwvlAB3L1wYzRpm3KZhRC0%3D&__EVENTVALIDATION=%2FwEWAwLXzuATAuzRsusGAoznisYGSYOqDGy4vMunY6A8xi6ahQEPI5Q%3D&TextBox1=Issam&Button1=Button

This is the new request as when posted form the Delphi example:

__VIEWSTATE=%2FwEPDwUKMjA3NjE4MDczNmRk5%2FC2iWwvlAB3L1wYzRpm3KZhRC0%3D&__EVENTVALIDATION=%2FwEWAwLXzuATAuzRsusGAoznisYGSYOqDGy4vMunY6A8xi6ahQEPI5Q%3D&TextBox1=Issam&Button1=Button

The below sample now gives this result (note the "Issam" in the result):

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"&gt;

<html xmlns="http://www.w3.org/1999/xhtml" >
<head><title>

</title></head>
<body>
    <form name="form1" method="post" action="page1.aspx" id="form1">
<input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE" value="/wEPDwUKMjA3NjE4MDczNg9kFgICAw9kFgICBQ8PFgIeBFRleHQFDVdlbGNvbWUgSXNzYW1kZGSCDMOkTMjkZJgqLkhpK99twpD5+A==" />

<input type="hidden" name="__EVENTVALIDATION" id="__EVENTVALIDATION" value="/wEWAwKr+O/BBQLs0bLrBgKM54rGBlueO5BU/6BAJMZfHNwh5fsQFuAm" />
    <div>

        <input name="TextBox1" type="text" value="Issam" id="TextBox1" />
        <input type="submit" name="Button1" value="Button" id="Button1" />

        <br />

        <span id="Label1">Welcome Issam</span>
        <br />

    </div>
    </form>
</body>
</html>

Edit3: 20090831

And indeed the VIEWSTATE and EVENTVALIDATION changed again: somehow they keep changing: there seems to be a time factor involved.

So I adapted the code, now it it can extract the VIEWSTATE and EVENTVALIDATION from a GET request, then put that into a POST request.

In addition, it appeared that the '+' and ':' also needs to be escaped. So I dug into the ASP.NET page generation logic, found out that they use HttpUtility.UrlEncode to do the escaping, which ultimately checks HttpUtility.IsSafe which characters to escape. So I adapted took the Reflector reverse engineerd Delphi code into a THttpUtility class.

This is the final code:

{$DEFINE FIDDLER}

unit HttpPostExampleFormUnit;

interface

uses
  Windows, Messages, SysUtils, Variants, Classes, Graphics, Controls, Forms,
  Dialogs, StdCtrls, IdBaseComponent, IdComponent, IdTCPConnection, IdTCPClient, IdHTTP;

type
  THttpPostExampleForm = class(TForm)
    Button1: TButton;
    Memo1: TMemo;
    procedure Button1Click(Sender: TObject);
  private
    procedure Log(const What: string; const Content: string);
    procedure LogClear;
  end;

var
  HttpPostExampleForm: THttpPostExampleForm;

implementation

uses
  HttpUtilityUnit;

{$R *.dfm}

procedure AddFormParameter(const StringStream: TStringStream; const ParameterName: string; const ParameterValue: string);
var
  EncodedParameterValue: string;
begin
  StringStream.WriteString(ParameterName);
  StringStream.WriteString('=');
  EncodedParameterValue := THttpUtility.UrlEncode(ParameterValue);
  StringStream.WriteString(EncodedParameterValue);
end;

procedure AddFormParameterSeparator(const StringStream: TStringStream);
begin
  StringStream.WriteString('&');
end;

function ExtractHiddenParameter(const ParameterName: string; const Request: string): string;
//<input type="hidden" name="__VIEWSTATE" id="__VIEWSTATE" value="/wEPDwUKMjA3NjE4MDczNg9kFgICAw9kFgICBQ8PFgIeBFRleHQFDVdlbGNvbWUgSXNzYW1kZGSCDMOkTMjkZJgqLkhpK99twpD5+A==" />
const
  PrefixMask = 'input type="hidden" name="%s" id="%s" value="';
  Suffix = '" />';
var
  Prefix: string;
  PrefixLength: Integer;
  PrefixPosition: Integer;
  SuffixPosition: Integer;
begin
  Prefix := Format(PrefixMask, [ParameterName, ParameterName]);
  PrefixPosition := Pos(Prefix, Request);
  if PrefixPosition = 0 then
    Result := ''
  else
  begin
    PrefixLength := Length(Prefix);
    Result := Copy(Request,
      PrefixPosition + PrefixLength,
      1 + Length(Request) - PrefixPosition - PrefixLength);
    SuffixPosition := Pos(Suffix, Result);
    if SuffixPosition = 0 then
      Result := ''
    else
      Delete(Result, SuffixPosition, 1 + Length(Result) - SuffixPosition);
  end;
end;

procedure THttpPostExampleForm.Button1Click(Sender: TObject);
const
  DefaultVIEWSTATE = '/wEPDwUKMjA3NjE4MDczNmRk5%2FC2iWwvlAB3L1wYzRpm3KZhRC0=';
  DefaultEVENTVALIDATION = '/wEWAwLXzuATAuzRsusGAoznisYGSYOqDGy4vMunY6A8xi6ahQEPI5Q=';
  FORMPARAMS = 'TextBox1=Issam&Button1=Button';
  URL = 'http://issamsoft.com/app2/page1.aspx';
  __VIEWSATE = '__VIEWSTATE';
  __EVENTVALIDATION = '__EVENTVALIDATION';
var
  VIEWSTATE: string;
  EVENTVALIDATION: string;
  getHttp: TIdHttp;
  getRequest: string;
  postHttp: TIdHttp;
  ParametersStringStream: TStringStream;
  postRequest: string;
begin
  LogClear();
  getHttp := TIdHTTP.Create(self);
  try
    getRequest := getHttp.Get(URL);
    Log('GET Request', getRequest);
    VIEWSTATE := ExtractHiddenParameter(__VIEWSATE, getRequest);
    EVENTVALIDATION := ExtractHiddenParameter(__EVENTVALIDATION, getRequest);
    Log('Extracted VIEWSTATE', VIEWSTATE);
    Log('Extracted EVENTVALIDATION', EVENTVALIDATION);
  finally
    getHttp.Free();
  end;

  postHttp := TIdHTTP.Create(self);
{$IFDEF FIDDLER}
  postHttp.ProxyParams.ProxyServer := '127.0.0.1';
  postHttp.ProxyParams.ProxyPort := 8888;
{$ENDIF FIDDLER}
  postHttp.HTTPOptions := postHttp.HTTPOptions + [hoKeepOrigProtocol];
  ParametersStringStream := TStringStream.Create('');
  try
    AddFormParameter(ParametersStringStream, __VIEWSATE, VIEWSTATE);
    AddFormParameterSeparator(ParametersStringStream);
    AddFormParameter(ParametersStringStream, __EVENTVALIDATION, EVENTVALIDATION);
    AddFormParameterSeparator(ParametersStringStream);
    ParametersStringStream.WriteString(FORMPARAMS);
    postHttp.Request.ContentType := 'application/x-www-form-urlencoded';
    ParametersStringStream.Seek(0, soFromBeginning);
    Log('POST Parameters', ParametersStringStream.DataString);
    postRequest := postHttp.Post(url, ParametersStringStream);
    Log('POST Request', postRequest);
  finally
    postHttp.Free;
    ParametersStringStream.Free;
  end;
end;

procedure THttpPostExampleForm.Log(const What, Content: string);
begin
  Memo1.Lines.Add(What + '=');
  Memo1.Lines.Add(Content);
end;

procedure THttpPostExampleForm.LogClear;
begin
  Memo1.Lines.Clear;
end;

end.

And the THttpUtlity class:

unit HttpUtilityUnit;

interface

type
  THttpUtility = class
  private
    class function IsSafe(const ch: Char): boolean;
  public
    class function UrlEncode(s: string): string;
  end;

implementation

uses
  Classes, SysUtils;

class function THttpUtility.IsSafe(const ch: Char): boolean;
begin
  if ((((ch >= 'a') and (ch <= 'z')) or ((ch >= 'A') and (ch <= 'Z'))) or ((ch >= '0') and (ch <= '9'))) then
    Result := True
  else
    case ch of
      '''',
        '(',
        ')',
        '*',
        '-',
        '.',
        '_',
        '!':
        Result := True;
    else
      Result := False;
    end;
end;

class function THttpUtility.UrlEncode(s: string): string;
var
  ch: Char;
  HexCh: string;
  StringStream: TStringStream;
  Index: Integer;
  Ordinal: Integer;
begin
  StringStream := TStringStream.Create('');
  try
    //Note: this is not yet UTF-16 compatible; check before porting to Delphi 2009
    for Index := 1 to Length(s) do
    begin
      ch := s[Index];
      if IsSafe(ch) then
        StringStream.WriteString(Ch)
      else
      begin
        Ordinal := Ord(Ch);
        HexCh := IntToHex(Ordinal, 2);
        StringStream.WriteString('%'+HexCh);
      end;
    end;

    Result := StringStream.DataString;
  finally
    StringStream.Free;
  end;
end;

end.

This should get you going.

--jeroen

Jeroen Pluimers
+1.. That should get Issam Ali going. I'd give extra points for trying out and providing a code example, but SO doesn't let me.
Wouter van Nifterick
Issam Ali
Issam Ali
As I said in my first comment, I tried to Post Encoded parameters(using C# app to produce it), and tell TIdhttp not to Encode it again(hoForceEncodeParams = false) !, and also that doesn't work :)
Issam Ali
((parameters)) not ((parmeters)) sorry :)
Issam Ali
Somehow the VIEWSTATE and EVENTVALIDATION changed; I have adapted my example, and it works at the time of posting. If VIEWSTATE and EVENTVALIDATION are time or session sensitive, then you need to grab them from first making a 'GET' request.
Jeroen Pluimers
Did some more research; grabbed VIEWSTATE and EVENTVALIDATION from a GET request, then put them into a POST request.Then dug into the ASP.NET escaping, and wrote a simple class for that.
Jeroen Pluimers
Fabulous!, I think you should publish THttpUtility to the community. because this kind of encoding is missing from Indy liberary, I think ;)Answer accepted.
Issam Ali
Will write a blog on it after the conference season is over.
Jeroen Pluimers