HttpWebRequest with GET method in Delphi .net

Why HttpWebRequest/HttpWebResponse?

This tutorial is to show you how to write an application in Delphi using HttpWebRequest to submit and pull data from a web server programmatic-ally.

If you are looking for a method to make a web bot or http bot, this topic is for you too.

Delphi 2009 .net supports functions/API from .net frameworks. Using Delphi .net, you can make requests to a web server and get raw responses under html format. This method is totally different than using a WebBrowser object in .net. WebBrowser object offers you a visual web interface where you could see the web components and click on it. With HttpWebRequest, the response from the web server is under html format in plain text.

The applications which uses HttpWebRequest have no visual web interface. It has all the data processing behind the form. A best practice is to call HttpWebRequest/HttpWebResponse from a background thead and within a loop. The loop will keep the thread running and HttpWebRequest will continuously make requests to/process responses from the web server.

With HttpWebRequest applications, you must process everything manually by yourself. It won’t execute Javascripts. It won’t download images or anything. The result is just text.

To download images, you must make another request as File Download, and save the image to a file on local system.

To process Javascripts, I found there is no options to process directly the Javascripts, but you have to understand the scripts and make the application act the same way. For example, the Javascript will check for input on the web form before clicking Submit button; you will have to add the code in your application to check the code, instead of using web Javascripts.

How to make a basic raw http loader in Delphi using HttpWebRequest?

Before I go down to step-by-step guides, I’d like to briefly indicate some basic http information and overall steps.

To specify which website we will access to, we need to have a website address under URL format: http://websitename.com. For example, I will make a search request to Google.com, the URL is http://www.google.com

If you want your application to support cookies, you must use CookieContainer object.

Main function source code

Take a look at the full source code of the main function which I can use to pull a website’s content. I will have more explanation by line #.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
function LoadWebSolo(website: string): string;
var MyURI: uri;
  MyWebRequest: HttpWebRequest;
  MyWebResponse: HttpWebResponse;
  MyStream: stream;
  MyStreamReader: streamreader;
 
begin
  try
    MyURI := uri.Create(website);
    MyWebRequest := HttpWebRequest.CreateDefault(MyURI) as HttpWebRequest;
    MyWebRequest.Accept := 'application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5';
    MyWebRequest.Useragent := 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/8.0.552.224 Safari/534.10';
    MyWebRequest.KeepAlive := true;
    MyWebRequest.ContentType := 'application/x-www-form-urlencoded';
    MyWebRequest.AllowAutoRedirect := true;
 
    MyWebResponse := HttpWebResponse(MyWebRequest.GetResponse);
    MyStream := MyWebResponse.GetResponseStream;
    MyStreamReader := StreamReader.Create(MyWebResponse.GetResponseStream);
    if MyStreamReader <> nil then
      Result := MyStreamReader.ReadToEnd;
 
  except
    on e: exception do
    ShowMessage('Error message: ' + e.message);
  end;
end;
function LoadWebSolo(website: string): string;
var MyURI: uri;
  MyWebRequest: HttpWebRequest;
  MyWebResponse: HttpWebResponse;
  MyStream: stream;
  MyStreamReader: streamreader;

begin
  try
    MyURI := uri.Create(website);
    MyWebRequest := HttpWebRequest.CreateDefault(MyURI) as HttpWebRequest;
    MyWebRequest.Accept := 'application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5';
    MyWebRequest.Useragent := 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/8.0.552.224 Safari/534.10';
    MyWebRequest.KeepAlive := true;
    MyWebRequest.ContentType := 'application/x-www-form-urlencoded';
    MyWebRequest.AllowAutoRedirect := true;

    MyWebResponse := HttpWebResponse(MyWebRequest.GetResponse);
    MyStream := MyWebResponse.GetResponseStream;
    MyStreamReader := StreamReader.Create(MyWebResponse.GetResponseStream);
    if MyStreamReader <> nil then
      Result := MyStreamReader.ReadToEnd;

  except
    on e: exception do
    ShowMessage('Error message: ' + e.message);
  end;
end;
Uses libraries
HttpWebRequest and HttpWebResponse are from System.Net

Stream and StreamReader are from System.IO.

Make sure you use these two libraries.

The overall process of making an httpWebRequest application includes:

  • Initiate URI object (specify which URL to go to)
  • Create a httpWebRequest object
  • Configure http request parameters
  • Submit HttpWebRequest and get HttpWebResponse back
  • Cast HttpWebResponse as Stream data, process Stream data as raw String data.

Step-by-step guides

And here is the demonstration steps of how to do it.

Step 1 – Initiate URI object (specify which URL to go to)

Line #2 – I just defined a URI type variable, named it as MyURI. This variable is to specify which URL I would go to. And at line #10, I initiated this variable by creating an URI object, pointing it to any string URL. String URL must begin with http:// or https://

At this point, you could write a small function to properly check the input string.

Step 2 – Create a httpWebRequest object

Line #3 – I defined MyWebRequest as HttpWebRequest type. This variable is to store the request information to web server. At line #11, I initiated MyWebRequest by using the CreateDefault constructor of HttpWebRequest.

MyURI is the URI variable that I just created from line #10.

Step 3 – Configure http request parameters

Line #12 to #16 – These lines are to configure http request options, such as, Accept type, UserAgent, KeepAlive, ContentType, or AllowAutoDirect,…

At this section, if you would like to add some fancy thing, such as customized header, or X-Requested-With, you must work on something extra. In this tutorial, I just show the method of how to make a basic http GET Request to a web server. After you got the idea of how it works, you could modify it later.

Step 4 – Submit HttpWebRequest and get HttpWebResponse back

Line #18 – The response from web server is answered back to MyWebResponse, which was defined at line #4. You don’t have to initiate this variable. The structure

1
MyWebResponse := HttpWebResponse(MyWebRequest.GetResponse);
MyWebResponse := HttpWebResponse(MyWebRequest.GetResponse);

will initiate and execute your http request.

If there is any problems with your request, you should have some exceptional messages at this point.

Step 5 – Cast HttpWebResponse as Stream data, process Stream data as raw String data.

From line #19 to #22 –  There are so many ways for you to process the response from web server. I used the method of casting the Response Stream to Stream, and then to StreamReader type. From StreamReader type, I could pull out the whole result to a normal String type.

The final String result is input into Result variable. In Delphi, Result variable is the output of that function.

How to process the result of HttpWebRequest?

As I built the function LoadWebSolo with result type as String, I could call LoadWebSolo and return the result to a String variable.

For example

1
2
3
4
5
6
procedure Test;
var webResult: string; //Define webResult as string variable
begin
  webResult := LoadWebSolo('http://www.google.com'); //Call LoadWebSolo to process google.com and return results to webResult
  Form1.Memo1.Lines.Add(webResult); //Output webResult string variable to Memo1 object on the main form.
end;
procedure Test;
var webResult: string; //Define webResult as string variable
begin
  webResult := LoadWebSolo('http://www.google.com'); //Call LoadWebSolo to process google.com and return results to webResult
  Form1.Memo1.Lines.Add(webResult); //Output webResult string variable to Memo1 object on the main form.
end;

Full source code and demo

Draw a form with an Edit field, a Button field, and a Memo field. Link the LoadWebSolo function to the button.

HttpWebRequest GET method in Delphi

And this is the full source code behind the form

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
unit Unit1;
//##############################################
// PLAINTUTORIALS.COM DELPHI DEMO
// HttpWebRequest GET method with Delphi .net
//##############################################
interface
 
uses
System.Net,
System.IO,
Windows, Messages, SysUtils, Variants, Classes, Graphics, Controls, Forms,
Dialogs, Borland.Vcl.StdCtrls, System.ComponentModel;
 
type
TForm1 = class(TForm)
edtURL: TEdit;
btnGo: TButton;
Memo1: TMemo;
lblAddress: TLabel;
procedure btnGoClick(Sender: TObject);
private
{ Private declarations }
public
{ Public declarations }
end;
 
var
Form1: TForm1;
 
implementation
 
{$R *.nfm}
 
function LoadWebSolo(website: string): string;
var MyURI: uri;
MyWebRequest: HttpWebRequest;
MyWebResponse: HttpWebResponse;
MyStream: stream;
MyStreamReader: streamreader;
 
begin
try
MyURI := uri.Create(website);
MyWebRequest := HttpWebRequest.CreateDefault(MyURI) as HttpWebRequest;
MyWebRequest.Accept := 'application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5';
MyWebRequest.Useragent := 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/8.0.552.224 Safari/534.10';
MyWebRequest.KeepAlive := true;
MyWebRequest.ContentType := 'application/x-www-form-urlencoded';
MyWebRequest.AllowAutoRedirect := true;
 
MyWebResponse := HttpWebResponse(MyWebRequest.GetResponse);
MyStream := MyWebResponse.GetResponseStream;
MyStreamReader := StreamReader.Create(MyWebResponse.GetResponseStream);
if MyStreamReader <> nil then
Result := MyStreamReader.ReadToEnd;
 
except
on e: exception do
ShowMessage('Error message: ' + e.message);
end;
end;
 
procedure TForm1.btnGoClick(Sender: TObject);
var webResult: String;
begin
webResult := LoadWebSolo(Form1.edtURL.Text);
Form1.Memo1.Lines.Add(webResult);
end;
 
end.
unit Unit1;
//##############################################
// PLAINTUTORIALS.COM DELPHI DEMO
// HttpWebRequest GET method with Delphi .net
//##############################################
interface

uses
System.Net,
System.IO,
Windows, Messages, SysUtils, Variants, Classes, Graphics, Controls, Forms,
Dialogs, Borland.Vcl.StdCtrls, System.ComponentModel;

type
TForm1 = class(TForm)
edtURL: TEdit;
btnGo: TButton;
Memo1: TMemo;
lblAddress: TLabel;
procedure btnGoClick(Sender: TObject);
private
{ Private declarations }
public
{ Public declarations }
end;

var
Form1: TForm1;

implementation

{$R *.nfm}

function LoadWebSolo(website: string): string;
var MyURI: uri;
MyWebRequest: HttpWebRequest;
MyWebResponse: HttpWebResponse;
MyStream: stream;
MyStreamReader: streamreader;

begin
try
MyURI := uri.Create(website);
MyWebRequest := HttpWebRequest.CreateDefault(MyURI) as HttpWebRequest;
MyWebRequest.Accept := 'application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5';
MyWebRequest.Useragent := 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/534.10 (KHTML, like Gecko) Chrome/8.0.552.224 Safari/534.10';
MyWebRequest.KeepAlive := true;
MyWebRequest.ContentType := 'application/x-www-form-urlencoded';
MyWebRequest.AllowAutoRedirect := true;

MyWebResponse := HttpWebResponse(MyWebRequest.GetResponse);
MyStream := MyWebResponse.GetResponseStream;
MyStreamReader := StreamReader.Create(MyWebResponse.GetResponseStream);
if MyStreamReader <> nil then
Result := MyStreamReader.ReadToEnd;

except
on e: exception do
ShowMessage('Error message: ' + e.message);
end;
end;

procedure TForm1.btnGoClick(Sender: TObject);
var webResult: String;
begin
webResult := LoadWebSolo(Form1.edtURL.Text);
Form1.Memo1.Lines.Add(webResult);
end;

end.

What’s next?

You could apply this method to continuously and automatically crawl and parse web site contents. Putting the call function in a loop of a background thread, your application could act as a crawler bot. When you master the techniques, you can do whatever you can think of, such as WordPress content auto creator, copier, linker,

Or, if you’re a fan of auctions bidding online, you could apply this technique to create a auctions bidding bot, auctions watcher, or whatsoever.

Anyway, do not apply this technique for bad purposes.

 

About The Author

Hao Nguyen

Hao Nguyen

Hello! I'm Hao Nguyen and I'm currently working as a Network Engineer for a small firm in Houston. I enjoy writing technical documents and blog, such as PlanTutorials.com

Trackbacks / Pingbacks

  1. Delphi HttpWebRequest with custom fields | Plain Tutorials

Leave a Reply