Library tutorials & articles

Retrieving HTTP content in .NET

We can walk and chew gum at the same time!

Events are cool, but they're not all that useful if you're running in blocking mode as I've shown so far. HttpWebRequest/HttpWebResponse can also be run in asynchronous modes using the BeginGetResponse/EndGetResponse methods. Most of the stream classes provide this mechanism which allows you to specify a callback method to collect output retrieved from these requests (you can also send data asynchronously this way).

However, after playing with this for a while and then looking at the native thread support in the .NET framework it turned out to be easier to create a new thread of my own and encapsulate the thread operation in a class. The following example runs multiple wwHttp objects on a couple of threads simultaneously while also updating the form with information from the OnReceiveData event. Figure 1 shows what the form looks like. While the HTTP requests are retrieved the main form thread is still available to perform other tasks, so you can move around the form, so the form's UI remains active all the while.

This process is surprisingly simple in .NET partly because .NET makes it easy to route thread methods into classes. This make it both easy to create a thread's processing into a nicely encapsulated format as well as providing an easy packaging mechanism for passing data into a thread and keeping that data isolated from the rest of an application.

Make sure you add the System.Threading namespace to your forms that use threads. The following code defines the thread handler class that fires the HTTP request with the FireUrls() method (Listing 5).

Listing 6: Implementing a Thread class

public class GetUrls
{
  public string Url = "";
  public wwHttpMultiThread ParentForm = null;
  public int Instance = 1;

  public void FireUrls()
  {
   wwHttp oHttp = new wwHttp();
   oHttp.OnReceiveData +=
             new wwHttp.OnReceiveDataHandler(this.OnReceiveData);
   oHttp.Timeout=5;
   string lcHTML = oHttp.GetUrlEvents(this.Url,4096);
   if (oHttp.Error)
     this.ParentForm.lblResult.Text = oHttp.ErrorMsg;
  }

  public void OnReceiveData(object sender,
             wwHttp.OnReceiveDataEventArgs e)
  {
   if (this.Instance == 1)
     this.ParentForm.lblResult.Text =
        e.CurrentByteCount.ToString() + " bytes";
   else
     this.ParentForm.lblResult2.Text =
        e.CurrentByteCount.ToString() + " bytes";
  }
}

There's nothing special about this class – in fact any class would do as thread handler (as long as you write thread safe code). This simplified implementation includes a reference back to the ParentForm that makes it possible to access the status labels on the form. The Instance property is used to identify which request is updating which form control. The actual code here is very much like code I've previously shown using the wwHttp object. Note that this code assigns the event handler a method of the thread action class. This method then calls back to the ParentForm and updates the labels.

The calling code on the form creates two threads that call the FireUrls method as follows (Listing 6):

Listing 7: Creating and running the Thread

private void cmdGo_Click(object sender, System.EventArgs e)
{
  GetUrls oGetUrls = new GetUrls();
  oGetUrls.Url = this.txtUrl.Text;
  oGetUrls.ParentForm = this;
 
  ThreadStart oDelegate =
        new ThreadStart(oGetUrls.FireUrls);
  Thread myThread = new Thread(oDelegate);
  myThread.Start();

  GetUrls oGetUrls2 = new GetUrls();
  oGetUrls2.ParentForm = this;
  oGetUrls2.Url = this.txtUrl2.Text;
  oGetUrls2.Instance = 2;
 
  ThreadStart oDelegate2 =
       new ThreadStart(oGetUrls2.FireUrls);
  Thread myThread2 = new Thread(oDelegate2);
  myThread2.Start();
}

To start a thread the ThreadStart() function is called which takes a function pointer (basically a reference that points at a specific method in a class) as an argument. This returns a Delegate that can be used to create a new thread and tell it to start running with this pointer. You can pass either an instance variable of a static address of static class method. In most cases you'll want to use a method of a dynamic instance object variable, because it gives you the ability to fully set up the instance by setting properties that you'll need as part of the processing. Think of the actual thread implementation class as wrapper that is used as the high level calling mechanism and parameter packager to your actual processing code. If you need to pass data back to some other object you can make this instance a member of another object. For example, I could have made this object part of the form which would then allow the form to access the members of the 'thread' class and both could share the data.

Creating threads and running them is very easy but make sure you manage your data cleanly to prevent access of shared data from different threads at the same time. Shared data needs to be protected with synchronization of some sort. In fact, you can see why this is an issue if you click the Go link on the sample form a few times while downloads are still running. You'll see the numbers jump back and forth as multiple threads update the same fields on the form. As multiple instances are writing to the labels at the same time the code will eventually blow up. The workaround for this is to use synchronized methods to handle the updates or to use separate forms to display the update information (new form for each request). The topic of synchronization is beyond the scope of this article and I’ll cover basic multi-threading concepts in a future article.

The Web at your fingertips

This cliché has been so overused, but I can say .NET really delivers on this promise in a number of ways. Yes you could do all of this and most other Internet related things before, but .NET brings a consistent model to the tools. In addition you get straight forward support for advanced features like easy to implement multi-threading and (albeit somewhat complex) event support that make it easy to created complex applications that utilize the tools in ways that were either not possible in, or took a lot more work previously.

The tools are all there to access the Web very easily whether it's through the high level tools in .NET or the more lower level network protocol classes. The HttpWebRequest class is a fairly easy and powerful implementation that provides an excellent balance between flexibility and ease of use. For me the wwHttp class has been an exercise in understanding and utilizing the HttpWebRequest/Response classes. I hope you find the class useful if not as is then as an example of a variety of things that need to be done with HTTP requests frequently. The class is not complete and some features will need expansion in the future, but it's good starting point.

Comments

  1. 10 Jul 2007 at 15:29

      if (oReqCookie.Name ==
            oRespCookie.Name)  {
            oReqCookie.Value =
                   oRespCookie.Name;

    Do you mean?








     if (oReqCookie.Name ==
            oRespCookie.Name)  {
            oReqCookie.Value =
                   oRespCookie.Value;




  2. 12 Mar 2007 at 18:57

    You have a paragraph that states:

    "The most common use of delegates is an eventhandler, which uses the delegate to fire events. When the event publisher fires the event method, the delegate that is assigned to handle the event is called and you're event subscriber object then can simply handle the event by implementing a method in your class."

    Unfortunately, for someone trying desparately to learn C#, this sounds like:

    "blah blah blah blah blah blah"

    I'm sorry but could you publish a REAL SIMPLE example and explanation of asynch calls in Asp.net/C#.  I have been all over the web looking for one and they all assume you already know C# so all the delegate calling callback calling method calling delegate calling calling callingcallingcalling.................has just finally gotten on my nerves.

  3. 29 Dec 2006 at 22:27

    I'm just new to c# coming from VB.NET.  This is a great example!  Thanks

  4. 10 May 2006 at 04:30

    hi

    i have similar problem like you (in a previous time) with in using httwebrequest to login in a website... especially how to handle event clicking ex: onkeypress="checklogin()"

    i hope you can help me to solve the problem..

     

    thnx b4

  5. 18 Oct 2005 at 02:51

    hi
    I use httpwebrequest to login into a website (using Networkcrendential). When i got that page i need to click a link. How can we achieve this (axwebbrowser ?) and it contains frames. And also after clicking that menu in second page i have to click a Javascript button. Is there anyway to achieve this.


    Rajesh

  6. 25 Mar 2005 at 00:48

    Your article is very good and your source code is a good start.  It has some cool features but is lacking some good programming practices.  Most noteable is the lack of finally blocks used.  You should try to insure that the Close() method on the WebRequest and WebResponse objects are always closed before the object reference goes out of scope.  You can ensure that happens by placing your code in a try/finally block.


    Also, you should consider having your class implement the IDisposable interface.  And in that implementation clean up, close, and release the resources used by your private variables.

  7. 01 Jan 1999 at 00:00

    This thread is for discussions of Retrieving HTTP content in .NET.

Leave a comment

Sign in or Join us (it's free).

AddThis

Related podcasts

  • CodeCast Episode 4: State of .NET, IE8, ASP.NET MVC, and O'Reilly Media

    CodeCast Episode 4: State of .NET, IE8, ASP.NET MVC, and O'Reilly MediaHosts Ken Levy and Markus Egger discuss the new State of .NET events, IE8, ASP.NET MVC, followed by an interview from PDC with two editors from O'Reilly Media. More on ASP.NET MVC can be found at http://asp.net/mvc. Interview...

Related jobs

Events coming up

  • Dec 6

    Developing AJAX Web Applications with Castle Monorail

    London, United Kingdom

    Monorail is the model-view-controller engine of the Castle Project, bringing many of the best ideas of Ruby on Rails to the .NET world. In this talk, David De Florinier and Gojko Adzic show how Monorail makes it easy to develop .NET based AJAX applications, and how to use the Castle Project to build Web 2.0 applications effectively. Come to this session if you are a .NET web developer. Everyone is welcome!