How to Create a Scraping Tool for Efficient Data Extraction in C#

Originally published on Medium · January 31, 2023 — examples target the tooling of that time.

Web scraping means pulling data off a page and turning it into something you can actually work with — rows in a database, entries in a report, whatever the task calls for. My tool of choice for this is Selenium. It's heavier than firing off a plain HTTP request, but it drives a real browser, which means it sails past a lot of the anti-scraping checks that trip up simpler approaches. Here's how I put together a scraping tool using Selenium and .NET.

Setting up the environment

Before we start, we need to have the following software installed:

Setup Selenium — we have several options, but the two most popular are: use a docker image, or install the Selenium driver on your local machine. A more scalable solution will be to use a docker image, so let's do that. The first step is to download the image:

docker pull selenium/standalone-chrome

Then to run the image:

$ docker run --rm -it \
    -p 4444:4444 -p 7900:7900 \
    --shm-size 2g selenium/standalone-chrome

To use the image, the only thing we will have to do is point to http://localhost:4444.

To see what is going on in the container, go to http://localhost:7900/?autoconnect=1&resize=scale&password=secret.

Building the scraping tool as a .NET 7 console app in Visual Studio

Once we have our development environment set up, we can start building our scraping tool. To do this, we will create a new .NET 7 Console project in Visual Studio. Once the project is created, we can add the Selenium.WebDriver and HtmlAgilityPack NuGet packages to the project. Selenium.WebDriver will be used to interact with the running docker image, and HtmlAgilityPack to get the data we need from the HTML document. To install these packages, simply open the NuGet Package Manager Console and run the following commands:

Install-Package Selenium.WebDriver
Install-Package HtmlAgilityPack

Next, we will add a reference to the Selenium library in our project. This can be done by using the using statement at the top of our code file.

using OpenQA.Selenium.Chrome;
using OpenQA.Selenium.Remote;

Now, we can use the RemoteWebDriver class from the Selenium library to create a driver and navigate to a website that you want to scrape.

Console.WriteLine("Creating Driver");

var driver = new RemoteWebDriver(
    new Uri("http://localhost:4444"), 
    new ChromeOptions());

Console.WriteLine("Getting the needed website HTML");

driver
    .Navigate()
    .GoToUrl("http://quotes.toscrape.com/");

var websiteHtml = driver.PageSource;

Extracting the data we need

Next, we will add a reference to the HtmlAgilityPack library in our project. This can be done by using the using statement at the top of our code file.

using HtmlAgilityPack;

Once we have obtained the HTML content, we can use HtmlAgilityPack to parse the HTML document and extract information from it. We start by creating a new HtmlDocument object and loading the HTML content into it.

var htmlDoc = new HtmlDocument();
htmlDoc.LoadHtml(websiteHtml);

With the HTML document loaded, we can use the methods of the HtmlDocument class to find elements and extract information. For example, to extract the text of a specific element, we can use the following code:

var title = htmlDoc
    .DocumentNode
    .SelectSingleNode("//h1");

Console.WriteLine($"Website title = {title.InnerText.Trim()}");

In this example, we use the SelectSingleNode method to locate an element in the HTML document based on its XPath. This method returns a single HtmlNode object that represents the element. We then use the InnerText property to obtain the text of the element.

Full .NET solution

using OpenQA.Selenium.Chrome;
using OpenQA.Selenium.Remote;
using HtmlAgilityPack;

Console.WriteLine("Creating Driver");

var driver = new RemoteWebDriver(
    new Uri("http://localhost:4444"), 
    new ChromeOptions());

Console.WriteLine("Getting the needed website html");

driver
    .Navigate()
    .GoToUrl("http://quotes.toscrape.com/");

var websiteHtml = driver.PageSource;

Console.WriteLine("Extracting wanted information");

var htmlDoc = new HtmlDocument();
htmlDoc.LoadHtml(websiteHtml);
var title = htmlDoc
    .DocumentNode
    .SelectSingleNode("//h1");

Console.WriteLine($"Website title = {title.InnerText.Trim()}");

Console.ReadLine();

Conclusion

In this article, we have explored how to build a website scraping tool using Selenium. By using HtmlAgilityPack to parse HTML documents, we can extract information from websites in an efficient and automated manner. Whether for research purposes, data analysis, or for use in data-driven applications, web scraping with Selenium and .NET is a valuable skill for any programmer looking to work with online data.