In the beginning stages of a web crawling project or when you have to scale it to only a few hundred requests, you might want a simple proxy rotator that uses the free proxy pools available on the internet to populate itself now and then.
We can use a website like https://sslproxies.org/ to fetch public proxies every few minutes and use them in our C# projects.
This is what the site looks like:
And if you check the HTML using the inspect tool, you will see the full content is encapsulated in a table with the id proxylisttable
The IP and port are the first and second elements in each row.
We can use the following code to select the table and its rows to iterate on and further pull out the first and second elements of the elements.
Fetching the Proxy List
First, we need to install the HtmlAgilityPack NuGet package:
PM> Install-Package HtmlAgilityPack
Then we can make a request to the proxy website and parse the HTML:
using HtmlAgilityPack;
// Make request
var web = new HtmlWeb();
var doc = web.Load("<https://sslproxies.org/>");
// Parse HTML
var nodes = doc.DocumentNode.SelectNodes("//table[@id='proxylisttable']/tr");
The
var proxies = new List<Proxy>();
foreach (var node in nodes)
{
if(node.SelectNodes("td").Count < 2) continue;
var ip = node.SelectNodes("td")[0].InnerText;
var port = node.SelectNodes("td")[1].InnerText;
proxies.Add(new Proxy{
Ip = ip,
Port = port
});
}
This will give us a list of
Using a Random Proxy
To use a random proxy from the list, we can generate a random index:
var random = new Random();
var index = random.Next(0, proxies.Count);
var randomProxy = proxies[index];
Then we can make a request using
var handler = new HttpClientHandler{
Proxy = new WebProxy(randomProxy.Ip + ":" + randomProxy.Port),
};
var client = new HttpClient(handler);
var response = client.GetAsync("<https://example.com>");
Full Code
Here is the full code to fetch proxies and make a request with a random one:
using System;
using System.Net.Http;
using System.Collections.Generic;
using HtmlAgilityPack;
public class ProxyRotator
{
public class Proxy
{
public string Ip { get; set; }
public string Port { get; set; }
}
public static async Task Main()
{
var proxies = FetchProxies();
var randomProxy = GetRandomProxy(proxies);
var handler = new HttpClientHandler{
Proxy = new WebProxy(randomProxy.Ip + ":" + randomProxy.Port),
};
using (var client = new HttpClient(handler))
{
var response = await client.GetAsync("<https://example.com>");
// use response...
}
}
public static List<Proxy> FetchProxies()
{
var web = new HtmlWeb();
var doc = web.Load("<https://sslproxies.org/>");
var nodes = doc.DocumentNode.SelectNodes("//table[@id='proxylisttable']/tr");
var proxies = new List<Proxy>();
foreach (var node in nodes)
{
if(node.SelectNodes("td").Count < 2) continue;
var ip = node.SelectNodes("td")[0].InnerText;
var port = node.SelectNodes("td")[1].InnerText;
proxies.Add(new Proxy{
Ip = ip,
Port = port
});
}
return proxies;
}
public static Proxy GetRandomProxy(List<Proxy> proxies)
{
var random = new Random();
var index = random.Next(0, proxies.Count);
return proxies[index];
}
}
This provides a simple way to implement a rotating proxy in C#. Call
If you want to use this in production and want to scale to thousands of links, then you will find that many free proxies won't hold up under the speed and reliability requirements. In this scenario, using a rotating proxy service to rotate IPs is almost a must.
Otherwise, you tend to get IP blocked a lot by automatic location, usage, and bot detection algorithms.
Our rotating proxy server Proxies API provides a simple API that can solve all IP Blocking problems instantly.
Hundreds of our customers have successfully solved the headache of IP blocks with a simple API.
A simple API can access the whole thing like below in any programming language.
curl "<http://api.proxiesapi.com/?key=API_KEY&url=https://example.com>"
We have a running offer of 1000 API calls completely free. Register and get your free API Key here.