Using AFNetworking Proxies for Web Scraping in 2024

Setting up a Basic AFNetworking Proxy

The first step is importing AFNetworking and creating an AFHTTPSessionManager:

#import <AFNetworking.h>

NSURLSessionConfiguration *configuration = [NSURLSessionConfiguration defaultSessionConfiguration];
AFHTTPSessionManager *manager = [[AFHTTPSessionManager alloc] initWithSessionConfiguration:configuration];

We can then make a request by specifying the proxy server's hostname and port:

[manager setSessionProxyConfiguration:@"HTTP" host:@"192.168.1.100" port:8080];

NSURL *url = [NSURL URLWithString:@"<https://api.example.com/data>"];
NSURLRequest *request = [NSURLRequest requestWithURL:url];

[manager GET:url.absoluteString
   parameters:nil
     progress:nil
      success:^(NSURLSessionDataTask * _Nonnull task, id  _Nullable responseObject) {

			// Request succeeded

     } failure:^(NSURLSessionDataTask * _Nullable task, NSError * _Nonnull error) {

           // Request failed

}];

This proxies all requests through the server at 192.168.1.100 on port 8080.

Easy enough! Now let's look at configuring other proxy types...

Working with Different Proxy Protocols

AFNetworking supports the three major proxy protocols:

HTTP Proxy

These are the simplest and most common proxies on the internet. To use an HTTP proxy:

[manager setSessionProxyConfiguration:@"HTTP" host:@"proxy.example.com" port:3128];

HTTP proxies are easy to setup but lack encryption. For secure proxy connections, we need to use SOCKS instead.

SOCKS Proxy

SOCKS proxies route traffic at the TCP level for added privacy. Configuring them takes a bit more work:

NSDictionary *proxyConfig = @{
  (NSString *)kCFStreamPropertySOCKSProxyHost : @"proxy.example.com",
  (NSString *)kCFStreamPropertySOCKSProxyPort : @1080,
};

NSURLSessionConfiguration *configuration = [NSURLSessionConfiguration defaultSessionConfiguration];
configuration.connectionProxyDictionary = proxyConfig;

AFHTTPSessionManager *manager = [[AFHTTPSessionManager alloc] initWithBaseURL:url sessionConfiguration:configuration];

We specify the SOCKS host and port in a proxy dictionary, and pass it to the session configuration.

One catch with SOCKS proxies is that domain name resolution happens on the client first. This leaks your actual IP address before traffic is routed through the proxy. To fix this, make sure your proxies support "remote DNS resolution".

Self-Signed SSL Certificate Proxy

If your proxy uses a custom SSL certificate, you need to implement the NSURLSessionDelegate method to allow the unrecognized cert initially:

- (void)URLSession: (NSURLSession *)session
              task: (NSURLSessionTask *)task
didReceiveChallenge: (NSURLAuthenticationChallenge *)challenge
 completionHandler: (void (^)(NSURLSessionAuthChallengeDisposition disposition, NSURLCredential *credential))completionHandler
{

    if ([challenge.protectionSpace.authenticationMethod isEqualToString:NSURLAuthenticationMethodServerTrust]) {
            NSURLCredential *credential = [NSURLCredential credentialForTrust:challenge.protectionSpace.serverTrust];
            completionHandler(NSURLSessionAuthChallengeUseCredential, credential);
        } else {
            completionHandler(NSURLSessionAuthChallengePerformDefaultHandling, nil);
        }

}

This allows what would normally be blocked as an invalid SSL certificate, by explicitly trusting it.

Advanced Proxy Functionality

Now that you know the basics, let's look at some advanced configuration and features available for AFNetworking proxies:

Authentication

If your proxy requires a username and password, you can specify them like so:

NSString *username = @"testuser";
NSString *password = @"testpass";

NSDictionary *proxyConfig = @{
   (NSString *)kCFStreamPropertyHTTPProxyHost : @"proxy.example.com",
   (NSString *)kCFStreamPropertyHTTPProxyPort : @8080,
   (NSString *)kCFStreamPropertyHTTPProxyUser : username,
   (NSString *)kCFStreamPropertyHTTPProxyPassword : password
};

Custom HTTP Headers

To add a custom header like User-Agent with your proxied requests:

[manager.requestSerializer setValue:@"My App Name" forHTTPHeaderField:@"User-Agent"];

Caching Support

To avoid repeat requests, you can check for cached responses first by specifying cachePolicy:

NSURLRequestCachePolicy policy = NSURLRequestUseProtocolCachePolicy; // Cache if directed by server headers

[manager GET:url cachePolicy:policy
   parameters:nil
     progress:nil
      success:^(NSURLSessionDataTask * _Nonnull task, id  _Nullable responseObject) {}

      failure:^(NSURLSessionDataTask * _Nullable task, NSError * _Nonnull error) {}
];

There are several other cache policies like forcing a reload that you can experiment with.

Debugging Tools

If you run into issues with your proxy implementation, enable logging to help diagnose:

manager.taskDidCompleteWithErrorBlock = ^(NSURLSession *session, NSURLSessionTask *task, NSError *error) {
 	// Log errors
  	if (error != nil) {
     NSLog(@"%@", error);
  	}
};

This lets you catch errors from individual requests.

Example: Building a Weather App with AFNetworking Proxies

To see proxies in action, let's walk through a simple weather app that fetches data from a third-party API:

// Import frameworks
@import AFNetworking;

// Create manager with proxy
AFHTTPSessionManager *manager = [AFHTTPSessionManager manager];
[manager setSessionProxyConfiguration:@"HTTP" host:@"proxy.example.com" port:8080];

// API and app keys
NSString *apiEndpoint = @"<https://api.weatherapi.com/v1/current.json>";
NSString *apiKey = @"123abc";

// city to fetch weather for
NSString *city = @"Boston";

// Parameter dict
NSDictionary *parameters = @{@"key": apiKey, @"q": city};

// Fetch weather data
[manager GET:apiEndpoint parameters:parameters progress:nil success:^(NSURLSessionDataTask * _Nonnull task, id  _Nullable responseObject) {

  // Parse JSON response
  NSDictionary *current = [responseObject valueForKey:@"current"];

  // Extract values
  NSString *conditionText = [current valueForKey:@"condition"]["text"];
  NSString *tempC = [current valueForKey:@"temp_c"];

  // Update UI
  self.conditionLabel.text = conditionText;
  self.temperatureLabel.text = tempC;

  NSLog(@"Success!");

} failure:^(NSURLSessionDataTask * _Nullable task, NSError * _Nonnull error) {

  NSLog(@"Error: %@", error);

}];

Here's what's happening step-by-step:

1. Import AFNetworking
2. Create an AFHTTPSessionManager instance and configure an HTTP proxy server
3. Define the API endpoint, keys, and city search parameter
4. Make a GET request with the search criteria
5. On success, parse the JSON response
6. Extract the current weather condition text and temperature
7. Update the UI labels
8. Handle any errors from the network request

And that's it! By routing requests through the proxy server, we can reliably access the remote weather API, even from areas that may be blocked or to bypass usage limits.

The proxy integration was only a few lines of additional code but unlocks lots of possibilities.

Let's wrap up with some best practices for diagnosing issues...

Troubleshooting Common AFNetworking Proxy Problems

Despite the simplicity above, I've faced my fair share of frustrations getting proxies in iOS apps to work properly.

Here are some common pitfalls and how I debug them:

SSL Errors

If you see SSL certificate errors like NSURLErrorDomain error -1012 connect failures when trying to use a proxy server, double check:

HTTP Status Codes

If requests succeed but you receive unexpected HTTP codes like 403 or 500 errors, ensure:

Enabling debug logging is invaluable here to pinpoint if issues are from the proxy itself versus app code.

JSON Parsing Failures

If data loads fine over a proxy but JSON serialization is failing:

Again, adding debug checks saves tons of time finding parsing issues.

App Store Submissions

When submitting your app using proxies to the iOS App Store:

Following best practices here ensures a smooth review process.

Try Proxies API for Simple, Scalable Web Scraping

While AFNetworking proxies work well, managing scrapers involves dealing with proxies getting blocked, needing CAPTCHA solving, and handling IP rotations.

If you just want to focus on writing your web scrapers, check out Proxies API:

It provides a single API to fetch web pages that handles:

There is even a generous free tier to get started.

I recommend giving Proxies API a try if you want to skip proxy management headaches! It makes web scraping projects much easier to work on at scale.