Making HTTP Requests Through a Proxy in Elixir with HTTPoison in 2024

Installing HTTPoison

I'm assuming you have an Elixir project set up. If not, quickly initialize one:

$ mix new http_proxy_example

First let's add HTTPoison as a dependency.

Open up mix.exs and under deps add:

def deps do
  [
    {:httpoison, "~> 1.8"}
  ]
end

Now fetch the new dependency:

$ mix deps.get

With HTTPoison installed, we're ready to make some requests.

Making a Basic Request

Let's start by making a simple GET request without a proxy configured:

response = HTTPoison.get!("<https://api.ipify.org>")

case response do
  %{status_code: 200, body: body} ->
    IO.puts body

  _ ->
    IO.puts "Request failed"
end

This will print out your public IP address.

Now let's try routing that request through a proxy using some different configuration approaches.

Configuring a Global Proxy

One method for proxying all HTTPoison requests is to define a global proxy at the application level.

In config/config.exs, add the following:

config :httpoison,
  http: [proxy: "<http://proxy.example.com:8080>"]

Now any call to HTTPoison.get, HTTPoison.post etc will route through that proxy URL.

While convenient, this applies the proxy globally which may not always be desirable.

Setting a Proxy Per-Request

For more control, you can specify a proxy when making the request:

HTTPoison.get!("<https://api.ipify.org>", [], [proxy: "<http://proxy.example.com:8080>"])

Here we are passing proxy details in the optional third argument.

This overrides any global proxy configs, allowing you to proxy requests on a one-off basis.

Let's look at some more advanced configurations next.

Using SOCKS Proxies

Basic HTTP proxies work by intercepting traffic at the HTTP protocol level.

SOCKS proxies operate at a lower OSI network layer, providing transparency for any protocol like HTTP, SMTP, FTP etc.

To use a SOCKS proxy with HTTPoison:

# Specify SOCKS5 proxy host/port
HTTPoison.get!("<https://api.ipify.org>", [], proxy: {:socks5, "proxy.example.com", 1080})

The proxy URL is wrapped in a tuple, indicating to use the SOCKS5 protocol.

Authentication & TLS

Some proxies require authentication credentials or transport layer security:

# Define username/password
auth = {"proxyuser", "pa$$word"}

HTTPoison.get!("<https://api.ipify.org>", [],
  proxy: {:socks5, "proxy.example.com", 1080},
  proxy_auth: auth)

Here we pass the proxy_auth option with the credential tuple.

For TLS, use the :ssl option and configure SSL certificates if needed:

HTTPoison.get!("<https://api.ipify.org>", [ssl: [{:ssl_options, ssl_options}]],
  proxy: {:socks5, "proxy.example.com", 1080},
  proxy_auth: {"proxyuser", "pa$$word"})

This covers the basics of configuring HTTP and SOCKS proxies with HTTPoison!

Proxy Rotation & Captcha Solving

As you start sending thousands of requests, IP blocks can pose challenges. This is where having access to a pool of proxies to rotate becomes useful.

Handling other protections like captchas also adds complexity.

Rather than rebuilding those capabilities yourself, services like Proxies API provide simple APIs that handle proxy rotation, captcha solving, and more for you automatically.

I've used them successfully in projects where avoiding scrapers and bot detection was important. Quick to integrate too with only minimal code changes required.

Key Takeaways

Here are the core concepts we covered for using proxies with HTTPoison:

HTTPoison makes it relatively painless to add proxy capabilities for flexibility.

There are more advanced patterns like setting up proxy chains/trees to further obscure traffic origins, but this should give a solid foundation for most use cases.

This is great as a learning exercise but it is easy to see that even the proxy server itself is prone to get blocked as it uses a single IP. In this scenario where you may want a proxy that handles thousands of fetches every day using a professional rotating proxy service to rotate IPs is almost a must.

Otherwise, you tend to get IP blocked a lot by automatic location, usage, and bot detection algorithms.

Our rotating proxy server Proxies API provides a simple API that can solve all IP Blocking problems instantly.

With millions of high speed rotating proxies located all over the world,With our automatic IP rotationWith our automatic User-Agent-String rotation (which simulates requests from different, valid web browsers and web browser versions)With our automatic CAPTCHA solving technology,

Hundreds of our customers have successfully solved the headache of IP blocks with a simple API.

The whole thing can be accessed by a simple API like below in any programming language.

In fact, you don't even have to take the pain of loading Puppeteer as we render Javascript behind the scenes and you can just get the data and parse it any language like Node, Puppeteer or PHP or using any framework like Scrapy or Nutch. In all these cases you can just call the URL with render support like so.

curl "http://api.proxiesapi.com/?key=API_KEY&render=true&url=https://example.com"

We have a running offer of 1000 API calls completely free. Register and get your free API Key here.

Frequently Asked Questions

Here are some common questions around using proxies with HTTP clients:

What are the main benefits of using a proxy?

Proxies provide security, performance gains through caching, geo-targeting capability, and obscurity regarding the original client IP address.

Do I need to authenticate to use some proxies?

Some proxies do require authentication credentials to use them. HTTPoison supports passing a username/password tuple for this.

Can I use proxies to bypass scrapers or bot detection?

Yes, having a pool of proxies to rotate through can help circumvent protections looking for repeated traffic from the same IPs. Services like Proxies API make automation easy.

What are the downsides to using a proxy?

Potential downsides include cost if using paid proxies, latency, lack of support for TLS in some cases, and having to handle authentication. So tradeoffs exist.