May 2nd, 2020

Selenium Web Driver Vs Proxies API

The world of web scraping is varied and complex and Proxies API sits at one of the most crucial junctions. Allowing web scrapers/crawlers to bypass IP blocks by using a single API endpoint to access our 20 million-plus high-speed proxies on rotation.

Example:

curl "http://api.proxiesapi.com/?auth_key=YOUR_KEY&url=URL"

One of the questions we get frequently is how we are different from services like OctoParse or Diffbot. Many times it is like comparing Apples and Oranges but when we send this comparison table to our customer's developer team, their CXO, their marketing or SEO team, they typically get it quite easily if we are a suitable service or not.

So here is how we are different from Selenium Web Driver...

Selenium was built for automating tasks on web browsers but is very effective in web scraping as well...

Here you are controlling the Firefox browser and automating a search query...

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support.expected_conditions import presence_of_element_located

#This example requires Selenium WebDriver 3.13 or newer
with webdriver.Firefox() as driver:
    wait = WebDriverWait(driver, 10)
    driver.get("")
    driver.find_element_by_name("q").send_keys("cheese"   Keys.RETURN)
    first_result = wait.until(presence_of_element_located((By.CSS_SELECTOR, "h3>div")))
    print(first_result.get_attribute("textContent"))

Its language agnostic so here is the same thing accomplished using Javascript.

const {Builder, By, Key, until} = require('selenium-webdriver');

(async function example() {
    let driver = await new Builder().forBrowser('firefox').build();
    try {
        // Navigate to Url
        await driver.get('');

        // Enter text "cheese" and perform keyboard action "Enter"
        await driver.findElement(By.name('q')).sendKeys('cheese', Key.ENTER);

        let firstResult = await driver.wait(until.elementLocated(By.css('h3>div')), 10000);

        console.log(await firstResult.getAttribute('textContent'));
    }
    finally{
        driver.quit();
    

Selenium Web Driver vs Proxies API

AspectProxies APISelenium Web Driver
Who is it for?DevelopersDevelopers
Cost1000 free CallsStarts at $49 pmOpen Source
API accessYesyes
Size of projectenterprisemediumsmallenterprisemediumsmall
Easy to setupsingle api call for everythingmanual setup
Product/Serviceproductproduct
Rotating ProxiesYesno
Single API?Yesno
Desktop Appnono
Visual Scrapingnono

Share this article:

Get our articles in your inbox

Dont miss our best tips/tricks/tutorials about Web Scraping
Only great content, we don’t share your email with third parties.
Icon