eBay is one of the largest online marketplaces with millions of active listings at any given time. In this tutorial, we'll walk through how to scrape and extract key data from eBay listings using Visual Basic and the HtmlDocument library.
Setup
We'll need to add a reference to System.Xml.dll to work with HTML documents.
We'll also define the starting eBay URL and a string for the user agent header:
Imports System.Xml
Module eBayScraper
Dim url As String = "<https://www.ebay.com/sch/i.html?_nkw=baseball>"
Dim userAgent As String = "Mozilla/5.0..."
Replace the user agent with your browser's user agent string.
Fetch the Listings Page
We can use the WebClient class to download the HTML:
Dim webClient As New WebClient()
webClient.Headers.Add("User-Agent", userAgent)
Dim html As String = webClient.DownloadString(url)
Dim htmlDoc As New HtmlDocument()
htmlDoc.LoadHtml(html)
The user agent header is added to the request. We load the HTML into an HtmlDocument.
Extract Listing Data
We can use XPath syntax to find the listings divs:
Dim listings = htmlDoc.DocumentElement.SelectNodes("//div[contains(@class,'s-item__info')]")
For Each listing In listings
Dim title As String = listing.SelectSingleNode(".//div[@class='s-item__title']").InnerText
Dim url As String = listing.SelectSingleNode(".//a[@class='s-item__link']").GetAttributeValue("href", "")
Dim price As String = listing.SelectSingleNode(".//span[@class='s-item__price']").InnerText
' Get other fields like seller, shipping, etc
Next
We extract the text and attributes from the selected elements.
Print Results
We can print the extracted info:
Console.WriteLine("Title: " & title)
Console.WriteLine("URL: " & url)
Console.WriteLine("Price: " & price)
Console.WriteLine(New String("=", 50)) ' Separator
This will output each listing's data.
Full Code
Here is the full code to scrape eBay listings:
Imports System.Xml
Module eBayScraper
Dim url As String = "<https://www.ebay.com/sch/i.html?_nkw=baseball>"
Dim userAgent As String = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36"
Dim webClient As New WebClient()
webClient.Headers.Add("User-Agent", userAgent)
Dim html As String = webClient.DownloadString(url)
Dim htmlDoc As New HtmlDocument()
htmlDoc.LoadHtml(html)
Dim listings = htmlDoc.DocumentElement.SelectNodes("//div[contains(@class,'s-item__info')]")
For Each listing In listings
Dim title As String = listing.SelectSingleNode(".//div[@class='s-item__title']").InnerText
Dim url As String = listing.SelectSingleNode(".//a[@class='s-item__link']").GetAttributeValue("href", "")
Dim price As String = listing.SelectSingleNode(".//span[@class='s-item__price']").InnerText
Dim details As String = listing.SelectSingleNode(".//div[@class='s-item__subtitle']").InnerText
Dim sellerInfo As String = listing.SelectSingleNode(".//span[@class='s-item__seller-info-text']").InnerText
Dim shippingCost As String = listing.SelectSingleNode(".//span[@class='s-item__shipping']").InnerText
Dim location As String = listing.SelectSingleNode(".//span[@class='s-item__location']").InnerText
Dim sold As String = listing.SelectSingleNode(".//span[@class='s-item__quantity-sold']").InnerText
Console.WriteLine("Title: " & title)
Console.WriteLine("URL: " & url)
Console.WriteLine("Price: " & price)
Console.WriteLine("Details: " & details)
Console.WriteLine("Seller: " & sellerInfo)
Console.WriteLine("Shipping: " & shippingCost)
Console.WriteLine("Location: " & location)
Console.WriteLine("Sold: " & sold)
Console.WriteLine(New String("=", 50))
Next
End Module