Website Scraping Is an Easy Growth Hack You Should Try As one of the oldest tricks in the digital marketing book, website scraping is still a premiere way of getting data.
By Andrew Medal Edited by Dan Bova
Opinions expressed by Entrepreneur contributors are their own.
Website scraping has been a growth hacking technique since well before the term growth hack even existed. Starting out from simple copy and pasting information from one page to a spreadsheet or database has now transformed into a tried-and-true strategy.
Web scraping is a method of extracting data from a website. This can be used for a number of different reasons, including building up a sales pipeline to determining where your competitors are setting their prices. Even as it's considered an age-old practice (at least, in internet terms), it can also be an excellent way to spark growth. However, before we dive into web-scraping methods, let's explore how web scraping first got on the digital marketing map.
Related: 5 B2B Growth Hacking Tools for Your Business
Even though web scraping is a widely practiced facet of digital campaigns, it has had a bumpy history. After all, regardless of whether you're using bots to scan pages or even just copying the important data snippits, you're still taking information that may or may not be OK to use (despite its public status).
Where is the web-scraping line?
Perhaps one of the first examples of the possible illicitness of web scraping is the case of eBay vs. Bidder's Edge. In the early 2000's Bidder's Edge was a data aggregator for auction sites, with eBay being one of its primary sources for prices. While eBay was aware of Bidder's Edge scraping the site for prices, it eventually grew to a point where Bidder's Edge was using so much data that it was disrupting eBay's servers. The court essentially ruled that because Bidder's Edge disrupted eBay's servers, that caused a loss of revenue, making it liable. Yet the actual practice of screen scraping was deemed OK.
The ruling set a precedent that has given way to countless growth opportunities for companies spanning industries. In my opinion, website scraping is still one of the most ethical forms of growth hacking. It's a tried-and-proven strategy that dates back to Web 1.0, and is more effective than ever.
Related: SEO Growth Hacking Techniques to Scale Your Business
The overall practice of it has been questioned in courts for a number of years, but luckily, we've established somewhat of a status quo on its legality. According to Icreon, some basic tips to keep in mind include looking out for copyrights, not being in violation of privacy laws or terms of use, and (like in the case above), doesn't burden the host's service.
How can you incorporate web-scraping?
Now that we've gotten into what exactly is allowed, let's get to the fun part: the actual scraping. For starters, one of the most common uses is setting up a robot.txt file. These essentially tell a web crawler what to look for on a page. For example, if I was a sneaker reseller and a new Jordan just released, I can tell a robot.txt to go through other stores (eBay, StockX, etc.) and pick out terms such as "Jordan," "Air Jordan" and whatnot to aggregate the price.
This method doesn't require nearly as much coding as you think and can be an excellent source of getting the info you need quickly. However, if you're somebody that doesn't know how to code (or wants to learn), there are some surefire ways to scrape without ever learning a lick. And no, it's not copying and pasting.
As the practice of screen scraping has become much more ubiquitous, a lot of firms have been delivering some great products along the way to help out. Platforms such as Parsehub allow you to open any web page and extract the data you need into one place, and its free version can be a solid introduction to get your feet wet. Additionally, Import.io is also an excellent choice, but I'd recommend trying a few different methods before committing to a paid service. Remember, this is about saving money and time, so finding a balance is key.
Related: Use This 1 Marketing Tactic If You Want to Double Your Business Overnight
What is the future of web-scraping?
The capabilities of using web scraping in your data mining could be endless. In fact, the growth of collecting big data has spawned how AI can be used to assess the relationship between data points. And as most of us have heard, AI is changing the way we look at marketing in a significant manner.
While most of us have an array of needs when it comes to the information we collect, this practice can be a quick way to get a competitive advantage. And in such cutthroat industries, who doesn't want to scrape together an edge?
