Scrapebox not returning results from Google [Solved]

I've been tinkering with Scrapebox in the past but tbh I only scratched the surface what what this thing can do. So I decided to pick this up again and level up.

‍

The more tuts I watched, the more I realized that this is one hell of a tool that ought to be part of every SEO professional out there if you ask me. Plus, it still comes with a perpetual license - not en vogue these days but I love it! Sure, the interface might seem a bit dusty by today's standards, but don't be fooled by its appearance - there's plenty under the hood! 

‍

‍

Proxies

Scrapebox without proxies is pretty much useless. Sure, you can harvest public proxies but I found these to not be worth my while - either they don't work at all or they are so overused that your IP gets block in an instant.

‍

ScraperAPI

Couple of folks on BHW recommended this service for Scrapebox proxies so this was my go-to solution for a couple of months now. Gotta say, it really works like a charm and the integration is as easy as it gets but I found it to be a bit costly over time. The free plan is great if you only use Scrapebox once in a while but if you want to use it somewhat regularly, you'll find that you burn through your credits quite fast. And for me personally, the first paid tier (hobby plan - $49/m at the time of writing this), does not make that much sense as I mostly use ScrapingBee on a day-to-day.

‍

ScrapingBee

This one is my all-time favourite, particularly because it comes with pre-built Integromat modules. ScrapingBee's web scraping API not only handles headless browsers but it also automatically rotates proxies for you. It took me a while to figure out that I could 'just' use their proxy mode for Scrapebox as well - therefore making ScraperAPI redundant and saving 49 bucks per month.

Barking up the wrong tree

The problem was I couldn't get it to work. ScrapingBee's customer service is exceptional and they pointed me in the right direction (adding params like render_js=False and custom_google=True) but it still wouldn't work. I almost gave up on this. But then I stumbled upon a forum post on BWH where somebody described the exact same problem that I was seeing - not getting any results from Google or in some cases on page 1 results while Bing was working just fine. I learned that it wasn't the proxies - it was Google! 

‍

Solution: add a new custom engine

I learned that Google have slightly changed the HTML of their results pages, which unfortunately breaks Scrapebox's parser. I'm sure that Scrapebox already have or will soon fix this but in the meantime, here's a solution to make it work again: 

1. Go to Settings >> Harvester Engines Configuration
2. Change the Just before the url [...] parameter to <a jsname="UWckNb" href="
3. Change the Marker for "Next page" parameter to aria-label="See more"
4. Change the Display name to something else (e.g. Google new)
5. Click Add as new engine

How to update Scrapebox

Instead of adding a new custom engine, you can probably also solve this by updating Scrapebox to a newer version (I'm assuming Scrapebox have already fixed this by now). Luckily, the great Loopline already made a video on how to do this. Check it out 👇

‍