In 2021, scraping Linkedin for profile URLs, or buying them from a third party vendor, is a common tactic for most digital marketers. Unfortunately, buying the data is expensive and scraping lists yourself puts your own LinkedIn profiles at risk and does not scale.
Enter Bing. What most marketers don't realize is that all of this data is already available for free, without a login wall, and in almost unlimited quantities in Bing’s search index. Because Microsoft owns both LinkedIn and Bing, Linkedin is seemingly perfectly indexed in Bing's search and, unlike Google, Bing has few limitations on scraping their search results.
When scraping Linkedin directly, you are limited to one thousand profiles per day and can scrape roughly 80 of those for names, job titles and other information without getting your profile banned. However, by using Bing, we can instantly pull millions of profiles.
Searching for Profiles
First, query indexed LinkedIn profiles using Bing's search operators. Each search will start with site:linkedin.com/in in order to focus on LinkedIn alone. Then, to filter LinkedIn profiles by profession and location, we use logical operators. With Bing, this looks like "CTO" AND "San Francisco", and will match profiles that mention both terms. You can see this example search here and you can learn more about Bing's search operators here.
You will notice that this particular query has over two million results, but also that we can only access one hundred pages, totalling one thousand profiles. In order to overcome this, we will need to create search queries in bulk in Google Sheets.
If you are looking for any CTO in the US, you can take the beginning of our query, (site:linkedin.com/in "CTO" AND) and then create new queries by appending every city in the US to the end in quotes. Below I have created a sheet with our query in column A and a list of cities taken from wikipedia in column B. You can join the two columns with a formula in sheets or using an addon like Power Tools.
Scraping the results
If you aren't a developer, you can use NetPeak's SEPR scraper to pull the results from Bing. All you will need to do is upload your list of queries and Netpeak will handle the rest. You can find a detailed tutorial here. If you are a developer and want an easier way to scrape searches at scale, SerpStack has a great SERP API.
Depending on the scale at which you are scraping search results, you may want to run more instances of Bing concurrently to speed things up. In this case, you will need to purchase proxies to run multiple threads from unique IP addresses. This has the added benefit of ensuring that your searches aren’t rate-limited by Bing. I use MyPrivateProxy and have been able to run upwards of 200 instances delivering millions of results per day using ten of their private proxies.
Once you have your resulting list of scraped profiles in a CSV, there are plenty of tools you can use to further enrich them, adding data likeemails, locations, full names, job titles, and more. My favorite is Snov.io. To find work and mobile phone numbers I use SalesQL. If you want to avoid paying for enrichment, you can also pull most of this data, not including emails, out of the description section from the SERPS themselves which Netpeak will return.
