I have been missing a convenient tool for Google search results scraping ever since Google redesigned the output screen. The tool’s need arose from Google switching the output format to scrolling with “more results” from the previous convenient page-by-page view. The change broke the beloved Instant Data Scraper.

Julia Tverskaya is our long-term Partner, Sourcer, Recruiter, and a good friend. She also writes code! And now Julia has created a Chrome Extension that does even better than automatically pressing the “more results” button.

(It is her second published extension. Julia’s Email Extractor, the first extension she coded, now has 9,000 users. The Extractor uniquely follows the user on all open pages, collects every email from the pages’ HTML, and stops at a user’s signal.  Its parsing formula is solid. It also cleans up the list: deduplicates, alphabetizes, and removes non-personal and junk emails. I use it regularly for sourcing. Our sourcing tool Social List is Julia’s creation as well.)

The brand new Julia’s second extension is

Google Search Results Scraper.

The tool relies on the trick described in the post Get Back More Google Results By Not Using More.

What the new Google Search Results Scraper does:

• Collects Google search results for a query, as many as possible
• Increases the max number of results returned to 300-400 from the current 100-130
• Exports results in Excel in this format:
  • URL
  • Title
  • Snippet
  • Emails (if found)

Get it here – and we would be glad to hear from you!

Please also share with colleagues who may benefit from the tool.

 

 

 

 

LinkedIn tells us we are moving toward skills-based hiring and stresses the updated skills search that is smart about the content of members’ profiles and can guess skills. Is that so? Please review LinkedIn Wants Me to Pack Groceries and Learn Fluoroscopy if you haven’t. It’s a (sad) joke.

LinkedIn Engineering observes that Recruiters find and contact only a small percentage of members. The idea to expand Skills beyond what members enter comes from their wish that we find others. (But we would find others if not for the issues described in LinkedIn’s Members and Companies Disconnect, You Are Missing Seniority, LinkedIn, and other similar cases where LinkedIn misinterprets profile data!)

So here is how the “new,” wide skills are assigned, per LinkedIn Engineering:

“Please don’t confuse skills with “that list of skills you entered on your LinkedIn profile”.  Yes, of course, we use those, but we intuit skills from MANY places.  For example, we will give you “credit” for a skill if you (these are examples and not exhaustive):

• Have it explicitly listed as a skill on your LinkedIn profile (what you were talking about)
• Have it in your profile some place.  We actually convert all the text in your whole profile into one big field behind the scenes and use it to keyword search for skills.  The goal is to give people “credit” for skills they likely have, even if they didn’t think to explicitly list it.  We don’t want everyone having to do some crazy optimization of entering every keyword they can think of.  That sucks.  We want to give people credit for skills.
• Have it in a resume we have access to (this is permission dependent).  We scan resumes for skills. 
• We also “give credit” for skills based on connections at times.  So, let’s say you have a bunch of connections, and you are all “similar” profiles, and all those other folks have X skill and you don’t.  We will “infer” that you have that skill.”

What happens in practice is that the Skill search more or less equals the keyword search. As an example, it would find an experienced Developer for “Java” which she used in college years ago. Or she didn’t, but her connections did!

Additionally, Skill search in Recruiter has a unique syntax (why?). Looking for clinical therapy is not the same as looking for clinical AND therapy.

However, there still is a way to search by self-entered skills, via search  operators.

Join us for the class

LinkedIn Recruiter Mastery

coming up on November 29, 20, 2023, and learn all about the tool that the LinkedIn documentation does not have.

At the beginning of this AI wave, services popped up that offered search engines and AI “integration,” like this:

There is no integration here other than combining results on the same screen. However, the latest developments in search-the-web-plus-AI tools are promising. It was welcome news that Bing is back with ChatGPT and the GPT-4 data has been updated to 2023. (I hope this stays in spite of the weekend news!) Check out our AI Chatbot Comparison Chart.

Algorithms behind major search-engine-AI integrations are complex and, to a large extent, unknown. Practice, discussions, and trusted newsletters will help us navigate them. But, say, ask Bing to source profiles of people with certain skills and experience, and it will, in most cases, either refuse or hallucinate. (Most AIs do not feel enthusiastic, particularly, about reading LinkedIn profiles.)

However, there are ways to combine search engines’ results and AI “on your own,” having clarity about the process. It is similar to scraping and processing search results but can be much smarter and faster with the right prompts. The approach can be customized in many concrete circumstances.

Utilizing AI and Search Engines in Candidate Sourcing

Search Engines: They provide an extensive range of basic candidate information, pulling from professional networks, publications, and online profiles. Their utility lies in initial data aggregation.

AI Tools like GPT: These tools offer a nuanced analysis. AI interprets and evaluates a candidate’s professional nuances, assessing skills and cultural alignment more deeply than surface-level data.

Integrated Approach: Begin with search engines for broad data collection. Then, apply AI for detailed analysis. For instance, in technical roles, search engines identify candidates with relevant backgrounds, while AI evaluates project intricacies and technological adaptability.

Summary: Search engines offer volume; AI provides insight. Combined, they form an effective strategy for informed candidate sourcing.

Our ChatGPT webinars have been a continued success. Please do not miss the next one –

ChatGPT and AI for Sourcing and Recruitment

December 20-21, 2023. Materials are provided; seating is limited.

GPTs are fascinating and promising. I want to share my experience creating one.

I regularly use SalesQL batch import of LinkedIn profiles and get a rich output where each record has detailed info about the members. The other two systems I use for the rich output are SeekOut and ContactOut. These outputs have much more detail than export from LinkedIn Recruiter.

Creating the “Batch Match” GPT, I told it that I would give it a job description and a list of potential candidates (which I would export from SalesQL), and wanted to see a table with assessments and scores.

I picked a job description from LinkedIn for a Unicorn Hunter (Corporate Recruiter) in Automation Machinery Manufacturing. I also gave BatchMatchGPT a SalesQL export with a list of profiles of Recruiters. The GPT replied: “The assessment will focus on their alignment with the key requirements of the job, which include experience in in-house recruiting, expertise in the industrial automation sector (if any), proficiency in various recruitment strategies and tools, and strong interpersonal and communication skills.” It sounded reasonable.

Here is what it gave me based on a job description and profile list – by all means, useful and time-saving:

GPT’s “understanding” of the job requirement was adequate. You can see in the table how it tried to derive info, even the soft skills, from the input. I could further refine the output by setting the column configuration.

To become an everyday tool, the GPT needs more corrections, rule setting, and debugging. This would probably take a few hours. I will take this route and hope to make BatchMatchGPT public. (You can also try to create and test your own, following the description.)

ChatGPT is always a bit stubborn. The GPT did not want to read the job description from the link, though it previously read my blog with no problem. It wouldn’t read profiles from URLs either. This could be a specific LinkedIn limitation imposed by Microsoft. The GPT also told me it couldn’t access external files when I imported an Excel file. But when I converted it to PDF, it digested the info.

Please join us at

“ChatGPT and AI for Sourcing and Recruitment”

November 15-16 (Wed-Thu), 2023, and learn to up your game in the new world of AI.

 

Search engines and LLMs work on very different principles. Tech companies have tried to marry the two, but integration is challenging.

I have just watched this video (recommended!) about Bing that sheds quite a bit of light on how the search engine and LLM integration works in Bing. The pic above is from the video and shows a diagram of the search engine-AI interaction ruled by the Orchestrator.

The “Sydney Orchestrator” (Or Bing Orchestrator) in Bing integrates ChatGPT-like AI into Microsoft’s search engine, enhancing Bing’s capabilities with GPT. Here’s how it works:

Query Processing and Iteration: There is “prompt chaining” in the background. The Orchestrator processes search queries iteratively. It generates internal queries, refining them each time to get more accurate and contextually relevant search results.

Information Grounding and Relevance: “Grounding” is to deal with AI hallucinations. The Orchestrator ensures that responses are relevant and current, crucial for recent or time-sensitive queries. It also generates iterative queries, broadening the context for the GPT model and providing a more comprehensive understanding of the subject matter. — The “precise” search option in Bing is for grounding.

Coordination Between GPT and Bing’s Indexing: The Orchestrator is an intermediary between GPT’s language processing and Bing’s search indexing.

Join us this week for the always-popular, constantly updated class

“ChatGPT and AI for Sourcing and Recruitment”

November 15-16 (Wed-Thu), 2023.

GPTs is wonderful AI news. Expect even more changes!

Full disclosure: I did not write or edit the post in italics below. I created a GPT, sent it to read this blog, and then asked to write a post related to AI and talent sourcing.

The GPT called itself “Blog Composer,” provided me with the picture above and wrote a post, entitling it

Combining AI and Web Search in Talent Sourcing: Proceed with Caution

Google’s latest tweak, pushing terms in quotes into snippets, is a boon for researchers. But when we integrate AI into talent sourcing, it’s a different ball game.

I recently tried using AI to sharpen my search strings. While it did refine the terms, I noticed a pattern – AI often overreached, suggesting related skills that weren’t always relevant. It’s a reminder that AI can guide, but can’t replace, the nuanced understanding of a skilled Sourcer.

In sourcing, precision is key. AI’s suggestions? Take them with a grain of salt and always double-check.

I thought it did rather well given only 5 minutes we had worked together. What do you think?

Join us at the updated, always popular session

ChatGPT and AI for Sourcing and Recruitment

November 15-16 (Wed-Thu), 2023.

 

 

Image by #Midjourney (which cannot write yet)

LinkedIn is going to lead its members in finding relevant jobs. I want to show you what I received yesterday from LinkedIn in that regard. It would be hilarious if it weren’t so sad!

Here is my profile – https://www.linkedin.com/in/irinashamaeva. Recently I have been into AI drawing with Midjourney as a hobby, and added it to my profile:

At that point, I got the industry “Graphic Design” in LinkedIn’s thinking.

Now for the email:

Is this good news for me? “Graphic Designer” has been quietly converted into “Image Specialist,” which is not a Graphic Designer, and can mean many things, including Radiology.

The most interesting part! In spite of the jobs uptick, we Image Specialists apply to pack the groceries at Whole Foods. We also assist with errands at MAC Partners, but this particular job is closed. I suppose these are my job recommendations if I want to follow the majority of fellow Image Specialists:

And then, more information is added. Perhaps this is even more striking! I have the X-Ray skill, in a sense, lol. But I have not thought of learning Fluoroscopy:

Going to the first job recommendation, I got one job to apply for – Chief Technologist. Unfortunately, it is onsite in IN while I am in CA; also, I don’t match it:

The second job link returned no results – and all of a sudden I am seeing remote Recruiter jobs (which would have been appropriate to start with):

This is messed up in every step.

We have learned some unexpected things about searching for LinkedIn members by Company, Company Size, and Type. It’s good to know what to do about those.

Let’s continue examining LinkedIn filters. Consider this search in LinkedIn Recruiter using operators:

• Current Job Title = director NOT (seniority:1 OR seniority:2 OR seniority:3 OR seniority:4 OR seniority:5 OR seniority:6 OR seniority:7 OR seniority:8 OR seniority:9 OR seniority:10)
• Years at Company > 1

The search finds people with the word director in the current job title and no Seniority assigned.

Among results, you will see members who should have a seniority value (of “Director”,) like these. But you will not find them by any level of seniority including “Director.”

Try a similar search with the word “Senior” instead of “Director” and encounter missing results again. There are 510K+ people with the word “Senior” in the current title, over 2 years at the current company, and with no assigned level of Seniority. Here is an example – the member should have a “Senior” level assigned, but he doesn’t:

Conclusion – in addition to what my previous LinkedIn search post says:

Do not use Seniority in most of your searches.

In fact, any value selection in LinkedIn people search cuts your results in half or more! I will continue the series to cover other filters.

Please join us at the expanded two-part class

How to Find Hidden LinkedIn Profiles

on November 8th and 9th (Wed-Thu), 2024. Materials are included; seating is limited.

 

image created with DALL-E

Did you realize that since Google had replaced the convenient multi-page display with the "More results" button, its users have encountered fewer results? If previously we got up to ~300-450 results, now, by pressing the "More" button until there is nothing left, gets us no more than 130-140. (Searching using quotes and operators, or Verbatim does not help). We have tested that.

To get back the results you no longer see from the "official" UI, do not use the "More" button. Instead, set your preferences to 100 results per page and add these to the search URL in turn (until there are no more results):

• &start=100
• &start=200
• &start=300
• &start=400

For example,

• "Rochester, NY" geriatrics site:doximity.com
• Link 2
• Link 3

The pages appear with different sets of up to 100 records. You can now collate them!

An even faster option is to use this script "Google Search Results Expander" from Mike Santoro (right-mouse-click/Inspect/Console/paste the script):

javascript:(function() {
var query = prompt("Enter your Google search query:");
if (query !== null) {
var base_url = "https://www.google.com/search?q=" + encodeURIComponent(query);
var urls = [
base_url + "&num=100&start=0&filter=0",
base_url + "&num=100&start=100&filter=0",
base_url + "&num=100&start=200&filter=0",
base_url + "&num=100&start=300&filter=0"
];
urls.forEach(function(url, index) {
setTimeout(function() {
window.open(url);
}, index * 2000);
});
}
})();

Since the Google UI change, Instant Data Scraper no longer extracts all results out-of-the-box, but you can open the tool over these pages and merge the lists of results.

Note that registration is open for the 7-Day Sourcing Bootcamp that David Galley and I will hold together, starting January 2nd - register here early:

Seven-Day Sourcing Bootcamp for Recruiters - January 2024.

We are covering all aspects of sourcing, providing materials, and support for the attendees.

Feedback has been excellent and the Bootcamp has proven to be popular, so we will likely continue updating and scheduling it 2-3 times per year.

Sourcing Summit Europe-2023 was awesome in every way – just look at all the reports on #sosueu on LinkedIn. It’s hard to describe how wonderful the event feels; you must go there!

My presentation was  called “Unscrambling LinkedIn for More Efficient Sourcing.” As I was preparing the slides, I ran into several problems with LinkedIn Recruiter search, and I want to tell you about them.

What this post is about:

TL;DR LinkedIn Recruiter does not find many people whose profiles do point to their current company pages by:

• Company selection
• Company size
• Company type.

Consider this search: the negated company selection for Exxonmobil and Boolean (text) search for exxonmobil in the company field.

If you open the first profile in the results, you will see:

There is a current clickable company pointing to Exxonmobile’s LinkedIn company page. But LinkedIn fails to recognize that the member works at this company if you select it in the search dialog. This person will NOT be found by any company size or type either.

The majority of the 4K+ results for this search are also “lost souls;” only some work at Exxonmobil country offices (and at least have company sizes).

Take Ford Motor Company (depicted above with Midjourney) and you will see a similar picture. Profiles of many of its employees are not found by the company selection (or type, or size).

LinkedIn does a poor job categorizing its data. It means that Boolean rules – it does find Ford employees by the company name.

Bottom line:

1. Search for companies not by selection but by Boolean (text).
2. Do not use company size or type in most of your searches.

What does this mean for LinkedIn? Engineering wants us to find more profiles. They have redone the Skills search in Recruiter in a manner that makes Skills similar to keywords (and less useful). They could achieve the goal instead by closely examining their data and restoring the data connections that should be there but are not.

(I have discovered other things, along the same lines but different filters, and will post soon.)

To proceed with the promised AI “2024” search, it would be best if LinkedIn worked on better understanding their data first.

Speaking of AI, you are invited to the updated class “ChatGPT and AI for Sourcing and Recruitment” on October 18-19.