Learn how to scrape Google search results, the ethical way using the Google Search API. This article provides step-by-step instructions on setting up the API, obtaining an API key, and writing Python code to fetch and extract data from search results.
Before starting the basic tutorial to scrape Google search results, we first know why actually we require to scrape Google search results. Listed below are some of the major usages and needs for the users to scrape Google search results. This will help you in getting the exact application to Scrape Google search results.
Why do we require to scrape google search results.
There can be various legitimate reasons why someone might need to scrape Google search results. Here are a few examples:
Research and Analysis:
Scraping Google search results can be useful for conducting market research, competitive analysis, or gathering data for academic or scientific research. It allows you to analyze trends, identify patterns, and extract insights from the search results.
SEO Analysis:
Scraping Google search results can provide valuable data for search engine optimization (SEO) purposes. By analyzing the search results, you can examine keyword rankings, competitor strategies, featured snippets, and other factors that impact organic search visibility.
Content Monitoring:
Monitoring search results for specific keywords or phrases can help you keep track of industry trends, brand mentions, or online reputation management. By scraping the search results, you can automate this monitoring process and capture relevant information in a structured manner.
Data Aggregation:
Scraping Google search results can be part of a broader data aggregation strategy. For example, if you’re building a directory or a specialized search engine, you may need to gather data from various search queries to create a comprehensive database.
The best and ethical way to scrape Google search results is to use the Google Search API. Google provides an API called Google Custom Search JSON API, which allows you to programmatically fetch search results. For this we would require obtaining an API key and configure the API to access search results.
Setting up Google Custom Search Engine
Google Programmable Search Engine (formerly known as Google Custom Search Engine) is a service provided by Google that allows users to create customized search engines for their websites or applications. It enables you to add search functionality to your website by creating a search engine tailored to your specific needs.
With Google Programmable Search Engine, you can define a specific set of websites or pages to search, prioritize or exclude certain content, and customize the look and feel of the search results. It provides an API that allows developers to programmatically interact with the search engine and retrieve search results in various formats.
Visit the Google Custom Search Engine website (Programmable Search Engine by Google) and click on “Get started” button. After that use your google account to sign in.
After successfully login in into the Google Custom Search Engine website you will get the interface as shown in the picture below. Here you have to Create a new search engine in Programmable Search Engine by Google. Enter the Name of your search engine, select Search the entire web option and click on ‘Create‘ button.
Once done you will get your new search engine created a code provided for you which you can copy paste. 
After this you have to click on ‘Customize’ button which will take you to the Overview page which will show you the Basic credentials of the new search engine which you have created. Here you will get your Search engine ID for the new Search engine which you have just created. 
Take a note of the Search engine ID, we will be using this in the coming code to Scrape Google search results.
Obtaining an API Key from Google Cloud Console
Google Cloud Console is a web-based management interface provided by Google Cloud Platform (GCP) that allows users to manage and interact with their cloud resources and services. It provides a centralized platform for managing various aspects of your cloud infrastructure, including computing, storage, networking, and data analytics.
Google Cloud Console provides a user-friendly graphical interface that simplifies the management of your cloud resources. It also offers command-line tools and APIs for programmatic access and automation.
Through the console, you can monitor resource usage, configure settings, view logs and metrics, provision new resources, and perform administrative tasks to manage your cloud environment effectively.
With Google Cloud Console we will create a Project and will enable the Goggle Custom Search API for the project. Now follow the following steps to generate an API key for your project.
Visit Google Cloud Console and it will ask for your google account for login. Enter your credentials and you will be presented with the following interface.
Next Click on APIs and Services under Quick access tab as shown in the figure below:
Now click on ENABLE APIS AND SERVICES as shown in the figure below:
Next scroll down and click on Custom Search API as shown in the figure below:
Now click on Start button and there after click on Manage button as per below figure:
Now click on Credentials then CREATE CREDENTIALS and click on API key as shown in the figure below
This will create an API key. You have to note it down as we are going to use this API key credential in the Python code.
Python code to Scrape Google Search Results
Install the requests library using pip:
pip install requests
The complete python code to scrape Google Search results and exporting the results into an Excel Sheet goes here.
# Importing the requisite modules for our project.
# Requests for sending requests to a webpage and getting the response.
# Pandas for storing the scrapped data and exporting the data to Excel Sheet.
import requests
import pandas as pd
# Set up the API key and search engine ID provided by Google.
api_key = 'AIzaSyCTMgngC5PcVC62DnDGhnRfOpQD_cWy_UA'
search_engine_id = '203f267ecb8304793'
# Set the search query.
query = 'wowpython'
# Make a GET request to the Google Custom Search API.
url = f'https://www.googleapis.com/customsearch/v1?key={api_key}&cx={search_engine_id}&q={query}'
response = requests.get(url)
# Blank Lists for storing the Scrapped data.
titles = []
links = []
snippets = []
# Extract the search results from the response.
if response.status_code == 200:
search_results = response.json().get('items', [])
for result in search_results:
title = result['title']
link = result['link']
snippet = result['snippet']
titles.append(title)
links.append(link)
snippets.append(snippet)
else:
print('Error occurred while fetching search results.')
data = {
'Title': titles,
'Link': links,
'Snippet': snippets
}
df = pd.DataFrame(data)
excel_file = 'search_data.xlsx'
df.to_excel(excel_file, index=False)
print(f'Data scraped successfully and saved to {excel_file}.')
Replace 'YOUR_API_KEY' and 'YOUR_SEARCH_ENGINE_ID' with your actual API key and search engine ID obtained in Step 2. Set the 'Your search query' to the desired query you want to search for.
Scrape Google Search Results Python Code Explanation
We have used requests library for sending requests to a webpage and getting the response.
We have made a GET request to the Google Custom Search API with the following code:
url = f’https://www.googleapis.com/customsearch/v1?key={api_key}&cx={search_engine_id}&q={query}’
response = requests.get(url)
The above code will fetch search results in JSON format, and extract information such as the title, link, and snippet from each search result.
Thereafter we have looped through the search results and stored the extracted data in the Pandas DataFrame. With Pandas dataframe it is easy to interact with Excel and store the data. After extracting the data and storing the same in the Pandas DataFrame we have exported the stored data into an Excel Sheet search_data.xlsx.
You can experiment with different search query as per your requirements and can store the extracted data into different Excel files as per the search query. This will help you to get the deeper insight into Scrape Google Search Results for your search query which you have provided.
This example demonstrates how to make a request to the Google Custom Search API using the requests library, fetch search results in JSON format, and extract information such as the title, link, and snippet from each search result.
Note: For making a request to the Google Custom Search API there is price for each request which is charged by Google. There is limit for the Free usage and also after exceeding the particular limit you will be charged. So, before usage kindly check the pricing before making regular calls to the Google Custom Search API.
There are also our other blog posts wherein you can learn more about Web Scrapping, follow our post below: