I am looking to move beyond manual SEO audits. I want to build a Python script that can crawl a site, extract all H1 tags, and then use a machine learning library to automatically categorize the search intent (Informational vs Transactional) of the ranking keywords. Which libraries are best for this, and can I integrate this directly with the Google Search Console API?
3 answers
For the crawling aspect, Scrapy is much more robust than Beautiful Soup for large-scale audits because it handles asynchronous requests and middleware out of the box. For intent mapping, I’d recommend using the spaCy library for Natural Language Processing or even a pre-trained BERT model from the transformers library if you want high accuracy. You can definitely connect to the Search Console API using the google-api-python-client. This allows you to pull click and impression data directly into a Pandas or Polars dataframe, join it with your crawled data, and run your intent classification. This workflow turns a three-day manual audit into a ten-minute automated script.
Have you looked into the "Advertools" library by Elias Dabbas? It’s specifically designed for SEO and has built-in functions for crawling and API integrations that might save you from writing everything from scratch.
Python is a game-changer for SEO. Just make sure you follow robots.txt protocols in your script so you don't accidentally get your IP blocked during a crawl.
Good point, Susan. I always add a DOWNLOAD_DELAY in my Scrapy settings. It's better to be a bit slower and respectful than to crash a client's server during an audit!
Thomas, I’ve heard of Advertools but I was worried it might be too high-level. If I want to build a custom dashboard in Streamlit to show these results to clients, does Advertools play well with interactive visualization libraries like Plotly, or should I stick to raw data extraction with Scrapy?