Keyword research is actually more useful when you stop treating it as a process for finding individual keywords to optimise individual pages for. Instead, keyword data can act as a guiding star for understanding what people are searching for, which topics a website should cover, and whether the site architecture actually supports those topics.
This becomes particularly interesting on large websites.
The question is no longer simply: Which keyword should this page rank for?
It becomes: What does the market search for, which parts of that demand are relevant to us, where do we currently cover it, and where are the gaps?
That is a much larger analysis problem. It is exactly the kind of problem where MCP servers and tools such as Claude Code become very useful.
Keyword research has changed
Traditional keyword research often starts with a keyword or a group of keywords. You collect search volumes, classify search intent and map each keyword to a target URL. From there you identify missing pages, optimisation opportunities and possible cannibalisation.
There is nothing inherently wrong with this approach. The problem is scale.
If you have thousands of keywords and hundreds or thousands of pages, someone still has to understand what the keywords mean, understand what the pages are about and decide how they should be connected.
Even with custom tools that combine data from several sources, a large part of the actual analysis has historically remained manual and can take weeks to finalise.
This was one of the first things I wanted LLMs to solve
When LLMs started becoming useful, large-scale keyword research and enterprise keyword mapping was one of the first use cases I investigated because the idea looked obvious. Give an LLM a large keyword set and a list of URLs and ask it to understand both sides and map them together.
I quickly realised that in practice, the models and tooling were not really ready for it. The model might understand a few pages and keywords very well, but working reliably across a complete website was something else entirely. Context limitations, access to external data and orchestration quickly became problems.
That is what has changed with Claude Code and MCP servers. The LLM does not have to work from one enormous prompt. It can work with files locally, query external systems when needed, inspect individual webpages and build up the analysis step by step. That makes the original idea much more realistic.
Give the LLM access to the same data an SEO would use
My current approach is to give Claude Code access to several different sources.
For example:
- Google Search Console through an MCP integration/API, or preferably a BigQuery export for larger datasets from GSC and GA4.
- GA4 data for additional information about how pages are actually being used.
- Keyword data from a keyword research platform through an API or MCP integration.
- Alternatively, a local export of the relevant keyword dataset.
- The website itself, so Claude can inspect the actual pages rather than relying only on URL strings or existing metadata.
The important part is not the specific tools. The important part is that the model can connect the different datasets during the analysis.
For every URL, it can inspect the page and try to understand what the page is actually about. It can then compare that with what the page currently ranks for in Search Console. GA4 can add another layer of context around how the page is being used. Keyword data adds the wider search market.
You are no longer asking the model to analyse a spreadsheet containing two columns called keyword and URL. You are giving it evidence.
Mapping keywords becomes a reasoning problem
This is where things become much more interesting.
A keyword should not be mapped to a page simply because the words appear on that page. Claude can investigate the intent behind the query, compare it with the purpose of the page and decide whether the relationship actually makes sense.
For example, imagine a website selling robotic lawn mowers.
Queries around:
- robotic lawn mower
- best robotic lawn mower
- how robotic lawn mowers work
- robotic lawn mower installation
- robotic lawn mower for small gardens
might all belong to the same broader topic. But that does not mean they should all have the same target URL. Some might primarily belong to a category page. Others belong to buying guides, support content, product pages or informational articles. That distinction matters.
One keyword does not necessarily mean one page
I also think the traditional idea that a keyword should always have one canonical target page is becoming less useful as the months go by.
- A topic can legitimately be covered across several parts of a website.
- A category page might address the commercial intent behind a query.
- A buying guide can explain how to choose between products.
- A product page can cover the attributes relevant to a specific product.
- A support page might answer questions that appear after the purchase.
All of those pages can be relevant to the same broader search topic without necessarily competing with each other. Of course, that does not mean we should deliberately use the same keywords in the key areas of the on-page content across every page. But having the language users actually use appear naturally when describing a product feature, product attribute, use case or problem is still valuable, especially from an LLM visibility perspective.
The role of the keyword is different for each page type and that becomes even more relevant when thinking about AI search.
If we want search engines and AI systems to understand that a website has genuine depth around a subject, looking at coverage across the complete site becomes more useful than simply asking whether one page contains the correct keyword.
From keyword mapping to site architecture
This is probably the biggest change for me. Once the model understands the keyword dataset, the existing pages and their relationships, keyword research starts becoming an input into site architecture.
Claude can start answering questions such as:
- Which important topics do we not cover at all?
- Which topics are covered only superficially?
- Which URLs appear to target essentially the same purpose?
- Where does search demand suggest that a separate page could be useful?
- Are important topics buried in pages that serve a completely different purpose?
- Does the current information architecture reflect how users actually search for the subject?
- Are there clusters of related queries that should be represented by a group of connected pages rather than one page?
This is much closer to how I want to use keyword research today. Not as a list of phrases that need to be inserted into pages but as a dataset that helps describe the information landscape around a website.
The LLM still needs boundaries
I would not simply connect Claude to a few APIs and let it redesign the website.
The quality of the output still depends heavily on the rules you give it. Page types need to be understood, brand terminology matters, business priorities matter and search volume is not the same thing as business value.
The useful part is that a lot of the mechanical analysis can now be delegated. Instead of spending hours manually comparing keyword lists, URLs, rankings and search intent, I can spend more time reviewing the decisions that actually matter. That is a much better division of labour.
Keyword research is becoming an analysis problem
I do not think AI makes keyword research obsolete, but I think it changes what keyword research is for. The interesting question is no longer whether we can use an LLM to generate another list of keywords, as we already have plenty of ways to generate lists.
The interesting part is giving the model enough context to understand the search market, the website and the relationship between the two. With local files, MCP server, APIs and tools such as Claude Code, that is finally starting to become practical. For large websites, I think that changes keyword research quite significantly.
The road to an autonomous always-on keyword research system
The next step is creating an automated, always-on keyword research system.
Instead of running a large keyword research project once or twice a year, the system could continuously monitor which queries webpages are becoming visible for, which new search terms start appearing in Search Console and where the relationship between search demand and the existing content starts to change.
It could identify pages that begin ranking for topics they were never intentionally created for. It could detect emerging clusters of queries before they become large enough to show up in a traditional keyword research process. It could also flag situations where several URLs gradually start competing for the same type of search intent.
The system could then compare those signals with the actual content and suggest where an existing page should be expanded, where a different page should become the primary page for a topic, or where the site simply does not have suitable content yet.
At that point, keyword research stops being a project.
It becomes a continuously running system for understanding how search demand, website content and site architecture are changing in relation to each other.