What's the best method / tool to discover all entities for a given topic?
Keep in mind that all the tools – even ones that tie into Google's own Application Programming Interface (API) – are incomplete. I've seen some estimates that less than 20% of Google's knowledge graph is represented in the API.
I'm not sure if this is because it's only showing "well established" things – or if it's "popular" things or what… but it's not complete.
Also… before things get put into the knowledge graph, Google puts potential entities (including your brand, services you offer, and so on) into a state that I call "being In Play" prior to actually updating the knowledge graph. It can also put search terms that represent entities "in play" for a bit to see if the new understanding produces better or worse satisfaction scores for searchers. Some thing go into play and Google, after a few weeks of trying things, reverts back. Sometimes it tries to then go further and incorporate more into it. Sometimes it can try other variations of what it thinks it all might mean. And sometimes it might just say, "Hey, I think we've got this right – let's update the KG with these new facts and entities
I've seen things go "in play" for periods as short as a couple weeks. (On client sites where we have a brand and niche well designed and represented – I can expand or grow parts of it pretty much at will – and quickly). For other things – especially when Google is trying to understand large things like an entire niche – things can be in play for six months or even a year. You know they are in play because the Search Engine Result Pages (SERPs) change – day to day, hour to hour, week to week or other things.
As such – the MOST IMPORTANT TOOL is to actually run your searches on Google itself. Understand the landscape. See what types of rich results appear. See what type of pages are showing – informational, purchase, disambiguation leaders, and so on. When things start dancing around – if you don't know what it looked like a few days prior, then you can't really tell what changed or what Google is trying to figure out. And watching as the "in play" action evolves can give you clues as to the direction it's going as it learns so if something isn't going the way you want, you can jump in and do something to hopefully influence it to rethink and try again.
Your brand (web site/company/whatever) especially will start to take shape LONG before any tool picks up on it. And if you aren't liking how it takes shape, getting in control earlier is much easier than later. It's (relatively) easy to teach Google something new or to make a new connection to something it already knows. It's MUCH MUCH more difficult to get Google to unlearn something it has learned and locked in or to change an established knowledge graph connection that may not be exactly right.
Truslow so basically, immersion into the keywords and their related terms on a topic is what you suggest? Are there any tactics or methods you use to help track these SERP "in-play" evolutions? This kind of research sounds like a handful.
At this point – it's as much art as it is science. So my process evolves as I get better at it.
For tracking, I originally started out just doing most everything in my head. I'd take the key clients and watch search terms related to their brand especially and to their industry niche, products or services, and anything key. Our tracking and reporting team would also send me regular lists of terms we were targeting that were once stable but that now seemed to be moving.
As I got more and more things that I was looking at, my own memory wasn't capable of doing it well – and it also needed information from further back (which was starting to fade) so I started taking notes in a spreadsheet for various entities and terms. Basically I put a date down and count "features" – meaning enhanced listings and things besides ads that appear above and sometimes intermixed with the normal organic search results. Over time I'd start to spot things like – okay, when Google adds a product carousel, that often means that it will feature several fewer "buy intent" type product pages in the organic search results in favor of showing more informational ones.
Really – it evolves regularly for me right now. I'll add new things that I keep track of in my sheet (like recently I noticed some terms that have multiple site link representations in the organic searches – and they are less likely to require a "brand" name to trigger them. That's a sign that the knowledge graph is growing in that industry so that entities within that niche have enough features and details understood that Google can show more options in a smaller place).
So yeah – it's definitely a handful, but it's the future. I imagine that in 10 years time (or thereabouts) things will settle down. Right now, the knowledge graph is growing fast. In recent months, it's growing REALLY fast. Certain new things are coming into play which help Google learn faster and with more accuracy and certainty. (Among them, in my opinion, is the common complaint of "crawled not indexed" problems – where Google crawls it, but it's not ready to it into the index and start trying to rank or learn from it until it knows WHAT the page is actually about and if it's saying anything useful that it might learn from. Rather than grabbing every page as fast as it can, Google has started to try to grab the "informative" pages that are structured well and that are easy to extract information from so it has a much stronger signal to noise ratio).
Once the KG gets big enough to cover most of the important areas of knowledge to a certain level – this will slow and it will be easier to connect things where they belong because that "where" they need to go is already established. Right now, when we take on a new client I need to try to determine whether I'm going to need to connect to what is there, or if I have to create the things my things need to connect to because they don't exist – and then go the next step and connect those new things to the things that Google already understands. And at some level – it's almost always a mix of both.
And again.. it's evolving and changing so quickly, the things I think are best this week will likely be replaced by a new tactic or way of looking at it next week. Search Engine Optimization (SEO) right now is about being nimble and seeing where it's trying to go, not chasing what's there to try to outrank it. Optimize to be ready to land at the top as the dust settles, if you will. How? Lots of experimenting, learning from what I tried last time, and certainly a fair amount of luck. lol
Fitzjohn » Truslow
I love your answer on this. Maybe that's why we constantly try to understand Google and our visitors to our site regarding search intent
Honestly, SEO gets more fun and interesting every day. When this whole thing first started we had to build sites that fit into Google's idea of what a Good Site was. To rank for this, you'd do this, this, and this. And because it was "easy" it was difficult because it was just as easy for your competition to do it as you… so it evolved into a game of links. Everything else is done, now it's just staying ahead of the links game.
With the Knowledge Graph – all that is changing drastically – and in recent months, quickly, too. I no longer have to build a site that fits into Google's idea of what my site should look like, what it should say, and how it should work. I now have the ability to TELL Google what to think about my brand, my niche, and how it should connect me to all that and rank me accordingly.
I really adore working in a niche where the knowledge graph is week – because I not only get to say where my site fits into it all – but I get to define what it all is – with my site at the center of it. I've got one client now where there are about a dozen players in the niche but really only two of us going at the KG in any hardcore way. I don't know the SEO manager for this other company and have never spoken to them, but I have huge respect. We've both worked "together" in defining the industry itself – which has helped the knowledge graph grow faster in the niche because two competitors are agreeing on terminology, important features/aspects, and generally how it lays out. But at the same time, we're having a really fun war of getting our brands at the center.
So now… remember, there are 12 other major players in this construction equipment parts niche and, for the sake of this comment, I'll call the main product "torque widgets". And, though there isn't really a "low torque" version – we've been battling over "high torque widgets."
Now, when you search Google for "Who Makes Torque Widgets" – my client's site comes up – often with their brand in big giant letters above the listing. If you search for "Who Makes High Torque Widgets" – my competitor is wins and gets the same thing. And oddly enough, in February, the results were switched. And BOTH answers, while correct in that we do make them – they are incorrect because there are over a dozen companies that make them. BUT… because we created the knowledge graph for our industry, we control it with relative ease. Now THAT is fun.
And, of course, a funny side effect of this is that we're the only manufacturers that appear on the first page for that term and the rest of the stuff is the distributors we have who actually sell them… and the top ranking distributors are the ones that sell both of our lines of products. My client (and I assume the other company) is fine with this result too – since they both hit differently. We hit on quality and reliability to market and they hit on price and adaptability. So, while we compete – we're not really going after the same customer groups.
Fun fun fun. I love this job. lol
Truslow great info here. I appreciate your insights! Just so I understand some of the finer points and actionable intel here. Are you expanding the Knowledge Graph on said topics by:
A. Adding to Google's Knowledge gain by publishing new or lesser know information about the topic(s)
B. Claiming Knowledge Panels for certain search terms
C. Showing topical authority through highly relevant topic clusters
D. Semantic schema
E. A combination of these.
All of the above… but A isn't quite right. Publishing new information is a part of it, but making the connections to things Google understands already is key to getting Google to start picking it up.
Dixon » Truslow
OK. I partly dsagree. Yes, Google's knowledge Graph is of course massive, but Google's Natural Language Processing (NLP) API does NOT show as many entities as other tools (Inlinks/Textrazor and IBM Watson). We know this because every week for two years we compared the number of entities returned by Google's API with the number of entities returned by InLinks in any given verticle. Currently Google's API is reporting just 15.1% of the entities that Inlinks finds, on average, in a given piece of content.
Now – that deserves some clarification, as the Inlinks Named Entity Extraction algorithm is DESIGNED to be aggressive in extracting topics, whilst the Google Artificial Intelligence (AI)/NLP tends to report "proper nouns" much more than "generic topics", but still, just because Google HAS a big knowledge graph, does not mean their API finds topics particular;y efficiently.
Right – I agree – the other ones have "more" – but you can't necessarily rely on them to have understood them in the same way that Google does. And if they do understand them the way Google understands them today – once something goes in-play, then that all changes and the tools need to re-assess.
If the knowledge graph your tool creates doesn't line up with Google's actual knowledge graph (that we can't really ever see – so we basically need to run searches and do some deductive reasoning by hand – and even that is imperfect) then that "different" KG you're working from won't really help you rank in Google.
It's a tough one – InLinks definitely gives me more (and it's one of my favorite tools lately) BUT I also end up tossing out or adjusting 60-70% of the terms it presents because they don't "really" fit. Another benefit of Google's own KG is that if that Application Programming Interface (API) is confused, it's not because the KG might be flawed, it's a sure sign that I need to disambiguate better rather than deciding it's not really related the right way.
So… while it sounds like we disagree – ultimately we agree. lol I don't think there is a tool that can "discover all the entities" and give us a list but there are starting to be quite a few tools that can give us some good starting points. You are focusing more on the "start with volume and whittle down" and I'm looking at starting with "less, get good focus, and then grow". And I think both those approaches can work and have their strengths and weaknesses.
All the entities for a given topic is indeed a current holy grail. we are working on it… but context becomes important. For example – take "Bridges" as an example. If you are writing about travel, then the "Golden Gate Bridge" is indeed a relevent entity. But Concrete isn't. If, however, you are an engineering site, then concrete (and steel) become much more important entities than actual bridges.
So – a complete knowledge graph of entities is relative. This is whty we do HAVE a knowledge graph of everything (well… everything in our world view, not Google's) but we currently create knowledge graphs to view based around your site or your competitor pages, because the relationships change in different situations.
I cannot say (yet) that we can show you more entities PER TOPIC than anyone /any thing else, although I expect we can, but we CAN extract more entities in a page of text than all the other tools we have tested against. (Google's Natural Language Processing (NLP), IBM Watson and Text Razor). I cannot test against Wordlift, but I expect they are also pretty impressive.
(Disclosure – I am Chief Executive Officer (CEO) of InLinks, so biased).
Keyword Research, Keyword Cluster, and The Group of Topics for Content Niche
Expanding Topics in Search Engine Optimization | SEO Niche
Definitions of Search Intent and Definitions of Entities and Utilisation of Both
Are Keywords You Mean Valuable in Google First Search Result Page?
Can Content that gets Written by Artificial Intelligence (AI) Win In Google SERPs?
Does Google Crawl and Index Users Generate Content Sites like Quora and Medium Easier than Yours?
Tips to Hire SEO writers, SEO and Backlink Assistants
8 SEO Tips: Voice Search Optimization, EAT, Featured Snippet, NLP instead of LSI, Quality Backlinks, Local Search Listings, UX, Schema Markup
If You Acquire a Competitor’s Website, Options: Redirects, Demolition, or Leave It Ranking on the SERPs
Backlink Effect for Local SEO