Publishers Block Just 22% of AI Bots. Here’s What They’re Missing
In terms of AI bot traffic, have you met someone more knowledgeable about it than you?
Gavin King:I'm sure they're out there, but I've gotta say I'm probably in the top, like, five. Like, I I like stare at this every day. My background is in, you know, network performance at big tech companies. I like... I spend, like, more than a normal workday every day on it.
Gavin King:So, you know, this is kind of what I live and breathe right now.
Rob Kelly:Welcome to Media and the Machine. I'm Rob Kelly. This show is about AI and how to profit from it. I believe this is the biggest technology shift of our lifetime. I'm a three time founder and CEO of content driven businesses, and each week, I talk with the people on the inside of AI and content, the ones building this new world and learning what works firsthand.
Rob Kelly:I'm an optimist about AI, but I'm also a realist, so every show ends with
Rob Kelly:me asking our guests what AI means for our jobs, our families, and our future.
Rob Kelly:My guest today is Gavin King, founder and CEO of Known Agents. He was a software engineering manager for Meta's Instagram for seven years before that. And as you just heard, he lives and breathes AI bot traffic. Gavin and I dig into a remarkable shift happening on the web.
Rob Kelly:One third of website traffic is now coming from bots. We break down what those bots are actually doing, and bots often get a bad rap. And according to Gavin's data, 96% of bots actually follow the rules when publishers tell them to stay away. Meanwhile, publishers block only 22% of bots. We go over a live data about what percentage of bots are blocked by such companies as Gannett, CNN, and Adweek.
Rob Kelly:We also get into the bad news. Just point 1% of human traffic is currently coming from AI chat referrals, and what publishers can do about it, from paywalls and subscriptions to MCPs, and building businesses that AI agents can transact with directly. And at the end, Gavin shares his advice for young people entering a job market being reshaped by AI, and whether his own future kids will even go to college. Please enjoy my conversation with Gavin King. Can you just share kind of the history of bots and how important it is to both technology and how it touches pretty much any human using the web?
Gavin King:Yeah. So, I mean, bots really aren't a new thing. You know, they're as old as the Internet itself. Like, Google bot's been around for a while, but I think the big change that's happened recently is that there's just, like, this, like, Cambrian explosion of bots and, like, what they're actually doing.
Rob Kelly:So pre AI, Google would make, in effect, a photocopy of the web. The bot would go out, grab the content that it wanted to help serve to people doing Google searches. What's changed now with AI?
Gavin King:Yeah. So there's... So instead of just kind of one category of bots, these search engine crawlers, now there's, you know, dozens, probably, like, about 15 to 20, at least the way that we categorize different categories of bots.
Rob Kelly:Okay. So we're looking at your knownagents.com forward /insights page. One headline here is 33% of traffic now to websites is coming from bots. These are good bots, you know, average medium bots, bad bots, all the bots, about 33%. And I double checked because I have three sites up now that Node Agents is running on, and I ranged from 32 to 38%.
Rob Kelly:So I'm sort of in the normal Right. Kind of range there. But there's kind of a surprising stat here on the robots.txt effectiveness. Can you first off Yeah. Just define what that means and what's kind of the surprising part of the data here?
Gavin King:Yeah. Totally. So what a robots dot TXT file is is it's a file that you put on your website. Before a bot starts looking at any of your pages, it should read this file, see if it's allowed to visit particular pages. And if not, then it should back off.
Gavin King:It is sort of an honor system. And, yeah, there's sort of a current narrative right now that many bots, if not all bots, just ignore this completely. And that's just not true. Right? Like, this dataset, this is thousands of websites that we're looking at the traffic of.
Gavin King:Many of the big bots, especially of the bigger companies, they actually do follow the rules. So if you put something in for OpenAI or Anthropic, they actually will back off and not visit your pages. So, yeah, I think a lot of people, like I said, immediately jump to firewalls or bot blocking enterprise software when, you know, they can get 96% of the problem solved just by having a full coverage robots. Txt.
Rob Kelly:So let's talk about the four types of AI activity, and we're still on this known agents.com forward /insights. So you got AI training first. These are the the major bots like Clogbot and GPT bot and so forth. Right?
Gavin King:Yeah. Glanthropic or OpenAI or Google or Meta, they have their own bots. It's collecting, you know, website data, and then that data is basically used to train AI models, whether that's LLMs or something else, like the first party scrapers.
Rob Kelly:How can you tell? How can you tell that someone's training on someone else's data?
Gavin King:Yeah. So a lot of the times, which is good, like, they'll identify themselves. So they'll say, you know, I'm GPT bot, for example. Actually, there's literally sometimes like a URL in the request itself that says, hey, if you wanna learn more, go to openai.com slash, like, whatever bot. And then you can visit that website, and then it will describe, like, what this bot's for, and it will say, hey, this is used for training.
Gavin King:Like, you can block it by adding it to your robots. Txt. Here's maybe some IP addresses that it should be coming from.
Rob Kelly:So in this case, this is saying roughly 5% of traffic of the... How many sites you have in the insights page?
Gavin King:Yeah. This page, the Agenic Web Index here is built by about 5,000 websites.
Rob Kelly:Okay. 5,000 websites. Around 5% of those websites traffic is coming from what we can just call AI training related bots. Yep. You got Anthropic, number one.
Rob Kelly:Meta, number two. Amazon, number three. OpenAI, number four. ByteDance, owner of TikTok, ByteDance's
Gavin King:Correct.
Rob Kelly:Yep. As number five. Can you describe why the numbers are like they are in that order? Meta and Amazon kinda surprising given that they're not as big market shares
Gavin King:and traffic But they're they're trying to be. Right? So that could explain some of it.
Rob Kelly:Okay. So they might be playing catch up just to Exactly. Okay.
Gavin King:Right. That could be indicative of maybe they're more kind of in the training phase of their new models, like, you know, Meta's coming out with a bunch of new models. Amazon's starting to get in the mix with their own models. So, yeah, maybe they're just doing more training these days or they have less already crawled because they're, you know, kind of falling behind a little bit in terms of timeline of developing these big LLMs.
Rob Kelly:Yeah. So if you see OpenAI at number four, it's possible by far. They're 7% versus Anthropic up top at 26. It might just be OpenAI had a bigger head start on training, and now they need to train less. Their bots need to hit less.
Gavin King:Yeah. Maybe there was a big surge months ago or years ago, and now they're kind of just incrementally crawling web pages. Whereas that might not be the case for these other websites who are kind of crawling things for the first time.
Rob Kelly:And what's going on sort of the first unknown name or sort of non big companies, Parallel? What's going on with Parallel at number six?
Gavin King:Yeah. Yeah. So this has been sort of a recent surge in the past couple months. There's been a kind of like this proliferation of third party crawlers. But now you have a bunch of SaaS businesses spinning up who are kind of like doing that on their behalf.
Gavin King:So you might get a crawl from, yeah, Parallel or, you know, you.com or, like, Exa. So they're kind of like these multipurpose crawlers. And there's this whole other side thing that we can go off on about Cloudflare kind of like getting them to split these up into use case. But this is, another sort of cohort of bots that are visiting websites. These, you know, one size fits all bot.
Rob Kelly:So Cloudflare is asking them to do more than one at a time or saying Yeah. They'll be
Gavin King:Yeah. So they're basically saying break up your use cases or, you know, we'll start blocking you. Taking kind of the common denominator of what this bot is used for. They're basically gonna assume that it's for the worst unless you start breaking it up into multiple bots. So they're saying we'll block you unless you start splitting up your use cases.
Rob Kelly:You've also got the top visited website categories for these AI training bots, and it's surprising. So the top two are law and government and online communities. Can Can you just give me your take on why those are the top two categories?
Gavin King:I think online community is a good example. Like, you want them to talk like a human talks, and the best way to do that is to crawl user generated content. Right? So if you think about, you know, four
Rob Kelly:Reddit is Reddit is the number one source of training data for AI models. It's because they have this nicely formatted question and answer that real humans, last I checked, are providing there, and that's very easy and effective and valuable for an LLM to train on. Is it as simple as that?
Gavin King:Yeah. Like, if you want your LLM to talk like a human, you wanna train on humans talking to each other. Like, you don't wanna... I mean, you can, but you can train on, like, an article, but that's not gonna tell you how humans interact. It will just tell you, like, you know, what concepts go together.
Gavin King:So, yeah, online communities, it's top tier for a reason.
Rob Kelly:Wow. So it's not just on Reddit, it's not just the actual, like, content and getting the answers. It's also how humans are speaking to each other when they're getting to that question and answer.
Gavin King:Fascinating. Exactly. Yep. Yep. Yeah.
Gavin King:Because it wants to mimic human interactions.
Rob Kelly:And that really is the beauty and the the big breakthrough of LLMs. Right? Is it's looking at language and training for now. It's the main trainer of of AI. Right?
Rob Kelly:Of AGI? Yep. Getting to AGI. Obviously, physical world models and stuff. Cue the dog.
Rob Kelly:I love the dog barks. Keep them coming. So then that's Let me let
Gavin King:just quiet her real quick before we
Rob Kelly:Yeah. What's her name?
Gavin King:Jill. Zoe. She's like this little Yorkie.
Rob Kelly:Alright. That's Zoe the Yorkie barking if anyone cares. Okay. So that online communities was number two Yeah. Trained on type of category.
Rob Kelly:Number one is law and government. Why is that number one?
Gavin King:If you think about common use cases of why people use LLMs and especially, like, why enterprises would pay for LLMs, you know, you want them to be good at things like legal or making decisions. The best way to do that is to actually just read legal documents or read government websites. This has more to do with what people use LLMs for, especially like enterprise customers.
Rob Kelly:And it's authoritative, kind of the primary source
Gavin King:Yep.
Rob Kelly:In case of government. Okay. So that's the AI training tab there. Should we click on what next? AI search indexing?
Rob Kelly:Kinda going in order of traffic roughly?
Gavin King:Let's do fetching first.
Rob Kelly:Fetching next. Okay. What are the insights to read into here?
Gavin King:This category kind of encompasses two types of bots. By far, the most important one for most websites is AI assistants. So this category is when there's literally like a human in the loop on the other end. They're working with an AI. They're asking questions.
Gavin King:They're doing some research, And the AI is going off and pinging websites based on what the user is asking it. RAG is an acronym that people are probably familiar with, but that's exactly what this is. So, yeah, for example, if I'm looking up sports scores or I'm writing a research paper, let's say I'm I'm typing something, I say, what's the current score of the game? It might go off and, like, ping ESPN and say, hey. What is the actual score right now?
Gavin King:I'm gonna respond with that to the user. You know, maybe I'll include a citation as well that the user can explore more and click into the page. But, yeah, that's basically what this is, is that kind of human in the loop going off and pinging websites if you're ChatHPT.
Rob Kelly:And by far, the number one operator is OpenAI with 88.4% of the fetching activity, and then Anthropic is next at 4.7%. Is that just really market share of people using chatbots?
Gavin King:Yeah. I I... So it is mostly just market share. Like, you know, more people use across the world OpenAI products than Anthropic products.
Rob Kelly:Gotcha. What should we do next? Search indexing? Yeah. So in this case, give an example.
Rob Kelly:What's an AI search indexing type of tool? This would be the Apple bot perplexities in here too? Or
Gavin King:Yeah. So AI search indexing, these are how the chatbots know what's even available. Right? Like visiting like a science website, it has to know that that science website exists in the first place. So it might have already kind of like done a rough crawl of that website, says, okay, it probably has answers for these things.
Gavin King:It's kind of saving what could be a good reference. That's what this bot does. It builds up that index, which, you know, later an AI system might ping that website, the same website to get fresh information. But this is just kind of like similar to Google bot. Like, it's how it knows the lay of the land, like, what's out there, what's available.
Rob Kelly:Alright. So in this case, some of the top agents are, like, Perplexity and then the search part of Claude and the search part of ChatGPT. But one surprise here is there's no Google, and you explained it to me in a past conversation. The reason there's no Google is because
Gavin King:I mean, Google, because they already have such a big index. Like, they might be using some crawlers for multiple use cases, which is something that people are pushing back against. So that's why it's not here is because they already kind of have a big index, which they are allegedly pulling from instead of kind of crawling this as a special case.
Rob Kelly:Okay. So next is AI browsing.
Gavin King:Yeah. So this is really interesting. These are the the actual, like, AI agents. People use that word a lot. They throw that around for, like, bots in general.
Gavin King:You, as a human, are still on the other side. But instead of it doing research and it's kind of like pinging random things, you actually give it a task. So if you say, you know, I wanna book this dinner reservation or I wanna book this flight or, you know, eventually, like, I wanna buy this thing. It's the thing that actually uses a browser. Like, it literally has, like, a a window of a browser.
Gavin King:It has a cursor. It's going off and, like, clicking on things. So it's getting, like, multiple steps on a website with some goal in mind.
Rob Kelly:So if I went to sleep and said, wanna build a new business, go ahead and build it for me by the time I wake up at 05:30AM. All that activity would be the type activity you're describing here in AI browsing.
Gavin King:Yeah. It could be. So, like, if it's building a website, it might be... You know, some of that might be done by kind of like those AI coding agents, but some of it might be just testing what it built. So it could be spinning up a browser.
Gavin King:It's like clicking through the website, making sure it coded the thing correctly. So, yeah, that would be like an AI agent.
Rob Kelly:Okay. Taking over the browsing of pages and the web.
Gavin King:Exactly. Yep. Yep. Exactly.
Rob Kelly:And this is interesting in this case. I mean, I've looked at, like, you had a lot of Google Analytics in my day from different blogs and websites created. So it spends on average one minute, but looking at 13 and a half pages Mhmm. So that would be, you know, roughly a little more than four seconds per page.
Gavin King:Mhmm.
Rob Kelly:So that would be way more productive than if you or I were going to look at pages and actually use them ourselves as a human. Right?
Gavin King:Yeah. Yes and no. No? I actually think generally, like, the models are pretty slow at this still. They literally, most of the time, take a screenshot of the page, and then they look at the image of the screenshot and then, like, take an x and y coordinate from that of, like, where you wanna move the cursor.
Gavin King:So it's a very slow process. If you look at the actual session replays of what they're doing, a lot of the time, it's just, like, randomly reloading the page or like clicking on something twice or like going back because it messed something up. So there's a lot of stumbling along the way. But, yeah, I I expect over time as the models get better especially, and there's there's actually been a lot of progress on this in the past like month or so, they'll get more efficient at completing their task faster with maybe fewer page views. But, yeah, this is...
Gavin King:I think this is very up and coming. Know, Sam Altman tweeted about this the other day or I guess retweeted something from Chattypati. Like, they're doing this more and more, you know, at SpaceX AI. They also have GrockBot, which was just released. Like, that heavily relies on browser use.
Gavin King:They're partnering with Stripe and Link to do shopping. So there's a lot going on here, and I do expect this is gonna probably ramp up pretty quickly over time, you know, and these products are gonna be doing more and more interesting things.
Rob Kelly:Okay. So now we're looking at dot t x t, and sort of the big headline here is 95.7% of bots are following the rules, basically. If you put in rules in your robots dot t x t, they are following them. And what about this list of top blocked agents here? Can you read through Yeah.
Rob Kelly:Who each one is and what that means? Just maybe the top five.
Gavin King:Yeah. So as part of this dataset, we look at the top thousand websites according to SimilarWeb, and we look at their robots. T x c's every day. We take a snapshot of all of them, and then we derive these insights from them. So, yeah, the top block bots by far are AI data scrapers.
Gavin King:So these are the first party scrapers that are collecting data. CC bot is also sort of a multipurpose crawler. Like, it is used to train many of these AI models.
Rob Kelly:The CC bot stand for common crawl?
Gavin King:Exactly. Yep. Yeah. So that's the top one. ByteSpider, that one is just kind of particularly, like, aggressive.
Gavin King:Like, it might be hitting many pages in a very short amount of time. So, you know, that kind of makes the people running the websites angry, so they might block that one over the other ones. So that one's kind of at the top.
Rob Kelly:That's ByteDance's spider. Yep. Exactly. Yeah.
Gavin King:Then you have, like, GPT bot, Claude bot, you know, Metas bot. So kind of like the big AI companies.
Rob Kelly:What's a worst case scenario you've heard about an AI bot hitting a site? Yeah.
Gavin King:I mean, like, you will 100% see some of these bots. A lot of times, it's like the the scraping ones kind of hit your website hundreds of thousands of times, you know, depending on the size of your website, but, you know, within a couple of minutes. That happens a lot. That's actually a pretty common pattern where you'll see nothing for, you know, a week, and then it will just hit everything at once. A lot of CDNs are are pretty good at handling that kind of traffic, but some aren't, one thing.
Gavin King:And then two, that's just costing you money. Right? Like, especially these bot visits that aren't giving anything back. You know, you pay per request a lot of the time because these are usage based billing systems with the CDNs. So, yeah, it could just cost you a lot of money and potentially take your server down.
Gavin King:Although, I will say that's not super common.
Rob Kelly:Yeah. I remember when I first started using your known agents tool, it was amazing. Like, you know, sixty seven milliseconds, sixty milliseconds. I'm looking right now in real time. Four hundred and eighty two milliseconds.
Rob Kelly:Bots just hitting like crazy.
Gavin King:Right. Yeah. It's like this fire hose of visits.
Rob Kelly:Alright. So now we're on AI chat referrals on your insights page. First off, just define this. What does it mean? It's looks like it's around point 1%.
Rob Kelly:So what does that mean?
Gavin King:Yeah. So this is the fraction of your human traffic that comes from people clicking in from AI chat responses.
Rob Kelly:Okay. So this is the big bad news, only point 1%. Yeah. Obviously, tiny compared to the days of Google. That's a bad news.
Rob Kelly:What can publishers do about this?
Gavin King:One, I think this is just bad. You can spin it, but I do think this is just bad sometimes, especially if your business relies on people coming to your page specifically and, like, seeing ads. That's gonna be Yeah.
Rob Kelly:It's just bad. Right. Right. We gotta say that.
Gavin King:You know, I see a lot of people converting, for example, more of their content to paid or, like, behind a paywall or they're offering subscriptions. So these are things that, you know, an agent can't get past a paywall. Right? So they can say that, hey, if you want to sign up for this thing, you have to click through. You have to subscribe to this publication.
Gavin King:And that might be a better model in like the AI world if you want kind of that premium curated content. But something needs to change. There's not a real future where we're gonna get as much traffic from AI chat referrals as we got from Google. Like, that's just not gonna happen. So, like, something needs to change and for better or for worse.
Gavin King:So I mean, there is sort of a silver lining, not enough to kind of capture the whole thing, but I've heard anecdotally that these humans are more valuable. Like they're more likely to subscribe or to buy something or to sign up or to convert in some way. But, you know, even if they were 10 times more valuable, that would still be 1%. Right? Which is nowhere near the value you're getting from Google.
Rob Kelly:Okay. So now we're looking at this publisherrobots.txt leaderboard on your site and back to the AI bots are kinda doing their jobs very well. 95.7% of them are honoring robots.txt rules, but publishers are only blocking 22%. And I just remember that most people are just gonna be listening. So I'm just gonna read off some of these so people can understand.
Rob Kelly:So
Gavin King:Sure.
Rob Kelly:The ones with a 100% coverage are Adweek, Arkansas online, golf.com, San Diego, uniontribune.com, slate the star, and then it starts falling off to way less than a 100%. The list goes down to some folks who have tiny coverage. I don't wanna pick on anyone here, but if you look at fox5atlanta.com, you know, 2% coverage or fool.com. Is that Motley's fool, I think? Yep.
Rob Kelly:Yep. And Country Living, only 2%. A lot of big companies. American Banker, only 2%. Four zero four Media, they love to report about things like bad bots.
Rob Kelly:They have 2% coverage. Tvguide.com, only 2% coverage. Meaning, they're only blocking 2% of the AI bots. So a huge, huge delta here and bad coverage. But as you were saying earlier
Gavin King:So, basically, like, across these 1,000 top publishers, yeah, only 22% of the bots are actually blocked. So if you were to click into the robots. T x t's of some of the websites that you just mentioned, maybe they block two or three of them, but, you know, there's over a 100 AI bots that we know about that are actively scraping things to train LLMs. So, yeah, the coverage is surprisingly low here for, like, how big of a topic it is and for, I guess, this new insight of how well bots actually do follow it. There's a huge discrepancy here that I think is pretty surprising to people.
Gavin King:So to summarize, bots follow the rules, like, 96% of the time. So the bigger issue or another issue is that coverage is just really low. So I'm kind of on this crusade right now to get all the publishers to actually make sure that they have a full coverage robots. Txt because that's the first line of defense, and that will solve 97% of the problem of AI scraping at least.
Rob Kelly:But say in Adweek or golf.com, one of the few, and Slate who are at the 100% blocking, is that a good strategy to block a 100% AI bots?
Gavin King:So these AI bots in this list are exclusively the ones that do training without giving anything back. So this isn't including, like, those AI search crawlers or those AI
Rob Kelly:fetchers. Okay. These are only the AI training bots.
Gavin King:Exactly. Yep. Yep. And also, these are, like, running up your traffic bills. Right?
Gavin King:If, like, 30% of your traffic is this type of bot, then your CDN bill is 30% more expensive. So even if you don't believe that... Against the data, if you don't believe that many of these bots are following the rules, the ones that do will still save you money on your server bill.
Rob Kelly:And why do you think people are missing the 100% coverage? You know, I'm looking here at some of these others. CNN, 39%. It's like, what's
Gavin King:Yeah.
Rob Kelly:You know, they got a smart team over there. Why are they
Gavin King:Yeah.
Rob Kelly:Yeah. Missing this?
Gavin King:Yeah. I think it's just hard. Like, it's a hard problem. There's just so many bots. There's new AI companies popping up all the time.
Gavin King:Within the companies, they're popping out, like, new bots all the time. So it's just hard to keep track of.
Rob Kelly:So I'm looking here at all your individual web properties, and I see a bunch of Gannett ones, and I know Gannett came up recently. This is Harold Tribune, IndyStar, jacksonville.com, a bunch of them. They're huge. Got a lot of local sites in addition to USA Today. But, anyway, in your numbers here, it looks like Gannett Properties very consistently block around 65% of bad AI bots.
Rob Kelly:I know I heard recently they said a 100%. What's the difference?
Gavin King:Yeah. So it could be a few different things. I mean, first of all, 65% is far above the average, which is, like, 20%.
Rob Kelly:Yeah. They're way high up. They're way, way above average, probably in the top five publishers or something doing it.
Gavin King:That's why there's like a leaderboard. Yeah.
Rob Kelly:But I was just curious what's missing. Are they missing some bots? Is that simple as that?
Gavin King:It could be a few things. One, it's like... I mean, they're... Like, licensing deals play into this. Like, sometimes you do wanna allow certain bots if they do have some deal.
Gavin King:Like, they might not block certain bots to let them in. They could mean that they're blocking them in some other way. So maybe they block them with their firewall, which is like hard blocking instead of kinda like this, like, soft blocking, like robots. Txt. Or, yeah, they might just not know about all of them.
Gavin King:This is our full time job. We have our own AI that scans the traffic of the thousands of websites that are connected to us. It finds new bots literally every single day. So we we also probably just know about more bots.
Rob Kelly:So there's this new New York law. It's called the Stealth Crawler Prohibition Act. It looks like there's fines of $15,000 per day if there are stealth crawlers that fail to comply with disclosure requirements. Is that gonna hit up these bad bots out there?
Gavin King:Yeah. So, I mean, what these laws are is basically just making it so they can't pretend to be human. There's a few versions of this. There's the New York one. I think there's, like, an analog in Europe.
Gavin King:Basically, that will do is make bots have to identify who they are, and also in some cases, like, comply with robots.txt. So if you already have a full coverage robots.txt setup, it will only become more and more effective as more bots start following the rules. K.
Rob Kelly:Just for some positive news on AI bots in general, like I saw one the other day, which was Yelp did a deal with OpenAI where OpenAI will now with ChatGPT use Yelp's localized info, and, of course, you know, they're probably getting paid some amount of money for that licensing deal, Yelp is. But also Yelp got... As part of it, they said, and soon, there'll be a link when folks are on ChatGPT and looking up Yelp stuff. There'll be links for requesting a quote from a plumber, know, as part of Yelp. You know, they provide local contractors and plumbers and other stuff.
Rob Kelly:True commerce coming from it.
Gavin King:I was
Rob Kelly:like, what a great use of AI for a content driven company with Yelp. Can you just give me a few other use cases like that that have got you excited to counterweight some of the bad bot news that comes out constantly?
Gavin King:No. I I mean, I think that's my my point. Like, I think some of these businesses will have to adjust their model for this new reality that's definitely not going away. So, yeah, whether that's selling to AI agents rather than selling to humans, making that easy, like working with some of these new paradigms to be able to, like, offer these things through AI platforms, that's a really good example. Yeah.
Gavin King:A lot of these, like, marketplaces or services, like, you really don't care if a human books your thing or, you know, an agent books your thing. So if you're booking a flight, you're making a dinner reservation, you wanna make sure that your website is easily accessible and optimal for this new type of user, which is like bots, because they... Their money is just as good. Their money is just as green as a human's. Right?
Gavin King:Like, they can do the same thing, and you really shouldn't care. So I think it's like important that you make sure you're not blocking them or your website's set up correctly. But, yeah, these are like these new kind of business use cases that especially for when your business is in page views, it's some kind of conversion whether it's buying something or booking something. You know, AI agents will actually probably be sort of a tailwind for
Rob Kelly:you. Great. Zoe? There's Zoe. Good.
Rob Kelly:If you were building a new site today and wanted a lot of AI traffic, what would you do?
Gavin King:I mean, luckily, don't have to do much. For example, like, you want to attract, like, this whole, like, geo thing, like, 5% of that is the same as SEO. If you have good SEO, that's 95% of the problem if you're trying to attract them. For the next kind of cohort of, like, AI agents, the ones who are clicking through your website, that's where some kind of like specific tactics come into play. So for example, you wanna make sure you have a good accessibility tree because that's something that Google has said that their agents kind of value and help them kind of navigate through your website more quickly.
Gavin King:If you can, many people have heard about like MCP, but it's basically an alternate way for agents to access your data or access your service. This is something that's basically through an API. It's not through a browser. And it's a lot more like token efficient. The agents usually can get something done much faster through this.
Gavin King:So if you were to set up an MCP kind of in parallel to building your website, that would a lot of the times be more optimal for AI agents to access the same data.
Rob Kelly:So last time we chatted, you talked about some good things to put behind an MCP for like a data owner or a content owner. You talked about, like, surveys, polls, research.
Gavin King:Yeah. Yeah. For sure. So, I mean, I think there's there's opportunity for, you know, publishers, like content creators to paywall more stuff on their website. Or another way to do it is if you want this successful for AI agents specifically, like you could put that behind an MCP, charge some kind of access for AI agents to be able to retrieve that data.
Gavin King:So, yeah, great stuff. Like, you know, a lot of these publications do a ton of good research. They do like polling. You know, they have these proprietary data sets that would be valuable for AIs or, like, the businesses that use AIs. That could be another revenue stream.
Gavin King:It's like putting that behind an MCP, of like locking it off and then paywalling that and then getting revenue from unique data.
Rob Kelly:What's something that leading publishers are doing in this space that has your respect?
Gavin King:I think a lot of kind of adjustments to the business model to depend less on kind of inbound traffic are always good. You know, you could have your kind of like free articles, but you might wanna decide to put more behind a paywall. Or, you know, maybe before you didn't have a subscription offering, but now you implement a subscription offering. But also there's more creative things. Right?
Gavin King:Like offering things like games or doing live events, hosting conferences. These things can also generate revenue without being disruptable by AI.
Rob Kelly:Going offline. Yeah. I was talking to Jimmy Hutchison, the CEO of Spin, who's got Spin Magazine, but they also do have events and other things. And Yep. Basically, AI is really not hurting his business much at all because he's got so much offline business and just a strong brand.
Gavin King:Totally. Yeah. And and I think the AI companies see this too. Like, you see these, like, mega salaries for event producers within Thropic and OpenAI and things like this. So I think just live events in general are gonna become more and more valuable.
Rob Kelly:What's your take on... This just happened recently. Reddit, they went from receiving 4% of all citations on AI searches to less than 1%. What's going on there? Any idea?
Gavin King:Yeah. That's one of those, like, black box decisions. I don't know if there's some behind the door deal going on or... Yeah. I don't know.
Gavin King:I I haven't actually, like, kept up with that.
Rob Kelly:It can be a function simply of an OpenAI or Google just changing their algorithm in terms of
Gavin King:Yeah. 100%.
Rob Kelly:What their bots are hitting. So it doesn't surprise you that that could happen to a publisher.
Gavin King:Like, you can just denialist someone from being surfaced as an AI assistant, or you can change the algorithm so it kind of, like, subtly edges someone out. Like, there's many ways that you could start blocking someone.
Rob Kelly:You came from Meta. What's your take on their AI strategy overall? Just how they're doing AI wise kinda compared to the OpenAI's and the Anthropics of the world.
Gavin King:I mean, obviously, I can't disclose anything confidential from when I was there, but I will say in general, I think that these incumbent tech companies that have kind of a portfolio of other products that they can kind of just plug AI into to make them that much better are in a better position than I think people think. Kind of the front runners right now are the OpenAI, the Anthropix. They have the best models right now, but they have to invent ways to use them from scratch. But, like, these are use cases that they're having to develop zero to one, whereas a Meta or a Google, you know, it's really easy to plug in AI into Gmail and make it like super super good. Or, you know, to use AI to do like creation tools on Meta or Instagram that, you know, make people use the product that much more.
Gavin King:So I don't think they fully realize that future state yet, but I do think they're in a much better place to capitalize on AI than people give them credit for.
Rob Kelly:So cool. Now we're just on to a couple of humanitarian questions and we wrap up. What are you telling the younger kids in your life? I know you don't have kids yet, but just the younger kids around you, what are you telling them about how AI is gonna impact their life or prepping them for this new world of AI?
Gavin King:I think the biggest change, you know, I'm not super young kids, but one thing I see is, you know, my my old university, they are seeing a lot of people, like, entry level engineers unable to get jobs or they're being, like, fewer jobs for computer scientists. We're kinda get... Probably gonna get into a trap where, you know, we're not up leveling these people who are just kinda coming into the market. And then, therefore, as the market matures, there's not gonna be people to, like, develop into those more senior roles, which is bad because I think you are always gonna need a human controlling a lot of these AI, especially with coding and with engineering. You need someone who's orchestrating it all.
Gavin King:So, I mean, the best one is just to, like, leverage AI as much as possible, especially if, like, you're in that situation to kind of, like, hate on AI and that it's ruining everything. But, like, at the end of the day, like, you... This is the reality, and, like, you need to learn how to leverage these tools because the genie's out of the bottle, and you need to learn how to use the genie to do what you want.
Rob Kelly:If you have kids, do you think they'll even go to college?
Gavin King:They'll probably have, like, brain implants that just, like, puts college into their neurons Yeah. Directly. So who knows?
Rob Kelly:Well, this is Media and the Machine. A few things about you and me. If wanna hear about the next new episode, make sure you hit follow on the show on your podcast app. And if you like something about this episode, the guest, the topic, even one part of the conversation, leave me a rating or review and tell me. That helps me know what you want more of, so I can make the show better for you.
Rob Kelly:And if you wanna go a little deeper, head to mediaandthemachine.com and subscribe to my newsletter. You'll get my favorite highlights from podcast episodes, as well as essays I write about AI. You can also email me directly from there. Recommend a guest. If you make it on the show, I'll give you a shout out.
Rob Kelly:And if this show made you think of someone, please pass it along. That's how people find us, and I'm genuinely grateful. Thanks again, and see you next time.
