The Artificial Investor #56: The publisher's dilemma
✍️ The prologue
My name is Aris Xenofontos and I am an investor at Seaya Ventures. This is the deepdive version of the Artificial Investor that comments on one of the top AI developments of the last few days.
This Week’s Story: Websites start charging AI bots for their content
Cloudflare, a website security solution used by 20% of the world’s websites, just launched a new feature called “Pay Per Crawl,” which allows publishers and ecommerce sites to charge AI bots for accessing their content.. In the meantime, Perplexity launched its AI browser, Comet, available for Max subscribers (200 dollars per month) with focus on AI-based search and email/calendar automation, and OpenAI is reportedly set to launch a browser by the end of the month.
What is the current status of the online search and content value chain? How is the value chain being disrupted by AI and who is winning and who is losing from it? What is the publisher’s great dilemma and what are the emerging content monetisation models? What are the opportunities and risks for Tech investors and founders?
Let’s dive in.
📃 Introduction
⛓️ The current search value chain
The Web search and content value chain has been relatively stable over the last two decades and has reached a healthy equilibrium among the different actors - content creators (publishers, software vendors, ecommerce sites, etc), search engines (Google, Bing, etc) and users.
The different roles and value flows are:
Content creators: They invest money in producing content that they make available for free to be indexed by search engines and discovered by users. They effectively get traffic in return, which in turn can monetise through ads (publishers) or lead conversion to clients (software vendors, ecommerce sites, etc).
Search engines: They index pages and aggregate content to present to their users and direct them to the underlying sites. They monetise through search ads.
Users: They use search engines to get relevant results. These relevant results, when clicked on, lead users to the content creators' or publishers' websites, helping them access further content or buy the product they are looking for.
This win-win equilibrium, based on the flow of traffic, has led to the emergence of the search engine optimisation (SEO) industry. The idea behind SEO is to help content creators attract traffic and rank at the top of relevant search results by optimising the text on their websites. Many companies, and a lot of VC investors, have invested billions of dollars in SEO tools and content marketing tools. Specialist jobs related to SEO and content marketing have also arisen as a result.
💥 The disruptions in the value chain
There have been two main disruptions in the search value chain: 1) The zero-click behaviour, and 2) the data gap on the analytics side.
Zero-click behaviour is actually a trend that has been growing since before GenAI. Google has been helping people answer their queries directly within search results, like feature snippets or knowledge panels, one-click purchases and reservations, which have resulted in fewer clicks. Recently (pre-GenAI), nearly 50% of the Google searches were not resulting in a click to an external site. With GenAI, this trend has accelerated, as AI-powered summaries of search results give users a rich answer, where they actually don't need to do any clicks through to find additional information.
Gartner forecasts a 25% decline in traffic from search engines in 2026. According to Seer Interactive, there's been a 70% drop in click-through rates (CTR) for organic results and a 12-percentage-point drop in CTR for paid search ads.
Now, who's losing and who's winning on this?
User experience is certainly winning: I get faster and often better answers with AI search than with the old search. On the other hand, traffic is suffering, which means content creators are suffering. We are decoupling information delivery from website traffic. Content creators are spending a lot of money and time to create content and then not get any reward for it because there's no traffic in exchange, thus no monetisation. Also, the entire SEO and content marketing industry is suffering. What's the point of coming out in the top of the rankings of a search if people are not going to look at the search results and just look at the AI summary?
On the other hand, in regards to the data gap in analytics, AI search is really a black box. When an AI assistant answers a question, a content creator doesn't have any data as to 1) whether their data is used for an answer or not, and 2) how the content was used. This is obviously problematic.
The publisher’s dilemma and market reaction
Publishers now face the “publisher’s dilemma”: Do you allow AI to use and summarise your content without permission, without paying you anything in exchange for broader reach (by being shown to hundreds of millions of AI search users)? Or do you restrict access against AI bots and demand direct monetisation? There's clearly a trade-off between reach and monetisation.
The second question for publishers is “how do you do this?” Some major publishers and news organisations are blocking AI bots used by the frontier labs. By the end of 2023, nearly half of all top news websites across 10 countries were blocking OpenAI’s bots. Examples include the New York Times, Guardian and CNN. Obviously, the downside here is that the cost of blocking AI sites will be that your publisher may not be so present in the AI results, in the summaries of the web search, they won't be cited or recommended in the AI answers, which could mean less monetisation for them as well. So it feels like a bit of a dead end at the moment.
Another type of reaction has been taking legal action. For instance, the New York Times filed a lawsuit against OpenAI and Microsoft for copyright infringement. It started with AI bots crawling publisher websites to gather content to train the AI models, but has expanded to using this knowledge to answer user questions on the AI Search user interface.
Some other publishers are focusing on a middle ground, not blocking the AI sites but trying to sign deals with them to monetise their content. However, this is clearly not favourable for smaller publishers that don’t have a strong negotiation power.
💵 The opportunity
🏪 The emerging monetisation models
Four content monetisation models that are emerging: 1) licensing deals, 2) micropayments, 3) revenue sharing and 4) subscription-based search.
Licensing deals is the first monetisation model we say with large publishers that make deals, license agreements with AI firms. OpenAI reportedly agreed to pay about 16 million dollars per year to license content from one publisher (Dotdash Meredith), while Thomson Reuters allegedly made about 30 million in 2023-24 from AI licensing deals. This seems similar to what we saw with cable TV bundles, where distributors of content guaranteed revenues to the content creators.
Another monetisation model, which we are just seeing developing, is micropayments or a pay-per-use model. This is how it works: a site blocks access to all AI bots sent by the AI model developers. Then, they activate a micropayment service, such as the one that was recently launched by Cloudflare, which effectively puts a toll on anyone who takes information from the website. This can be a fee per page or per article or even per day (for all content); the precise pricing models are still being defined.
A third monetisation model is revenue sharing. This is about the AI search provider sharing the revenues they make on a specific AI search with the providers of the content that powered it. Now this is easier said than done. First, one needs to know how much revenue an AI web search application makes on a specific search. Second, one needs to attribute that cost to the different publishers that have produced the content used in the AI summary and search results. These two calculations require advanced conversational analytics. There are already various startups working on this, such as Prorata, which treats AI answers like mini-publications with multiple contributors, which get assigned an attribution amount and then they get paid for it.
Finally, the fourth monetisation model is subscription-based search. This model has not appeared yet, but we expect it to emerge in specialist or premium segments. For instance, Economist and other premium business sites, or academic journals or even expert communities, may block access to AI applications and launch their own kind of subscription-based search. That could be done on an individual basis or we could also see consortiums formed among many content providers coming together to to launch a common AI application that is verticalised or linked to a specific topic.
These four different business models will likely co-exist and their penetration may vary across regionals, driven by regulation.
For instance, Europe has strong copyright and privacy laws and may put some strong restrictions on content or make direct content compensation mandatory. That would result in solutions that are broader and coordinated. On the other hand, in the US, a region that is more hands-off from a regulatory perspective, we will likely see market solutions based on supply and demand, and driven by the publishers themselves.
💻 Opportunities for Tech founders and investors
So, Web search and content is definitely a market under disruption. And, whenever we see a market being disrupted, this means great opportunities for newcomers, thus for Tech founders and investors. We will analyse market opportunities in three layers: 1) the publisher/ e-commerce side, 2) the user side and 3) infrastructure.
🗞️ The publisher/ e-commerce side
In terms of publishers and e-commerce, one emerging area is definitely conversational search analytics. Traditional web analytics that work on click-based interactions cannot capture conversations and their content. While the existing analytics providers are trying to expand, this is really not coming natural to them. We are seeing many new startups, such as Profound, that come out and offer analytics on conversational search performance.
Another area is Conversational SEO, or what is sometimes called Generative Engine Optimisation (GEO). This is about how you optimise the content that you produce, not for humans to read, not for Google's page rank algorithm to rank it high up on the search results, but in order to make it easy for AI bots of AI web search apps to pick up and consume. There is an opportunity here, not just for new content generation, but also in terms of how you convert all the content archives, the existing content that is human or search-friendly, into more AI-friendly knowledge bases.
While some publishers and ecommerce sites will focus on how to improve discoverability in AI search, others will focus on diversification, i.e. working on other channels to attract users or leads. Companies that help publishers diversify their monetisation by launching subscriptions, managing online events, introducing licensing deals, etc, are expected to see increasing demand.
📱 The user side
On the user side, the obvious opportunity is AI-native search applications, which are already pretty well advanced with companies like Perplexity and You.com, that combine LLMs with search functionality and indexing of the Internet.
Another kind of direct application opportunity is the one of more intent-driven search or agentic search. Here users can simply express their intention and their needs, like, “I have a budget of $1,000, want a new mobile phone with a great camera and a strong battery life - give me a recommendation” or even saying “buy it for me” to go all the way to making the actual purchase. The natural place for this is the actual browser, which is most users’ number application in usage and is the gateway for search. This is why we recently saw Perplexity launching its own browser and some rumours about OpenAI working on its own search-based and agentic browser. Nevertheless, new independent AI browsers may see the market.
Another area of direct applications are Vertical Search models, that may come with direct monetisation, which is an area we mentioned earlier.
Wherever you see applications, you will also find enablers, i.e. solutions that help application providers to outsource parts of their workflow or Tech stack and focus on what’s core for them to deliver a differentiated offering. One great example is how to smoothly integrate sponsored content into AI responses. For instance, when a user asks about what is a fair credit card rate, the AI application could mention a bank’s name and credit card offering, with a link to purchase the product. Of course, this is not that simple. Users could get alienated with too much sponsored content, while relevance is also key: imagine that the bank in our example offers the highest credit card rate in the market. What would be the impact on AI user retention? These are things that still need to be figured out, and there are already some companies that are working on helping AI search applications do this in an effective way.
🧱 The infrastructure
One area of infrastructure that will likely need some work is related to content licensing and payment infrastructure. There, we see companies like TollBit and ProRata providing the plumbing for the AI content economy. This is about how you measure content usage, how you handle micropayments and licensing fees, linking it all to conversational analytics to both publishers and AI model providers. These are interesting models and could scale well, as they benefit from platform network effects. On the other hand, there will likely be challenges with standardisation and interoperability.
Another area of opportunity in infrastructure is related to content verification and moderation. Think solutions like watermarking that can help tracking if an AI output actually matches a specific source. Also, as more content is served by AI search applications, the question will come on who is responsible for the integrity and legality of this content. As some liability will also be placed on AI search apps, solutions that help analyse and moderate content will see growing demand. On top of that, as more content gets created by AI, bias will be another topic for consideration.
🔮 Looking ahead
Wrapping up, search and the online content value chain is definitely being disrupted and where we have a big disruption in a market, there's a lot of opportunities. As mentioned earlier, there are opportunities in various areas, such as conversational analytics, new B2C AI search experiences, infrastructure plays, etc.
Nevertheless, investors and founders in the space should be cautious.
To begin with, the pace of disruption is uncertain. Let's not forget that Google currently serves trillions of traditional searches and makes tens of billions of dollars from its search business; meanwhile, revenue in AI search certs is almost negligible in the bigger picture of search revenues. Also, many users have still not changed their search habits yet.
A second risk is platform dependency, which typically comes with the MarTech and AdTech market segments. For instance, if Google decided to implement its own pay-per-crawl system across Chrome and Android, that could potentially diminish an independent startup's role in the market. The same goes in case OpenAI and other AI API providers started seeing all other AI applications as competitive and blocked or limited access to their APIs. Some products could also break because of that.
Thirdly, there are economic viability risks for the new models related to online search. Overall, if AI search means that we're going to have less traffic and less ad impressions, then potentially the total size of revenues within search might shrink, leading to a shift to other channels. So it means that an investor could be backing up a company in the AI search market, which may end up being a smaller market than the current one (and what the investors expected).
Finally, there are also regulatory risks. So what happens if, for instance, a new EU directive came out and forced revenue sharing for any news content used in AI search results? This way, if revenue sharing was to become the norm, any news licensing or pay-per-crawl systems would become irrelevant, at least in that region.
Overall, we do see many interesting opportunities in the space, particularly when taking into account the huge and active buyer landscape. Many BigTech companies are obvious acquirers of players in space (think Google and Microsoft). You also have traditional Media or Telco companies that can acquire solutions to protect their content monetisation. And as the emerging AI search provides, such as OpenAI and Perplexity, are getting scale, they can become acquirers themselves.
This is definitely a market that is being disrupted quite fast. We're definitely excited about what's yet to come, so stay tuned as we follow its twists and turns.
See you next time for more AI insights.