Gated Content Is Invisible to AI Search: What to Publish Openly

For years the standard B2B website playbook was simple: put the valuable material behind a form, collect an email address, and let sales follow up. That trade made sense when the visitor was a person who had already found you. It makes much less sense when the first reader is an AI system deciding which businesses to mention in an answer. Gated content is invisible to AI search, because an assistant can only cite what it can read, and nothing behind a lead form is readable.

The short answer: publish openly the answers a buyer needs before they would talk to you, structure each page so one question gets one direct answer in plain HTML text, make sure the crawlers you want can reach it, and keep the form for material that is genuinely deep. This is a decision about how your site exposes information, which makes it as much an engineering question as a marketing one.

Why Gated Content Is Invisible to AI Search

Google is explicit about what it takes to appear in its AI features. According to Google Search Central’s guide to AI features and your website, a page must be indexed and eligible to be shown in Google Search with a snippet to appear as a supporting link in AI Overviews or AI Mode, and there are no additional technical requirements. A whitepaper that only exists after someone submits a form is not indexed and has no snippet. It cannot be a supporting link.

The assistants that run their own search work the same way. OpenAI’s crawler documentation says OAI-SearchBot is the agent that surfaces websites in ChatGPT’s search features, and that sites opted out of it will not be shown in ChatGPT search answers, though they can still appear as navigational links. Anthropic’s crawler documentation describes Claude-SearchBot, which indexes content for search, and Claude-User, which fetches a page when a person asks Claude a question.

None of these agents fills in your contact form. Whatever is behind it does not exist for them.

The Question Has Replaced the Brand Name

A buyer who already knows your name will find you whichever way they search. The open question is how buyers who do not know you yet find a shortlist. When a buyer describes the problem to an assistant (“who can connect our warehouse system to our customers’ ordering portal?”) instead of typing a company name, they read the answer it assembles, with links to the pages it drew on.

A name the buyer never typed cannot help them; a page that answers the question can. A smaller business whose site plainly answers what the buyer asked has an opening that does not depend on outspending a larger competitor, but only if the answer is on a page a crawler can read.

What to Publish Openly

Start with the questions your sales team hears on every first call. If a buyer needs the answer before they would contact you, it belongs on a public page.

• What you do and do not do. Name the problems you take on and the ones you turn away.
• How the work runs. The steps from first conversation to delivery, who is involved, and what the client is expected to provide.
• What drives the cost. Not every business can publish a price, and some should not. Every business can explain what makes a project larger or smaller, and how engagements are usually structured.
• Technical specifics. Supported platforms, integration methods, security and compliance posture, data handling. IT buyers tend to ask about these early, and a vague answer gives them nothing to compare.
• Real answers to real questions. A frequently asked questions section with specific answers, not marketing copy rephrased as questions.

Buyers judge how a company thinks long before they talk to it. Our guide to evaluating a software development agency without being technical tells buyers to do exactly that: watch how a company thinks and communicates, not just what it says it builds. Open answers are where that judgment shows.

Structure Each Page to Answer One Question

Publishing the information is half of the work. The other half is structure, and most marketing sites were designed around a page layout rather than around questions.

1. One question, one answer, near the top. Phrase headings the way a buyer would ask the question, and put the direct answer in the first sentence or two beneath it. Detail and nuance come after.
2. Text in the HTML you serve. Google’s guide recommends making sure that important content is available in textual form. An answer that lives in an image or a slide embedded as a picture is harder to read and quote, and one in a linked PDF is split from the page that should rank. Serve the answer in the page’s own HTML rather than fetching it with scripts after load; it is the simplest way to be sure any crawler sees what a person sees.
3. Internal links between related answers. Google lists making content easily findable through internal links among its best practices for AI features. A service page should link to the articles that explain it, and the articles should link back.
4. Structured data that matches the page. Google asks that structured data match the visible text on the page. Markup that describes content a visitor cannot see does not help.
5. No special AI file required. The same Google guide says you do not need new machine-readable files, AI text files or special schema.org markup to appear in its AI features. Fundamentals come first.

Most of these are decisions about templates and rendering, not about copy. If your site makes them hard, that is a constraint in how it was built. Our web and mobile app development team works on exactly that layer.

Check That the Crawlers You Want Can Reach You

A page can be public and still unreachable. Robots.txt rules, CDN bot protection and firewall settings decide which automated visitors get through, and those settings were often chosen before anyone was thinking about AI search.

The AI vendors separate search from training, which lets you make a deliberate choice:

• OpenAI’s documentation states that its settings are independent: a site can allow OAI-SearchBot so it appears in ChatGPT search results while disallowing GPTBot, which collects content that may be used to train its models.
• Anthropic lists ClaudeBot for training, Claude-User for user-requested fetches and Claude-SearchBot for search, and warns that disabling Claude-SearchBot may reduce a site’s visibility in user search results.
• Google’s guide points to Google-Extended for limiting AI training and grounding in some of Google’s other systems, while appearance in AI Overviews follows normal Search crawling and its snippet controls.

Blocking every bot is the easy setting, and it quietly removes your site from answers your buyers read. Decide which agents you want, write the rules on purpose, and check the CDN as well as robots.txt, since Google names both.

What to Keep Gated

Gating is not wrong, just overused. Keep a form in front of material whose value depends on depth or on the visitor’s own information:

• A tool or calculator that needs the visitor’s data to produce a result
• Full templates, workbooks or implementation checklists
• Original research in full, with its data
• Recorded workshops and long-form training

The pattern that works is an open summary with gated depth. Publish the findings, the method and the key recommendations on a public page that answers the question, then offer the full workbook behind the form. The crawler gets the answer; the motivated reader still has a reason to hand over an email address.

If you run a registration wall over content you do want indexed, Google’s paywalled content guidance explains the structured data that lets it tell a legitimate wall from cloaking.

Where to Start

You probably do not need a new website. List the ten questions buyers ask most, find the public page that answers each one, and check whether the answer appears in the first two sentences as text. Then read your robots.txt and CDN bot rules against the agents above. The gaps you find are usually fixable inside the current site, the same incremental approach we describe in modernizing legacy systems without a rewrite.

If AI search is one piece of a broader push to take an AI use case into production, see how we approach AI implementation as engineering.

Frequently Asked Questions

Does gated content hurt AI search visibility?

Yes, for the gated material itself. Google requires a page to be indexed and eligible for a snippet before it can appear as a supporting link in AI Overviews or AI Mode, and content behind a lead form is neither. Publish a public summary that answers the question and gate only the depth.

Should we block AI crawlers in robots.txt?

Decide agent by agent. OpenAI and Anthropic both document separate agents for search and for model training, so you can allow the search agents that put you in answers while disallowing the training agents if you prefer.

Do we need an llms.txt file or special schema to appear in AI answers?

Not for Google’s AI features. Google states that no new machine-readable files, AI text files or special schema.org markup are needed. Indexable pages, text answers, internal links and structured data that matches the visible page matter more.

Make Your Site the Clearest Answer

If your site was built to capture leads rather than answer questions, we can help you work out what to open up, how to restructure it, and what to change in the build. Start a conversation with our team.