If you can read this on the site, pg_cron dispatched wp-publish on its own. No human pressed anything.
Blog
-
how do AI crawlers decide which pages to read
AI crawlers like GPTBot use your robots.txt file, sitemaps, and link structure to decide what to crawl. They prioritize pages that are linked from other pages, updated frequently, and marked as important in your sitemap.
## Robots.txt controls crawler access
Your robots.txt file tells GPTBot and other AI crawlers which pages to read and which to skip. If you don’t block a path, crawlers assume they can access it. The file lives at the root of your domain (example.com/robots.txt).
## Sitemaps help crawlers find pages
A sitemap.xml lists your pages and tells crawlers how often they change. Crawlers check sitemaps to find pages that might not be linked from other places. You can submit your sitemap to AI crawler providers.
## Link structure determines priority
Pages linked from your homepage or other popular pages get crawled more often. Crawlers follow links to discover new pages. Deeper pages (more clicks away from the homepage) get crawled less frequently.
## Page freshness affects crawl frequency
Pages that update regularly get revisited more often. Crawlers track when they last saw changes to a page and adjust their schedule accordingly.
## HTTP status codes signal page status
200 means the page is live and readable. 404 means the page is gone. 301 redirects tell crawlers to follow to a new location. Crawlers use these signals to decide whether to keep reading a page or move on.
## Meta tags provide hints
The noindex tag tells crawlers not to include a page in their index. The nofollow tag tells crawlers not to follow links on that page. These tags give you fine-grained control over what gets crawled.