Tag: AI

  • How to block AI crawlers with robots.txt [2 Updates]

    How to block AI crawlers with robots.txt [2 Updates]

    If you wanted your web page excluded from being crawled or indexed by search engines and other robots, robots.txt was your tool of choice, with some additional stuff like <meta name=”robots” value=”noindex” /> or <a href=”…” rel=”nofollow”> sprinkled in. It is getting more complicated with AI crawlers. Let’s have a look.