mr.w0bb1t<p>"[..] 65% of our most expensive traffic comes from <a href="https://tldr.nettime.org/tags/bots" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>bots</span></a>" · How <a href="https://tldr.nettime.org/tags/crawlers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>crawlers</span></a> impact the operations of the <span class="h-card" translate="no"><a href="https://wikimedia.social/@wikimediafoundation" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@<span>wikimediafoundation</span></a></span> projects.</p><p>👉🏻 <a href="https://diff.wikimedia.org/2025/04/01/how-crawlers-impact-the-operations-of-the-wikimedia-projects/" rel="nofollow noopener noreferrer" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">diff.wikimedia.org/2025/04/01/</span><span class="invisible">how-crawlers-impact-the-operations-of-the-wikimedia-projects/</span></a></p>
Recent searches
No recent searches
Search options
Only available when logged in.
en.osm.town is one of the many independent Mastodon servers you can use to participate in the fediverse.
An independent, community of OpenStreetMap people on the Fediverse/Mastodon.
Funding graciously provided by the OpenStreetMap Foundation.
Administered by:
Server stats:
261active users
en.osm.town: About · Profiles directory · Privacy policy
Mastodon: About · Get the app · Keyboard shortcuts · View source code · v4.3.7
#crawlers
1 post · 1 participant · 1 post today
Βασίλης Βαλατσός<p>The best way to stop <a href="https://social.apotheke.earth/tags/chinese" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>Chinese</span></a> <a href="https://social.apotheke.earth/tags/ai" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>ai</span></a> <a href="https://social.apotheke.earth/tags/crawlers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>crawlers</span></a> DDOSing your site is to make sure you are blocked by the <a href="https://social.apotheke.earth/tags/great" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>Great</span></a> <a href="https://social.apotheke.earth/tags/firewall" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>Firewall</span></a></p><p><a href="https://aethrvmn.gr/glory-ccp" rel="nofollow noopener noreferrer" target="_blank">https://aethrvmn.gr/glory-ccp</a></p><p>Now I just need a copypasta like that for <a href="https://social.apotheke.earth/tags/israel" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>Israel</span></a> so that <a href="https://social.apotheke.earth/tags/us" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>US</span></a> crawlers also stop crawling</p>
katzenberger 🇺🇦<p>"<a href="https://mastodon.de/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>AI</span></a>" <a href="https://mastodon.de/tags/crawlers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>crawlers</span></a> are a cancerous disease that must be eradicated, not just fended off.</p><p>»The costs are both technical and financial. The Read the Docs project reported that blocking AI crawlers immediately decreased their <a href="https://mastodon.de/tags/traffic" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>traffic</span></a> by 75 percent, going from 800GB per day to 200GB per day. This change saved the project approximately $1,500 per month in bandwidth costs, according to their blog post "AI crawlers need to be more respectful."«</p><p><a href="https://arstechnica.com/ai/2025/03/devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries/" rel="nofollow noopener noreferrer" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">arstechnica.com/ai/2025/03/dev</span><span class="invisible">s-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries/</span></a></p>
Veronica Olsen 🏳️🌈🇳🇴🌻<p>Who could have guessed that an industry whose entire business model is based on theft would behave like malware attacks on the Internet? 🤔</p><p><a href="https://arstechnica.com/ai/2025/03/devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries/?utm_source=mastodon&utm_medium=social" rel="nofollow noopener noreferrer" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">arstechnica.com/ai/2025/03/dev</span><span class="invisible">s-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries/?utm_source=mastodon&utm_medium=social</span></a></p><p><a href="https://mastodon.online/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>AI</span></a> <a href="https://mastodon.online/tags/DDoS" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>DDoS</span></a> <a href="https://mastodon.online/tags/Crawlers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>Crawlers</span></a></p>
Albert Cardona<p>Web crawlers are out of control. If anyone has a suggestion on how to block them entirely, not via robots.txt but with e.g., a prompt that only a human can answer, I'd appreciate it.</p><p>Here, a list of the operating systems of my website's visitors.</p><p><a href="https://mathstodon.xyz/tags/webmaster" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>webmaster</span></a> <a href="https://mathstodon.xyz/tags/crawlers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>crawlers</span></a></p>
mr.w0bb1t<p>Trap <a href="https://tldr.nettime.org/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>AI</span></a> Aggressive <a href="https://tldr.nettime.org/tags/Crawlers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>Crawlers</span></a> .. This is a simple proof of concept of using a Markov Chain to generate an infinitely large website.</p><p>👉🏻 <a href="https://github.com/gw1urf/spigot" rel="nofollow noopener noreferrer" translate="no" target="_blank"><span class="invisible">https://</span><span class="">github.com/gw1urf/spigot</span><span class="invisible"></span></a></p>
Codeberg.org<p><span class="h-card" translate="no"><a href="https://wetdry.world/@iagondiscord" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@<span>iagondiscord</span></a></span> The problem is that it's not targeting Codeberg. It's the <a href="https://social.anoxinon.de/tags/AIgoldrush" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>AIgoldrush</span></a>. The web was completely crawled, just not by everyone yet. So startups start their <a href="https://social.anoxinon.de/tags/crawlers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>crawlers</span></a>, carelessly and explicitly ignoring robots.txt to get the <a href="https://social.anoxinon.de/tags/biggestdata" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>biggestdata</span></a>.</p><p>It does not matter if the web can no longer serve humanity due to this. Training the <a href="https://social.anoxinon.de/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>AI</span></a> is the only thing that matters.</p><p>Maybe a bit like a sacrifice for faith.</p><p>~f <a href="https://social.anoxinon.de/tags/goldrush" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>goldrush</span></a></p>
George Ellenburg (he/him/his)<p><span>Have you taken a look at your website's access logs lately?<br><br>There are a LOT of spiders and bots and crawlers now. It's almost to the point where I'm getting more traffic from bots than I am from actual people.<br><br>And thus my new </span><code>robots.txt</code><span> file is born:<br></span></p><pre><code>User-agent: *
Disallow: /
User-agent: DuckDuckBot
Allow: /
User-agent: Baiduspider
Allow: /
User-agent: YandexBot
Allow: /
User-agent: Slurp
Allow: /
User-agent: Bingbot
Allow: /
User-agent: Googlebot
Allow: /</code></pre><span><br>Take that you mysterious bot and spider!<br><br></span><a href="https://bofh.social/tags/WebHosting" rel="nofollow noopener noreferrer" target="_blank">#WebHosting</a><span> </span><a href="https://bofh.social/tags/websites" rel="nofollow noopener noreferrer" target="_blank">#websites</a><span> </span><a href="https://bofh.social/tags/bots" rel="nofollow noopener noreferrer" target="_blank">#bots</a><span> </span><a href="https://bofh.social/tags/crawlers" rel="nofollow noopener noreferrer" target="_blank">#crawlers</a><span> </span><a href="https://bofh.social/tags/robotstxt" rel="nofollow noopener noreferrer" target="_blank">#robotstxt</a><span> </span><a href="https://bofh.social/tags/AdminStuff" rel="nofollow noopener noreferrer" target="_blank">#AdminStuff</a><p></p>
Mr. 30㎝ 🍆💦 :18only: :verified_gay: :heart_trans:<p>If you have a <a href="https://nsfw.lgbt/tags/website" rel="nofollow noopener noreferrer" target="_blank">#website</a> and <a href="https://nsfw.lgbt/tags/produce" rel="nofollow noopener noreferrer" target="_blank">#produce</a> <a href="https://nsfw.lgbt/tags/content" rel="nofollow noopener noreferrer" target="_blank">#content</a>, and you host it behind <a href="https://nsfw.lgbt/tags/Cloudflare" rel="nofollow noopener noreferrer" target="_blank">#Cloudflare</a>, don't forget to go into your website <a href="https://nsfw.lgbt/tags/Security" rel="nofollow noopener noreferrer" target="_blank">#Security</a> <a href="https://nsfw.lgbt/tags/settings" rel="nofollow noopener noreferrer" target="_blank">#settings</a> under <a href="https://nsfw.lgbt/tags/Bots" rel="nofollow noopener noreferrer" target="_blank">#Bots</a> and <a href="https://nsfw.lgbt/tags/enable" rel="nofollow noopener noreferrer" target="_blank">#enable</a> "<a href="https://nsfw.lgbt/tags/Block" rel="nofollow noopener noreferrer" target="_blank">#Block</a> <a href="https://nsfw.lgbt/tags/AI" rel="nofollow noopener noreferrer" target="_blank">#AI</a> <a href="https://nsfw.lgbt/tags/Scrapers" rel="nofollow noopener noreferrer" target="_blank">#Scrapers</a> and <a href="https://nsfw.lgbt/tags/Crawlers" rel="nofollow noopener noreferrer" target="_blank">#Crawlers</a>"!</p>
LeanderZeichnet❦<p><a href="https://mastodon.art/tags/leanderzeichnet" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>leanderzeichnet</span></a> a friendly <a href="https://mastodon.art/tags/mimic" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>mimic</span></a> for all you <a href="https://mastodon.art/tags/dnd" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>dnd</span></a> <a href="https://mastodon.art/tags/dungeoncrawler" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>dungeoncrawler</span></a> <a href="https://mastodon.art/tags/crawlers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>crawlers</span></a> <a href="https://mastodon.art/tags/pnp" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>pnp</span></a> <a href="https://mastodon.art/tags/gamers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>gamers</span></a></p>
LeanderZeichnet❦<p><a href="https://mastodon.art/tags/leanderzeichnet" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>leanderzeichnet</span></a> a friendly undead skeleton for all you <a href="https://mastodon.art/tags/pnp" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>pnp</span></a> <a href="https://mastodon.art/tags/gamers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>gamers</span></a> and <a href="https://mastodon.art/tags/dungeonsanddragons" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>dungeonsanddragons</span></a> <a href="https://mastodon.art/tags/dnd" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>dnd</span></a> <a href="https://mastodon.art/tags/crawlers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>crawlers</span></a></p>
Feilner IT<p><a href="https://mastodon.social/tags/konterfAI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>konterfAI</span></a>, the <a href="https://mastodon.social/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>AI</span></a> <a href="https://mastodon.social/tags/modelpoisoner" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>modelpoisoner</span></a> for unfriendly, respectless and malicious AI <a href="https://mastodon.social/tags/scrapers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>scrapers</span></a> / <a href="https://mastodon.social/tags/crawlers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>crawlers</span></a> has now an interesting statistics function that collects data about the nasty "guests" that don't respect your robots.txt.... Enjoy! <br><a href="https://mastodon.social/tags/robots_txt" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>robots_txt</span></a> <br>See update on <br><a href="https://korium.org/2024/08/02/konterfai/" rel="nofollow noopener noreferrer" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">korium.org/2024/08/02/konterfa</span><span class="invisible">i/</span></a> <br>and version 0.2.0 on <br><a href="https://codeberg.org/konterfai/konterfai/releases/tag/v0.2.0" rel="nofollow noopener noreferrer" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">codeberg.org/konterfai/konterf</span><span class="invisible">ai/releases/tag/v0.2.0</span></a></p>
George Ellenburg (he/him/his)<p><span>According to </span><a href="https://bofh.social/tags/Microsoft" rel="nofollow noopener noreferrer" target="_blank">#Microsoft</a><span>, all content on the Web is ripe for being stolen to train </span><a href="https://bofh.social/tags/AI" rel="nofollow noopener noreferrer" target="_blank">#AI</a><span> models unless you have a </span><code>robots.txt</code><span> file in the root of your site and explicitly disallow them.<br><br>In other news, </span><a href="https://bofh.social/tags/Cloudflare" rel="nofollow noopener noreferrer" target="_blank">#Cloudflare</a><span> now has a setting to </span><a href="https://bofh.social/tags/block" rel="nofollow noopener noreferrer" target="_blank">#block</a><span> all AI </span><a href="https://bofh.social/tags/crawlers" rel="nofollow noopener noreferrer" target="_blank">#crawlers</a><span> and </span><a href="https://bofh.social/tags/bots" rel="nofollow noopener noreferrer" target="_blank">#bots</a><span> looking for </span><a href="https://bofh.social/tags/TrainingData" rel="nofollow noopener noreferrer" target="_blank">#TrainingData</a><span>. Have you turned it on for your website, yet? We have for all of ours.<br><br></span><a href="https://slashdot.org/story/24/07/07/0054220/microsofts-ai-ceo-web-content-without-a-robotstxt-file-is-freeware-for-ai-training" rel="nofollow noopener noreferrer" target="_blank">slashdot.org/story/24/07/07/0054220/microsofts-ai-ceo-web-content-without-a-robotstxt-file-is-freeware-for-ai-training</a></p>
Ecologia Digital<p>"The <span class="h-card" translate="no"><a href="https://mastodon.archive.org/@internetarchive" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@<span>internetarchive</span></a></span> <a href="https://mato.social/tags/crawlers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>crawlers</span></a> moved through the pages of a hobby store multiple times over the yrs, capturing HTML, images,& time-stamping the process.<br>This agnostic, wide-ranging crawl likely represented both the original source of the image, and a persistent, dependable URL to reference back to it.<br>This is the mission of the <a href="https://mato.social/tags/WaybackMachine" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>WaybackMachine</span></a>—be the dependable, accessible connection to web history."<br><a href="https://blog.archive.org/2024/06/01/the-backrooms-of-the-internet-archive/" rel="nofollow noopener noreferrer" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">blog.archive.org/2024/06/01/th</span><span class="invisible">e-backrooms-of-the-internet-archive/</span></a></p>
nullagent<p>A few years ago I wrote a crawler that scrapes crowdsupply's website to build economic models of all of the projects listed. 🥷🏾</p><p><a href="https://github.com/sevenbitbyte/crowdsupply-crawler" rel="nofollow noopener noreferrer" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">github.com/sevenbitbyte/crowds</span><span class="invisible">upply-crawler</span></a></p><p><a href="https://partyon.xyz/tags/Economics" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>Economics</span></a> <a href="https://partyon.xyz/tags/Hacking" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>Hacking</span></a> <a href="https://partyon.xyz/tags/Crawlers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>Crawlers</span></a> <a href="https://partyon.xyz/tags/crowdfunding" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>crowdfunding</span></a></p>
Erik Jonker<p>Impressive overview of bots crawling the web, collecting content. For most we do not know exactly what they are used for which illustrates for me the futility of blocking OpenAI access to your website. If you don't want to be crawled or used as training data for AI, don't publish on the publicly available internet I would think.<br><a href="https://darkvisitors.com/agents" rel="nofollow noopener noreferrer" translate="no" target="_blank"><span class="invisible">https://</span><span class="">darkvisitors.com/agents</span><span class="invisible"></span></a><br><a href="https://mastodon.social/tags/bots" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>bots</span></a> <a href="https://mastodon.social/tags/crawlers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>crawlers</span></a> <a href="https://mastodon.social/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>AI</span></a></p>
Marcel Waldvogel<p><span class="h-card" translate="no"><a href="https://infosec.exchange/@realn2s" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@<span>realn2s</span></a></span> <span class="h-card" translate="no"><a href="https://henshaw.social/@jon" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@<span>jon</span></a></span> <span class="h-card" translate="no"><a href="https://mastodon.social/@nixCraft" class="u-url mention" rel="nofollow noopener noreferrer" target="_blank">@<span>nixCraft</span></a></span> The list above has many non-AI <a href="https://waldvogel.family/tags/crawlers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>crawlers</span></a>; also, the AdsBot and Mediapartners <a href="https://waldvogel.family/tags/bots" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>bots</span></a> in the OP seem to have no relation to <a href="https://waldvogel.family/tags/AI" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>AI</span></a>, either.</p><p>Here is my list of AI crawlers (will add <a href="https://waldvogel.family/tags/Anthropic" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>Anthropic</span></a> after verification). <a href="https://netfuture.ch/2023/07/blocking-ai-crawlers-robots-txt-chatgpt/#ai-crawlers" rel="nofollow noopener noreferrer" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">netfuture.ch/2023/07/blocking-</span><span class="invisible">ai-crawlers-robots-txt-chatgpt/#ai-crawlers</span></a></p>
Guilherme<p>Algumas alternativas para impedir que <a href="https://ursal.zone/tags/bots" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>bots</span></a> alimentem os <a href="https://ursal.zone/tags/LLM" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>LLM</span></a> com o seu site: <a href="https://neil-clarke.com/block-the-bots-that-feed-ai-models-by-scraping-your-website/" rel="nofollow noopener noreferrer" translate="no" target="_blank"><span class="invisible">https://</span><span class="ellipsis">neil-clarke.com/block-the-bots</span><span class="invisible">-that-feed-ai-models-by-scraping-your-website/</span></a></p><p>São sugeridas entradas para o robots.txt, para o novo ai.txt e uns "truques" com .htaccess, firewalls e tags meta.</p><p>Reiterar nunca é demais: jogar o que você escreve nas máquinas de plágio deveria ser opt-in. Infelizmente não é.</p><p><a href="https://ursal.zone/tags/crawlers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>crawlers</span></a> <a href="https://ursal.zone/tags/GPT" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>GPT</span></a> <a href="https://ursal.zone/tags/ChatGPT" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>ChatGPT</span></a> <a href="https://ursal.zone/tags/CCBot" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>CCBot</span></a> <a href="https://ursal.zone/tags/FacebookBot" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>FacebookBot</span></a> <a href="https://ursal.zone/tags/GPTBot" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>GPTBot</span></a></p>
Barry Schwartz<p>Google-Extended is a new Googlebot, a new Google crawler, you can use in your robots.txt to tell Bard or another Google AI to stop using your content <a href="https://www.seroundtable.com/google-extended-crawler-ai-bard-36130.html" rel="nofollow noopener noreferrer" target="_blank"><span class="invisible">https://www.</span><span class="ellipsis">seroundtable.com/google-extend</span><span class="invisible">ed-crawler-ai-bard-36130.html</span></a></p><p><a href="https://c.im/tags/google" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>google</span></a> <a href="https://c.im/tags/bard" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>bard</span></a> <a href="https://c.im/tags/ai" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>ai</span></a> <a href="https://c.im/tags/search" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>search</span></a> <a href="https://c.im/tags/seo" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>seo</span></a> <a href="https://c.im/tags/robots" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>robots</span></a>.txt <a href="https://c.im/tags/googlebot" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>googlebot</span></a> <a href="https://c.im/tags/crawlers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>crawlers</span></a></p>
Barry Schwartz<p>GPTBot - OpenAI's ChatGPT's web crawler, you can now block it with your robots.txt file - if you want <a href="https://www.seroundtable.com/openais-chatgpt-web-crawler-gptbot-35835.html" rel="nofollow noopener noreferrer" target="_blank"><span class="invisible">https://www.</span><span class="ellipsis">seroundtable.com/openais-chatg</span><span class="invisible">pt-web-crawler-gptbot-35835.html</span></a></p><p><a href="https://c.im/tags/chatgpt" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>chatgpt</span></a> <a href="https://c.im/tags/openai" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>openai</span></a> <a href="https://c.im/tags/gptbot" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>gptbot</span></a> <a href="https://c.im/tags/bots" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>bots</span></a> <a href="https://c.im/tags/spiders" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>spiders</span></a> <a href="https://c.im/tags/crawlers" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>crawlers</span></a> <a href="https://c.im/tags/search" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>search</span></a> <a href="https://c.im/tags/ai" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>ai</span></a> <a href="https://c.im/tags/indexing" class="mention hashtag" rel="nofollow noopener noreferrer" target="_blank">#<span>indexing</span></a></p>
ExploreLive feeds
Mastodon is the best way to keep up with what's happening.
Follow anyone across the fediverse and see it all in chronological order. No algorithms, ads, or clickbait in sight.
Create accountLoginDrag & drop to upload