--- title: How to Block OpenAI's New AI Training Web Crawler description: Want to block BotGPT? Here are the two codes you need for either full or partial block, keeping your data away from OpenAI. image: https://www.hireawriter.us/hubfs/HAW%20blog%20-%202023-08-18T155845.420%20(1).png --- [Skip to the main content.](https://www.hireawriter.us/business/how-to-block-openais-new-ai-training-web-crawler#main-content) [![Untitled design (90)](https://www.hireawriter.us/hs-fs/hubfs/Untitled%20design%20(90).png?width=500&height=500&name=Untitled%20design%20(90).png "Untitled design (90)")](http://hireawriter.us) [![Untitled design (90)](https://www.hireawriter.us/hs-fs/hubfs/Untitled%20design%20(90).png?width=500&height=500&name=Untitled%20design%20(90).png "Untitled design (90)")](http://hireawriter.us) - [FOR BUSINESSES](https://www.hireawriter.us/for-businesses) - [SEO COPYWRITING](https://www.hireawriter.us/seo-copywriting) - [SOCIAL MEDIA CONTENT](https://www.hireawriter.us/social-media-services) - [WEBSITE COPYWRITING](https://www.hireawriter.us/website-copywriting) - [LEAD GEN COPYWRITING](https://www.hireawriter.us/lead-generation-copywriting) - [MARKETING STRATEGY](https://www.hireawriter.us/marketing-strategy) - [FOR AGENCIES](https://www.hireawriter.us/agency-partnership) - [WHITE LABELED COPYWRITING](https://www.hireawriter.us/white-labeled-copywriting) - [FRACTIONAL SEO LEADERSHIP](https://www.hireawriter.us/fractional-seo-leadership) - [TECHNICAL SEO](https://www.hireawriter.us/technical-seo) - [TRAINING](https://www.hireawriter.us/training) - [ABOUT HAW](https://www.hireawriter.us/about) - [JOBS](https://www.hireawriter.us/hire-a-writer-jobs) - [LEARN](https://www.hireawriter.us/learn) - [LEARN SEO](https://www.hireawriter.us/seo) - [LEARN BUSINESS CONTENT](https://www.hireawriter.us/business) - [LEARN SOCIAL MEDIA](https://www.hireawriter.us/social) - [LEARN STORYTELLING](https://www.hireawriter.us/storytelling) - [LEARN CREATIVE WRITING](https://www.hireawriter.us/creative) - [LEARN TECHNICAL WRITING](https://www.hireawriter.us/technical-content) - [LEARN FREELANCE WRITING](https://www.hireawriter.us/freelance) - [CONTACT](https://www.hireawriter.us/contact) Search Toggle Menu Search Toggle Menu - [FOR BUSINESSES](https://www.hireawriter.us/for-businesses) - [SEO COPYWRITING](https://www.hireawriter.us/seo-copywriting) - [SOCIAL MEDIA CONTENT](https://www.hireawriter.us/social-media-services) - [WEBSITE COPYWRITING](https://www.hireawriter.us/website-copywriting) - [LEAD GEN COPYWRITING](https://www.hireawriter.us/lead-generation-copywriting) - [MARKETING STRATEGY](https://www.hireawriter.us/marketing-strategy) - [FOR AGENCIES](https://www.hireawriter.us/agency-partnership) - [WHITE LABELED COPYWRITING](https://www.hireawriter.us/white-labeled-copywriting) - [FRACTIONAL SEO LEADERSHIP](https://www.hireawriter.us/fractional-seo-leadership) - [TECHNICAL SEO](https://www.hireawriter.us/technical-seo) - [TRAINING](https://www.hireawriter.us/training) - [ABOUT HAW](https://www.hireawriter.us/about) - [JOBS](https://www.hireawriter.us/hire-a-writer-jobs) - [LEARN](https://www.hireawriter.us/learn) - [LEARN SEO](https://www.hireawriter.us/seo) - [LEARN BUSINESS CONTENT](https://www.hireawriter.us/business) - [LEARN SOCIAL MEDIA](https://www.hireawriter.us/social) - [LEARN STORYTELLING](https://www.hireawriter.us/storytelling) - [LEARN CREATIVE WRITING](https://www.hireawriter.us/creative) - [LEARN TECHNICAL WRITING](https://www.hireawriter.us/technical-content) - [LEARN FREELANCE WRITING](https://www.hireawriter.us/freelance) - [CONTACT](https://www.hireawriter.us/contact) [Facebook](https://www.facebook.com/Hireawriteronline) [Instagram](https://www.instagram.com/joywriteryouell/) [Linkedin](https://www.linkedin.com/company/hireawriter/) [X](https://www.hireawriter.us/business/how-to-block-openais-new-ai-training-web-crawler#0) [YouTube](https://www.youtube.com/channel/UC8sZOTANgCCjtn29m4J0Z6A) [Medium](https://medium.com/hireawriter) [Tiktok](https://www.tiktok.com/@joyyouell) [Pinterest](https://www.pinterest.com/joyfromhireawriter/) 1 min read # How to Block OpenAI's New AI Training Web Crawler [![Picture of Writing Team](https://www.hireawriter.us/hubfs/HAW%20Logo%20T%20%26%20White%20square-3.png) Writing Team](https://www.hireawriter.us/business/author/writing-team) : Aug 18, 2023, 3:59:04 PM [Business](https://www.hireawriter.us/business/tag/business) [Technical](https://www.hireawriter.us/business/tag/technical) [AI](https://www.hireawriter.us/business/tag/ai) [ChatGPT](https://www.hireawriter.us/business/tag/chatgpt) [Chatbots](https://www.hireawriter.us/business/tag/chatbots) ![How to Block OpenAI's New AI Training Web Crawler](https://www.hireawriter.us/hubfs/HAW%20blog%20-%202023-08-18T155845.420%20(1).png) OpenAI, the creator of ChatGPT and other AI systems, has introduced a new web crawler named GPTBot, which is used to train AI models like GPT-5. While web crawlers are valuable for AI training by scanning websites and gathering data, there is a way to block their access. ## OpenAI's Web Crawlers ChatGPT, a highly capable AI system, is being continually enhanced, but concerns about its intelligence have emerged. OpenAI employs web crawlers to assist in training its large language models (LLMs), such as GPT-3.5 and GPT-4. Web crawlers are instrumental for indexing website content, used by both search engines and AI developers. These crawlers expedite the process of training LLMs with extensive data. ## OpenAI's GPTBot Access OpenAI's GPTBot, the new web crawler, can be allowed access to sites to enhance AI models' accuracy and safety. OpenAI has safeguards in place to exclude paywall-restricted, personally identifiable, or policy-violating content. ## How to Block GPTBOT For developers who wish to prevent GPTBot from accessing their sites, OpenAI provides instructions. GPTBot's access can be tailored by permitting it to crawl specific sections of a site while blocking others. This can be done by adding GPTBot to the site's robots.txt and adjusting access permissions accordingly. ### Full stop = "Disallow: /". ### Partial stop = "Allow: /directory-1/" and "Disallow: /directory-2/" (etc.) ## Is OpenAI Coming for Your Data? Though it's unclear whether GPTBot contributed to training existing LLMs like [GPT-3.5](https://www.hireawriter.us/business/chatgpt-versus-professional-writers) and GPT-4, it might be utilized for training GPT-5 in the future. OpenAI has sought to trademark the name "GPT-5," hinting at its potential development. While GPT-5's release date remains undisclosed, it is expected to surpass GPT-4 in size and capability. OpenAI has faced legal challenges regarding data usage, with accusations of data infringement. Websites like Stack Overflow, Reddit, and Twitter have considered charging AI entities for data access in response. - [Tweet](https://twitter.com/share) [![A Billion(s) Dollar Game: OpenAI’s Latest Funding Round](https://www.hireawriter.us/hubfs/HAW%20blog%20-%202024-09-14T083957.753.png)](https://www.hireawriter.us/business/a-billions-dollar-game-openais-latest-funding-round) 1 min read #### [A Billion(s) Dollar Game: OpenAI’s Latest Funding Round](https://www.hireawriter.us/business/a-billions-dollar-game-openais-latest-funding-round) ![Picture of Writing Team](https://www.hireawriter.us/hs-fs/hubfs/HAW%20Logo%20T%20%26%20White%20square-3.png?width=30&name=HAW%20Logo%20T%20%26%20White%20square-3.png) [Writing Team](https://www.hireawriter.us/business/author/writing-team) : Sep 14, 2024, 8:40:11 AM In the high-stakes world of AI, even billions can feel like pocket change. OpenAI is preparing for a new funding round that could value the company... [AI](https://www.hireawriter.us/business/tag/ai) [ChatGPT](https://www.hireawriter.us/business/tag/chatgpt) [News](https://www.hireawriter.us/business/tag/news) [Read More](https://www.hireawriter.us/business/a-billions-dollar-game-openais-latest-funding-round) [![OpenAI Could Lose $5 Billion This Year](https://www.hireawriter.us/hubfs/HAW%20blog%20(23)-1.png)](https://www.hireawriter.us/business/openai-could-lose-5-billion-this-year) 1 min read #### [OpenAI Could Lose $5 Billion This Year](https://www.hireawriter.us/business/openai-could-lose-5-billion-this-year) ![Picture of Writing Team](https://www.hireawriter.us/hs-fs/hubfs/HAW%20Logo%20T%20%26%20White%20square-3.png?width=30&name=HAW%20Logo%20T%20%26%20White%20square-3.png) [Writing Team](https://www.hireawriter.us/business/author/writing-team) : Jul 29, 2024, 10:22:20 AM OpenAI, the company behind the popular ChatGPT, is facing significant financial challenges according to a recent report. The Information, citing... [AI](https://www.hireawriter.us/business/tag/ai) [ChatGPT](https://www.hireawriter.us/business/tag/chatgpt) [News](https://www.hireawriter.us/business/tag/news) [Read More](https://www.hireawriter.us/business/openai-could-lose-5-billion-this-year) [![OpenAI's Personality Problem: Why GPT-4o Got Rolled Back (and What It Means)](https://www.hireawriter.us/hubfs/HAW%20blog%20-%202025-05-08T124045.444.png)](https://www.hireawriter.us/business/openais-personality-problem-why-gpt-4o-got-rolled-back-and-what-it-means) 5 min read #### [OpenAI's Personality Problem: Why GPT-4o Got Rolled Back (and What It Means)](https://www.hireawriter.us/business/openais-personality-problem-why-gpt-4o-got-rolled-back-and-what-it-means) ![Picture of Writing Team](https://www.hireawriter.us/hs-fs/hubfs/HAW%20Logo%20T%20%26%20White%20square-3.png?width=30&name=HAW%20Logo%20T%20%26%20White%20square-3.png) [Writing Team](https://www.hireawriter.us/business/author/writing-team) : May 8, 2025, 12:44:35 PM Imagine having a conversation with someone who agrees with literally everything you say. Every idea, no matter how half-baked. Every opinion, no... [AI](https://www.hireawriter.us/business/tag/ai) [ChatGPT](https://www.hireawriter.us/business/tag/chatgpt) [News](https://www.hireawriter.us/business/tag/news) [Read More](https://www.hireawriter.us/business/openais-personality-problem-why-gpt-4o-got-rolled-back-and-what-it-means) HIRE US HIRE US - [MARKETING SERVICES](https://www.hireawriter.us/marketing-content) - [SEO SERVICES](https://www.hireawriter.us/seo-services) - [FOR YOUR AGENCY](https://www.hireawriter.us/agency-partnership) - [COACHING](https://www.hireawriter.us/coaching) ABOUT HIRE A WRITER (HAW) ABOUT HIRE A WRITER (HAW) - [ABOUT](https://www.hireawriter.us/about) - [CONTACT](https://www.hireawriter.us/contact) - [JOBS](https://www.hireawriter.us/hire-a-writer-jobs) LEARN LEARN - [BUSINESS CONTENT](https://www.hireawriter.us/business) - [SEO STRATEGY BLOG](https://www.hireawriter.us/seo) - [SOCIAL MEDIA MARKETING BLOG](https://www.hireawriter.us/social) - [STORYTELLING BLOG](https://www.hireawriter.us/storytelling) - [CREATIVE WRITING BLOG](https://www.hireawriter.us/creative) - [FREELANCE WRITING BLOG](https://www.hireawriter.us/freelance) - [TECHNICAL WRITING BLOG](https://www.hireawriter.us/technical-content) HOUSE OF BRANDS HOUSE OF BRANDS | ![2-Mar-03-2024-04-31-09-9544-PM](https://www.hireawriter.us/hs-fs/hubfs/2-Mar-03-2024-04-31-09-9544-PM.png?width=100&height=67&name=2-Mar-03-2024-04-31-09-9544-PM.png) | [![TRAINING (2)](https://www.hireawriter.us/hs-fs/hubfs/TRAINING%20(2).png?width=100&height=100&name=TRAINING%20(2).png)](https://www.academyofcontinuingeducation.com/) | | --- | --- | | [![5-3](https://www.hireawriter.us/hs-fs/hubfs/5-3.png?width=100&height=67&name=5-3.png)](https://babygotbacklink.com/) | [![4-Mar-03-2024-04-31-09-9108-PM](https://www.hireawriter.us/hs-fs/hubfs/4-Mar-03-2024-04-31-09-9108-PM.png?width=100&height=67&name=4-Mar-03-2024-04-31-09-9108-PM.png)](https://joyyouell.com/) | | ![3-Mar-03-2024-04-31-09-9008-PM](https://www.hireawriter.us/hs-fs/hubfs/3-Mar-03-2024-04-31-09-9008-PM.png?width=100&height=67&name=3-Mar-03-2024-04-31-09-9008-PM.png) | [![1-1](https://www.hireawriter.us/hs-fs/hubfs/1-1.png?width=100&height=67&name=1-1.png)](https://winsomepr.com/) |   Hire a Writer: 31 Newton Ave Woodbury, NJ 08096     [![HAW Logo T & White square](https://www.hireawriter.us/hs-fs/hubfs/HAW%20Logo%20T%20%26%20White%20square.png?width=300&height=169&name=HAW%20Logo%20T%20%26%20White%20square.png)](https://www.hireawriter.us/oldhome) - Legal Stuff - Privacy Policy © 2026 Hire a Writer™ [Facebook](https://www.facebook.com/Hireawriteronline)[Instagram](https://www.instagram.com/joywriteryouell/)[Linkedin](https://www.linkedin.com/company/hireawriter/)[YouTube](https://www.youtube.com/channel/UC8sZOTANgCCjtn29m4J0Z6A)[Medium](https://medium.com/hireawriter) [Tiktok](https://www.tiktok.com/@joyyouell) [Pinterest](https://www.pinterest.com/joyfromhireawriter/) Return to top ```json { "@context" : "https://schema.org", "@type" : "BlogPosting", "author" : { "@type" : "Person", "name" : "Writing Team", "url" : "https://www.hireawriter.us/business/author/writing-team" }, "dateModified" : "2023-08-18T19:59:04.639Z", "datePublished" : "2023-08-18T19:59:04.000Z", "headline" : "How to Block OpenAI's New AI Training Web Crawler", "image" : [ "https://www.hireawriter.us/hubfs/HAW%20blog%20-%202023-08-18T155845.420%20(1).png" ], "mainEntityOfPage" : { "@id" : "https://www.hireawriter.us/business/how-to-block-openais-new-ai-training-web-crawler", "@type" : "WebPage" }, "publisher" : { "@type" : "Organization", "logo" : { "@type" : "ImageObject", "url" : "https://www.hireawriter.us/hubfs/HAW%20Logo%20T%20&%20White%20square-1.png" } } } ```