---
title: Adding a robots.txt file to your website
description: The robots.txt file can allow or disallow crawlers - like Google's search bot. You can upload a robots.txt file in SiteManager.
---
[Skip to content](https://support.sitemanager.io/en/seo/advanced/adding-a-robots-txt-file#main-content)
[](https://support.sitemanager.io/en?hsLang=en)
Open main navigation
Close main navigation
- [Go to sitemanager.io](https://www.sitemanager.io/)
[Go to sitemanager.io](https://www.sitemanager.io/)
How can we help?
- There are no suggestions because the search field is empty.
1. [Knowledge Base](https://support.sitemanager.io/en?hsLang=en)
2. [SEO](https://support.sitemanager.io/en/seo?hsLang=en)
3. [Advanced](https://support.sitemanager.io/en/seo?hsLang=en#advanced)
# Adding a robots.txt file to your website
## The robots.txt file can allow or disallow crawlers - like Google's search bot. You can upload a robots.txt file in SiteManager.
In this article you'll learn how you can add a robots.txt file to your website. The robots.txt file can be used to (dis)allow crawlers to certain parts of your website. By default, a crawler will be able to access everything (except for pages with the noindex tag). For SEO reasons, it's considered a good practice to add a robots.txt file anyway.
To upload the file, head to Project > SEO > Sitemap. This window will open:
In this window you can generate a [sitemap](https://support.sitemanager.io/en/sitemanager-cms/content-management/build-and-submit-a-sitemap-in-sitemanager?hsLang=en) (using the disk icon), upload a custom sitemap, or upload a robots.txt file.
### Creating a robots.txt file
A robots.txt file is a TXT file which you can simply write using windows notepad. This file contains some rules for bots. To learn more about writing your own custom robots.txt file, read [this article by Google](https://developers.google.com/search/docs/advanced/robots/create-robots-txt).
If your goal is to optimize your website for SEO and allow all crawlers, you can upload a robots.txt file that allows everything:
```
User-agent: *Disallow:
```
With this rule you disallow nothing, which means crawlers can look into your entire website. The asterisk (\*) is used as a wildcard so the rule applies to any crawler.
- [Getting Started](https://support.sitemanager.io/en/getting-started?hsLang=en)
- [Dashboard](https://support.sitemanager.io/en/dashboard?hsLang=en#main-content)
- [Getting Started](https://support.sitemanager.io/en/dashboard?hsLang=en#getting-started)
- [Project Dashboard](https://support.sitemanager.io/en/dashboard?hsLang=en#project-dashboard)
- [White Label & Add-ons](https://support.sitemanager.io/en/dashboard?hsLang=en#white-label-add-ons)
- [Go Live](https://support.sitemanager.io/en/dashboard?hsLang=en#go-live)
- [Account](https://support.sitemanager.io/en/account?hsLang=en#main-content)
- [Account Settings](https://support.sitemanager.io/en/account?hsLang=en#account-settings)
- [My Company](https://support.sitemanager.io/en/account?hsLang=en#my-company)
- [Subscription](https://support.sitemanager.io/en/account?hsLang=en#subscription)
- [Project](https://support.sitemanager.io/en/project?hsLang=en#main-content)
- [General](https://support.sitemanager.io/en/project?hsLang=en#general)
- [Upgrade Project](https://support.sitemanager.io/en/project?hsLang=en#upgrade-project)
- [SEO](https://support.sitemanager.io/en/seo?hsLang=en#main-content)
- [Basics](https://support.sitemanager.io/en/seo?hsLang=en#basics)
- [Advanced](https://support.sitemanager.io/en/seo?hsLang=en#advanced)
- [Go Live](https://support.sitemanager.io/en/go-live?hsLang=en#main-content)
- [Prepare Project](https://support.sitemanager.io/en/go-live?hsLang=en#prepare-project)
- [Prepare Domain](https://support.sitemanager.io/en/go-live?hsLang=en#prepare-domain)
- [Going Live](https://support.sitemanager.io/en/go-live?hsLang=en#going-live)
- [Content](https://support.sitemanager.io/en/content?hsLang=en#main-content)
- [Getting Started](https://support.sitemanager.io/en/content?hsLang=en#getting-started)
- [Page Management](https://support.sitemanager.io/en/content?hsLang=en#page-management)
- [Content Management](https://support.sitemanager.io/en/content?hsLang=en#content-management)
- [Language Management](https://support.sitemanager.io/en/content?hsLang=en#language-management)
- [Image Library](https://support.sitemanager.io/en/content?hsLang=en#image-library)
- [Collaborating on a project](https://support.sitemanager.io/en/content?hsLang=en#collaborating-on-a-project)
- [Page Components](https://support.sitemanager.io/en/page-components?hsLang=en#main-content)
- [Getting Started](https://support.sitemanager.io/en/page-components?hsLang=en#getting-started)
- [HTML](https://support.sitemanager.io/en/page-components?hsLang=en#html)
- [CSS](https://support.sitemanager.io/en/page-components?hsLang=en#css)
- [JavaScript (JS)](https://support.sitemanager.io/en/page-components?hsLang=en#javascript-js)
- [How To](https://support.sitemanager.io/en/page-components?hsLang=en#how-to)
- [Layout Components](https://support.sitemanager.io/en/layout-components?hsLang=en)
- [No-Code Editor](https://support.sitemanager.io/en/no-code-editor?hsLang=en#main-content)
- [Getting Started](https://support.sitemanager.io/en/no-code-editor?hsLang=en#getting-started)
- [Properties](https://support.sitemanager.io/en/no-code-editor?hsLang=en#properties)
- [Advanced](https://support.sitemanager.io/en/no-code-editor?hsLang=en#advanced)
- [Design](https://support.sitemanager.io/en/design?hsLang=en#main-content)
- [Overview](https://support.sitemanager.io/en/design?hsLang=en#overview)
- [Design editor](https://support.sitemanager.io/en/design?hsLang=en#design-editor)
- [Design Panels](https://support.sitemanager.io/en/design?hsLang=en#design-panels)
- [Responsive](https://support.sitemanager.io/en/design?hsLang=en#responsive)
- [Fonts](https://support.sitemanager.io/en/design?hsLang=en#fonts)
- [Layouts](https://support.sitemanager.io/en/design?hsLang=en#layouts)
- [Code](https://support.sitemanager.io/en/code?hsLang=en#main-content)
- [Introduction](https://support.sitemanager.io/en/code?hsLang=en#introduction)
- [Token Management](https://support.sitemanager.io/en/code?hsLang=en#token-management)
- [Project Setup](https://support.sitemanager.io/en/code?hsLang=en#project-setup)
- [ToolStack](https://support.sitemanager.io/en/code?hsLang=en#toolstack)
- [Private Library](https://support.sitemanager.io/en/code?hsLang=en#private-library)
- [Dynamic Content](https://support.sitemanager.io/en/dynamic-content?hsLang=en#main-content)
- [Getting started](https://support.sitemanager.io/en/dynamic-content?hsLang=en#getting-started)
- [Basics](https://support.sitemanager.io/en/dynamic-content?hsLang=en#basics)
- [Advanced](https://support.sitemanager.io/en/dynamic-content?hsLang=en#advanced)
- [External Content](https://support.sitemanager.io/en/external-content?hsLang=en#main-content)
- [Getting started](https://support.sitemanager.io/en/external-content?hsLang=en#getting-started)
- [Configuration & implementation](https://support.sitemanager.io/en/external-content?hsLang=en#configuration-implementation)
- [Logic (Smart Content)](https://support.sitemanager.io/en/logic-smart-content?hsLang=en#main-content)
- [Getting Started](https://support.sitemanager.io/en/logic-smart-content?hsLang=en#getting-started)
- [Configuration & Implementation](https://support.sitemanager.io/en/logic-smart-content?hsLang=en#configuration-implementation)
- [Logic](https://support.sitemanager.io/en/logic-smart-content?hsLang=en#logic)
- [AB testing](https://support.sitemanager.io/en/logic-smart-content?hsLang=en#ab-testing)
- [Timed Content](https://support.sitemanager.io/en/logic-smart-content?hsLang=en#timed-content)
- [Password Protected Content](https://support.sitemanager.io/en/logic-smart-content?hsLang=en#password-protected-content)
- [Smart API](https://support.sitemanager.io/en/logic-smart-content?hsLang=en#smart-api)
- [Integrations](https://support.sitemanager.io/en/integrations?hsLang=en#main-content)
- [Getting Started](https://support.sitemanager.io/en/integrations?hsLang=en#getting-started)
- [Integrations](https://support.sitemanager.io/en/integrations?hsLang=en#integrations)
- [Advanced](https://support.sitemanager.io/en/integrations?hsLang=en#advanced)
- [Modules](https://support.sitemanager.io/en/modules?hsLang=en#main-content)
- [Getting Started](https://support.sitemanager.io/en/modules?hsLang=en#getting-started)
- [Modules](https://support.sitemanager.io/en/modules?hsLang=en#modules)
- [Upgrade Center](https://support.sitemanager.io/en/upgrade-center?hsLang=en#main-content)
- [Components](https://support.sitemanager.io/en/upgrade-center?hsLang=en#components)
- [Forms](https://support.sitemanager.io/en/forms?hsLang=en#main-content)
- [General](https://support.sitemanager.io/en/forms?hsLang=en#general)
- [Captcha](https://support.sitemanager.io/en/forms?hsLang=en#captcha)
- [Form Actions](https://support.sitemanager.io/en/forms?hsLang=en#form-actions)
- [Extending Forms](https://support.sitemanager.io/en/forms?hsLang=en#extending-forms)
- [Email Templates](https://support.sitemanager.io/en/forms?hsLang=en#email-templates)
- [Advanced](https://support.sitemanager.io/en/forms?hsLang=en#advanced)
- [Cookie Settings](https://support.sitemanager.io/en/cookie-settings?hsLang=en#main-content)
- [Getting Started](https://support.sitemanager.io/en/cookie-settings?hsLang=en#getting-started)
- [Configuration](https://support.sitemanager.io/en/cookie-settings?hsLang=en#configuration)
- [API Call](https://support.sitemanager.io/en/api-call?hsLang=en#main-content)
- [Configuration](https://support.sitemanager.io/en/api-call?hsLang=en#configuration)
[](https://support.sitemanager.io/en?hsLang=en)
Copyright © 2026, SiteManager Group