- How to create and configure robots.txt for WordPress
- How to check robots.txt for errors
- Why is it important to configure robots.txt correctly
robots.txt is a text file encoded in UTF-8 that tells search engines which pages of the site should be crawled and which should not. It is located in the root directory of the resource. How to view the robots.txt of a site? For example, on the Cityhost site, it can be found at https://cityhost.ua/robots.txt.
Also read: Where is robots.txt located in WordPress?
A correctly configured robots.txt will inform Google web crawlers and other search engines about what should not be crawled:
- pages with personal information of registered users (for example, customer accounts in online stores);
- pages with internal search results on the site;
- pages for logging into the site management panel;
- pages that cause content duplication.
And if you are just starting to delve into the basics of web mastering, you probably already understand that theoretical knowledge in this field should ideally be immediately reinforced by practice. To do this, order affordable CityHost hosting with the latest version of PHP, MySQL databases, and SSH protocol access, create a blog or online store on your favorite engine, and hone your website administration skills on a real example.
Also read: How to install WordPress on hosting
How to create and configure robots.txt for WordPress
To create a correct robots.txt, you will need:
- a text editor (for example, Notepad++, Atom, or even the standard Windows Notepad);
- an FTP client (for example, FileZilla, WinSCP, or Cyberduck);
- 10–15 minutes of free time.
Note. If you are not familiar with FTP clients and your site is hosted on CityHost, you can add the configured WordPress robots.txt through the file manager in the hosting control panel.
First, open a text editor, create a new file, and save it with the name robots and the .txt extension. It is important that all letters are written in lowercase. Variants Robots.txt, robots.TXT, or ROBOTS.txt are incorrect.
CONGRATULATIONS! You have found a gift from Cityhost — 50% discount on the renewal of your hosting, domain, or VPS for 1 year.
To receive the gift, take a screenshot of this image and send it as a comment to this post: https://t.me/cityhostua/489
If your comment with this image is the first — you have caught luck, as the gift can only go to one person. So don’t hesitate!
This bonus is available within the \"I am going to search\" promotion and is valid from April 30 to May 31, 2026, inclusive.
Next, add the following code to the file and replace the link in the last line with the URL of your site's XML sitemap:
User-agent: *
Disallow: /cgi-bin
Disallow: /?
Disallow: /wp-
Disallow: /wp/
Disallow: *?s=
Disallow: *&s=
Disallow: /search/
Disallow: /author/
Disallow: /users/
Disallow: */trackback
Disallow: */feed
Disallow: */rss
Disallow: */embed
Disallow: */wlwmanifest.xml
Disallow: /xmlrpc.php
Disallow: *utm*=
Disallow: *openstat=
Allow: */uploads
Sitemap: https://example.com/sitemap.xml
We recommend thoroughly understanding the syntax of this robots.txt example for WordPress:
- User-agent — specifies which web crawlers the rules below apply to. The value * indicates that everything written in robots.txt should be considered by all search engines.
- Disallow — tells web crawlers which directories or files should not be crawled during the crawl. For WordPress sites, it is recommended to block crawling of author archive pages, search results, access to the engine's Workshop, as well as the RSS feed, etc. This will protect the site from content duplication and prevent pages that do not belong in search results from appearing.
- Allow — tells search engines which directories or files should be crawled during the crawl. In our example, the uploads directory is open, which contains images stored on this site.
- Sitemap — indicates the sitemap. If there are two or more XML sitemaps, write each of them in robots.txt on a separate line starting with Sitemap:.
The last step is to upload the robots.txt to the root directory of the site using an FTP client or the file manager in the hosting control panel. To check if the upload was successful, go to http://example.com/robots.txt, replacing example.com with your site's domain. If everything is done correctly, you will see a page displaying the code presented above.
By the way, what will happen if you apply disallow all in robots.txt?
If you need to prevent all search engines from crawling the site, use the following content in the robots.txt file:
User-agent: *
Disallow: /
Such a configuration can negatively impact SEO, as search engines will not be able to index the pages. Use it cautiously, for example, for sites in the development stage.
Also read: What is WHOIS, what is it used for, and how to check a domain
How to check robots.txt for errors
You learned how to configure robots.txt for WordPress, and the next step is to check its correctness. To do this, follow these steps:
- Open the \"robots.txt file checking tool\" found in the old version of Google Search Console (under the \"Crawl\" menu).
- Copy the contents of the robots.txt file and paste it into the editing window.
- Make sure that the number of errors and warnings in the bottom left corner of the editing window is zero. If there are syntax issues, this line will indicate the number of errors or recommendations, and a red or orange icon will appear to the left of the line in the editor with incorrect information. Hovering over it will show you a description of the error.
If you see the line \"Errors: 0, Warnings: 0\" under the editing window of the \"robots.txt file checking tool,\" notify the search engine about the changes in robots.txt. To do this, click the \"Submit\" button in the lower right corner of the editor. Confirm the submission of the request to update the file by clicking the \"Submit\" button next to option № 3 in the pop-up window.
Also read: What are keywords and how to choose them.
Why is it important to configure robots.txt correctly
In January 2017, Gary Illyes, an analyst in the quality department of the \"good corporation,\" published an article in the official Google Webmaster blog titled \"What Crawling Budget Means for Googlebot.\" In it, he noted that if a web crawler encounters low-quality pages or pages that duplicate the content of another page while crawling the site, the speed and frequency of crawling will decrease. The negative consequence of this is that after adding new content to your site, it will appear in search results slowly.
A correctly configured robots.txt for WordPress prevents search engines from crawling duplicates and pages that do not provide any value to visitors. And, considering this, it is no less important element of technical optimization of the site than, for example, a properly composed and automatically updated sitemap or enabled gzip compression.
Another characteristic of the site that positively affects both search optimization and visitor satisfaction is high page load speed. Renting a dedicated server can help with this, or a cheaper option — a virtual server. There will be enough resources to make an online store, online portal, or blog fast, reliable, and secure.
Was this publication informative? Then share it on social media and join our Telegram channel. We remind you that the hosting company CityHost provides affordable hosting services for websites of any complexity. For technical questions, contact us via online chat or by phone ☎️ 0 800 219 220.
