WP Robots Txt
WordPress, by default, includes a simple robots.txt file that’s dynamically generated from within the WP application. This is great, but how do you easily change the content? Enter WP Robots Txt , a plugin that adds an additional field to the “Reading” admin page where you can do just that. No manua
WordPress, by default, includes a simple robots.txt file that’s dynamically generated from within the WP application. This is great, but how do you easily change the content?
Enter WP Robots Txt , a plugin that adds an additional field to the “Reading” admin page where you can do just that. No manual coding or file editing required!
Simply visit https://your-site.com/wp-admin/options-reading.php and you can control the contents of your https://your-site.com/robots.txt
Changelog
A closer look.
- 01Download the plugin
- 02Unzip it
- 03Upload the unzipped folder to wp-content/plugins directory
- 04Activate and enjoy!
Or you can simply install it through the admin area plugin installer.
Frequently asked questions.
I totally screwed up my `robots.txt` file. How can I restore the default version?
Delete all the content from the Robots.txt Content field and save the privacy options.
Could I accidently block all search bots with this?
Yes. Be careful! That said, robots.txt files are suggestions. They don’t really block bots as much as they suggest that bots don’t crawl portions of a site. That’s why the options on the Privacy Settings page say “Ask search engines not to index this site.”
Does discouraging search engines change robots.txt?
No. Since WordPress 5.3 that setting adds a noindex meta tag. It does not add Disallow: / to robots.txt . Your saved rules are still served. Add Disallow: / yourself if you want crawlers to skip the site.
