Recommended Free Tools
iTechGuides is reader-supported. When you buy through links on our site, we may earn an affiliate commission. As an Amazon Associate I earn from qualifying purchases. Learn more
Antihost offers managed WordPress hosting with free migration, one-click staging and a 14-day free trial.
To edit WordPress robots.txt safely, use the robots.txt editor in your SEO plugin or upload a UTF-8 plain-text file to the site root after saving a backup. A sensible starting configuration blocks the private /wp-admin/ area, keeps /wp-admin/admin-ajax.php crawlable, and declares your XML sitemap. Use a noindex meta tag or an X-Robots-Tag header—not robots.txt—to keep a page out of search results.
What robots.txt does in WordPress
robots.txt is a plain-text set of instructions for automated crawlers. On a WordPress site it normally lives at the host root, so a site at www.example.com exposes it at www.example.com/robots.txt. Crawlers request that file before fetching URLs and use matching user-agent, allow, and disallow rules to decide what they may request.
What’s actually slowing this PC down?
Pick the symptom - the matching free tool is one click away.
The file controls crawling, not indexing. A URL that is disallowed can still appear in search if Google discovers it through links or other signals. Conversely, Google cannot fetch a blocked page to see a noindex directive. If your goal is removal from search, leave the URL crawlable and apply noindex in its robots meta tag or send an X-Robots-Tag HTTP header.
Rules Google understands
User-agent: identifies the crawler to which the following rules apply.AllowandDisallow: permit or restrict matching paths.Sitemap: gives crawlers the absolute URL of an XML sitemap.
Google does not support crawl-delay. Rule paths are case-sensitive, while field names are not. When rules conflict, Google uses the least restrictive matching rule. Save the file as UTF-8 plain text; a word-processing document can add formatting or encoding that makes directives unreliable.
#1 Best Overall
- Read Before You Buy — No Video Output: These adapters support charging and USB 2.0 data transfer, but cannot transmit video signals. Except for standard USB webcams (which use USB data only), they are not compatible with HDMI/DisplayPort cables, video-capable USB-C hubs, or docking stations with video output.
- Convert USB-A Ports to USB-C: Designed to connect USB-C earphones, cables, flash drives, card readers, and other USB-C accessories to standard USB-A ports. Plug-and-play with no drivers or software required.
- Aluminum Alloy Housing: Built with a sturdy aluminum alloy shell that aids in heat dissipation and protects against daily wear and scratches. Designed to maintain a stable and secure connection.
- Compact & Travel-Friendly: The ultra-compact design allows the adapter to stay plugged into your device without blocking adjacent ports or adding bulk, reducing wear and tear on your original USB ports.
- 12-Month Warranty: Backed by a 12-month manufacturer warranty for peace of mind. Designed to meet strict quality control standards for reliable everyday performance.
Decide what you actually want to block
Start with a URL inventory rather than copying a large list of disallows. WordPress installations differ, and a path that is unnecessary on one site may contain useful content on another.
- Keep public posts, pages, category archives, and other pages that should be discovered crawlable.
- Restrict administrative paths that do not provide search value, beginning with
/wp-admin/. - Keep
/wp-admin/admin-ajax.phpallowed because front-end features can use it for asynchronous requests. - Decide separately which URLs should be deindexed. Those need
noindexor anX-Robots-Tag, not a robots.txt block. - Find the XML sitemap URL generated by your site and plan to add it as a
Sitemapdirective.
Do not assume that every URL containing “admin,” a query string, or a plugin name is safe to block. Test a proposed rule against real URLs from your analytics, internal links, XML sitemap, and Search Console reports before deploying it.
Back up the current file before editing
- Open the live file at your domain’s
/robots.txtpath and copy all of its contents. - Save that copy as a plain-text backup with the date in the filename. If the new rules cause a problem, you can restore the previous text instead of reconstructing it.
- Record the current sitemap URL and any custom rules already used for a staging environment, a multilingual setup, or a special crawler.
- Make one logical change at a time. A small, identifiable change is easier to test and roll back than a wholesale rewrite.
Choose a WordPress editing method
For most site owners, an SEO plugin editor is the least technical route. Manual editing is appropriate when you control hosting files, need a configuration independent of a plugin, or cannot access a suitable editor.
| Method | Where you edit | Advantages | Risks and checks |
|---|---|---|---|
| Rank Math | Rank Math → General Settings → Edit robots.txt | Convenient in-dashboard editing when the plugin is already installed. | Menu labels can change between plugin versions; confirm the live root file after saving. |
| All in One SEO | All in One SEO → Tools → Robots.txt tab → enable Enable Custom Robots.txt | Provides a dedicated custom-file switch and editor. | Turning the custom option on changes which content is served; keep a backup and verify the result. |
| Yoast | Use the robots.txt box, which pre-fills default settings, then add or adjust lines. | Lets you edit the generated text without leaving WordPress. | Defaults and labels are version-dependent; inspect the resulting file rather than assuming the editor’s preview is live. |
| Dedicated robots.txt plugin | An editor supplied by a plugin such as Virtual Robots.txt, WordPress Robots.txt Optimization, or Robots.txt Editor. | Useful if your existing SEO plugin has no editor. | Another plugin can overwrite or replace the file. Check ownership, backup behavior, and the public URL after every update. |
| Manual upload | FTP or your host’s file manager, in the site root. | Independent of plugin settings and easy to version-control locally. | An incorrect directory, encoding, or upload can leave the old file in place or break access. Confirm the live response. |
The menu paths above reflect labels observed on October 14, 2026. Plugin releases can rename or move them, so look for “robots.txt,” “robots,” or “tools” if your screen differs.
Rank #2
- 5-in-1 USB-C Hub: Experience comprehensive connectivity featuring a Power Delivery input, two USB-A 2.0 ports, a USB-A 3.0 port, and an HDMI port. (Note: The USB-C power delivery input port is only for connecting an external wall charger to power your laptop and cannot power peripheral devices.)
- 90W Pass-Through Charging: Achieve optimal charging with 90W pass-through power to your laptop, supported by a total input of 100W, with the hub reserving 10W for operational efficiency. (Note: Wall charger not included.)
- Quick Data Transfers: Accelerate your productivity with rapid data transfers using a high-speed 5Gbps USB 3.0 port and two 480Mbps USB 2.0 ports.
- 4K HDMI Display: Enhance your visual experience with a hub capable of delivering 4K resolution at 30Hz in both mirror and extend modes. Please note that this hub is compatible with MacBook (macOS 12 and newer), Windows 10 and 11, ChromeOS, and laptops equipped with DP Alt Mode and Power Delivery. Note: This device is not compatible with Linux.
- What You Get: Anker USB-C Hub (5-in-1, 4K HDMI), welcome guide, 18-month warranty, and our friendly customer service.
Edit robots.txt with an SEO plugin
Rank Math
- In the WordPress dashboard, open Rank Math → General Settings → Edit robots.txt.
- Review the existing text rather than deleting it automatically.
- Add only the rules required for your site, including the administrative exception and sitemap directive described below.
- Save the file, then open your public
/robots.txtURL in a private browser window and confirm that the saved text is being served.
All in One SEO
- Open All in One SEO → Tools → Robots.txt.
- Enable Enable Custom Robots.txt if the editor requires it.
- Paste or edit the backed-up rules, check spelling and capitalization of paths, and save.
- Request the public
/robots.txtURL directly to make sure the custom version—not a previous generated version—is returned.
Yoast
- Open the robots.txt editing area in Yoast. The box may already contain default settings.
- Preserve useful defaults and append only the paths you have decided to restrict.
- Save, then inspect the live root file and test the important URLs in Search Console.
If a plugin editor is unavailable, a dedicated robots.txt plugin can provide the same function. Treat it as a configuration owner: document which plugin controls the file so a later plugin change does not silently replace your rules.
Edit robots.txt manually with FTP or a host file manager
- Connect to the hosting account with FTP or open the host’s file manager.
- Navigate to the document root—the directory that contains the WordPress installation and serves your domain.
- Download the existing
robots.txtbefore changing it. If no file exists, save an empty backup note so you know the starting state. - Open the file in a plain-text editor such as Notepad or TextEdit. Do not use a word processor; it can insert rich-text formatting, smart quotes, or an incompatible encoding.
- Enter the tested directives, save as UTF-8 plain text, and upload the file to the same root directory. Do not place it inside
/wp-admin/,/wp-content/, or another subdirectory. - Visit the domain’s
/robots.txtaddress and compare the response with your local copy. If the old text appears, you uploaded to the wrong directory or another system is generating the file.
A practical WordPress starting configuration
The following is a starting point, not a universal policy. Remove a line if that path contains content you want crawled, and add your site’s actual sitemap URL in place of the code variable.
Quick wins for a faster PC:
Repair Windows errors before they cause bigger problemsFix Now →Scan for outdated or missing drivers - takes under a minuteDriver Scan →Clear out junk files and repair common Windows errorsFree Scan →User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /wp-content/plugins/
Disallow: /trackback/
Disallow: /readme.html
Sitemap: {{YOUR_SITEMAP_URL}}Why each line is there
User-agent: *applies the group to all crawlers that honor these directives.Disallow: /wp-admin/limits crawling of the administrative area.Allow: /wp-admin/admin-ajax.phpcreates an explicit exception for the asynchronous endpoint used by front-end functionality.Disallow: /wp-content/plugins/,/trackback/, and/readme.htmlare commonly suggested low-value paths, but their suitability depends on your installation. A plugin’s public assets or documentation may be intentionally exposed, so verify before keeping each line.Sitemap:should contain the exact absolute XML sitemap URL generated for your site. The directive helps discovery; it does not make a blocked URL indexable or unindexable.
Keep the file readable. Put one directive per line, use the exact path casing used by the site, and avoid broad disallows until you have checked representative URLs. A single prefix can match far more URLs than its short appearance suggests.
Robots.txt versus noindex: use the right control
These controls solve different problems:
| Goal | Use | Important consequence |
|---|---|---|
| Reduce crawler requests to a path | Disallow in robots.txt |
The crawler may not fetch the URL, so it may not see page-level directives. |
| Keep a crawlable page out of search results | noindex robots meta tag or X-Robots-Tag header |
Leave the page accessible to crawlers so they can read the directive. |
| Tell crawlers where structured URL lists are | Sitemap directive |
The sitemap is a discovery hint, not a permission override. |
A common error is to add Disallow to a URL that is already meant to be deindexed. That prevents Google from seeing the noindex signal and can leave the URL indexed. Remove the block, allow crawling, and apply the appropriate noindex control instead.
Rank #3
- Sleek 7-in-1 USB-C Hub: Features an HDMI port, two USB-A 3.0 ports, and a USB-C data port, each providing 5Gbps transfer speeds. It also includes a USB-C PD input port for charging up to 100W and dual SD and TF card slots, all in a compact design.
- Flawless 4K@60Hz Video with HDMI: Delivers exceptional clarity and smoothness with its 4K@60Hz HDMI port, making it ideal for high-definition presentations and entertainment. (Note: Only the HDMI port supports video projection; the USB-C port is for data transfer only.)
- Double Up on Efficiency: The two USB-A 3.0 ports and a USB-C port support a fast 5Gbps data rate, significantly boosting your transfer speeds and improving productivity.
- Fast and Reliable 85W Charging: Offers high-capacity, speedy charging for laptops up to 85W, so you spend less time tethered to an outlet and more time being productive.
- What You Get: Anker USB-C Hub (7-in-1), welcome guide, 18-month warranty, and our friendly customer service.
Understand matching and conflicts before adding exceptions
Path case matters
/Private/ and /private/ are different path strings for matching purposes. Copy the exact capitalization from the URL you intend to control.
Conflicting rules are resolved least restrictively by Google
If an allow and a disallow both match a URL, Google selects the least restrictive applicable rule. Make exceptions explicit and keep overlapping patterns to a minimum so that the result is predictable.
Crashes, No Sound, or Screen Glitches?
Random freezes, missing sound and display glitches usually trace back to one bad driver. Find and replace yours safely.Free scan · under a minuteWindows Errors? Fix Them Before They Spread
Repair common Windows errors and clear accumulated junk for a smoother, more stable PC - no reinstall needed.Free scan · no reinstallUnsupported directives do not create extra control
Google does not support crawl-delay. Adding it does not regulate Google’s crawl rate. Keep the file to supported fields and solve indexing decisions with the controls designed for indexing.
Test the live file before and after deployment
- Open the public
/robots.txtURL and check that it returns the text you intended, in the root of the correct host. - In Google Search Console, open the robots.txt report and use its testing interface to check representative URLs. Include a normal post, a page you deliberately restrict,
/wp-admin/, and/wp-admin/admin-ajax.php. - The tester lets you edit a proposed version and retest it without changing the live file. Those edits are only a simulation; copy the final text into the real robots.txt editor or upload and save it.
- Re-run the checks after the live save. Confirm that the sitemap URL remains present and that an allowed URL is not accidentally covered by a broader disallow.
- Keep the dated backup and note the change. If crawling or a front-end feature fails, restore the backup while you investigate.
Testing a rule in Search Console is not deployment. The live file changes only when you save it through the plugin or upload it to the root, and the live response is the authority for what crawlers receive.
Rank #4
- Dual Converters, Infinite Potential：Includes 2× USB C male to USB A female adapters and 2× USB A male to USB C female adapters. Perfect for a wide range of uses—tablets with Bluetooth keyboards, expand USB ports on macbook, and more. Two different converters for all your daily needs
- Next-Level 10Gbps & 3A Charging: No more slow 480Mbps, this usb to usb c adapter has a transfer speed of up to 10Gbps, allowing you to do more transferring in less time. This usb adapter fits both USB A and USB C charger, supporting up to 3A fast charging
- Upgraded Exquisite Craftsmanship: With an aluminum alloy housing and metal connector, the usbc to usb adapter is extremely durable and sturdy. Rigorously tested to withstand more than 10,000 times of plugging and unplugging, ensuring long-lasting performance
- Broad Compatible: The usb c to usb adapter widely supports all USB C/ USB A devices like laptops, tablets, cellphones, car chargers, and phone chargers. Such as compatible with MacBook Pro/Air 2023/2022, Thunderbolt 4/3 Devices,Apple MagSafe Watch 9/8/7/SE/Ultra, iPad Pro 2022/2021, Samsung Galaxy S23/S20/S10, and iPhone 17/16/15 Pro. Plug and play
- Please Note: To reach 10Gbps speed, keep the cable under 3.3 ft. For USB A Male to USB C adapters, try flipping the USB C connector. USB C Male to USB A adapters support bidirectional 10Gbps transfer within 3.3 ft
Troubleshoot common WordPress robots.txt problems
The browser shows an old file
Check that you edited the correct host and uploaded to the document root. A staging domain and production domain each have their own root file. If the root response still differs from your saved text, identify which plugin or hosting layer is generating robots.txt and edit that owner instead of repeatedly uploading another copy.
A page remains in Google after you disallowed it
That is expected behavior for a crawl block: robots.txt does not guarantee deindexing. Remove the disallow if Google needs to read the page, then apply a page-level noindex or an X-Robots-Tag header. The URL may take time to be recrawled and reevaluated.
A page intended for noindex still appears
Check whether robots.txt blocks it. If so, Google cannot fetch the page to see its noindex directive. Allow crawling, verify that the directive is actually present in the page response, and test the URL again.
An AJAX-powered feature stops working
Inspect whether a broad /wp-admin/ rule also matches /wp-admin/admin-ajax.php. Retain the explicit allow line, then test the feature and the endpoint in the robots.txt report.
Best Value
- 5-in-1 Connectivity: Equipped with a 4K HDMI port, a 5 Gbps USB-C data port, two 5 Gbps USB-A ports, and a USB C 100W PD-IN port. Note: The USB C 100W PD-IN port supports only charging and does not support data transfer devices such as headphones or speakers.
- Powerful Pass-Through Charging: Supports up to 85W pass-through charging so you can power up your laptop while you use the hub. Note: Pass-through charging requires a charger (not included). Note: To achieve full power for iPad, we recommend using a 45W wall charger.
- Transfer Files in Seconds: Move files to and from your laptop at speeds of up to 5 Gbps via the USB-C and USB-A data ports. Note: The USB C 5Gbps Data port does not support video output.
- HD Display: Connect to the HDMI port to stream or mirror content to an external monitor in resolutions of up to 4K@30Hz. Note: The USB-C ports do not support video output.
- What You Get: Anker 332 USB-C Hub (5-in-1), welcome guide, our worry-free 18-month warranty, and friendly customer service.
The sitemap is not being discovered
Confirm that the Sitemap: line contains the exact absolute URL of the XML sitemap, including the correct host and path. A sitemap directive cannot compensate for a typo, a different host, or a sitemap URL that your site does not serve.
Your changes disappear after an update
Determine whether the SEO plugin, a dedicated robots.txt plugin, or the host generates the file. Keep the configuration in one owner, retain your backup, and recheck the live root file after plugin or theme changes.
The Tool Desk
Outbyte PC Repair FREERepair Windows errors before they cause bigger problemsFix Now →Outbyte Driver Updater FREEScan for outdated or missing drivers - takes under a minuteDriver Scan →The file contains strange characters or formatting
Re-edit it in a plain-text editor and save as UTF-8. Replace smart punctuation with ordinary ASCII directive characters, upload again, and inspect the raw response at /robots.txt.
Maintenance checklist
- Revisit the file when your URL structure, plugins, sitemap, or hosting arrangement changes.
- Keep
/wp-admin/admin-ajax.phpexplicitly allowed if your front end relies on it. - Review every new
Disallowagainst a real URL list; do not block an entire directory merely because some files in it are low value. - Use noindex controls for deindexing and leave those URLs crawlable.
- Keep the sitemap directive accurate and absolute.
- Test the live file in Search Console after each material change, then save the dated backup.
The safest robots.txt is short, intentional, and tested. Start with the administrative restriction and sitemap, add only verified low-value paths, and use noindex when the objective is search exclusion rather than crawl reduction.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
