Should You Allow or Disallow /wp-admin/admin-ajax.php in WordPress?

When it comes to managing a WordPress website, there are many files and directories that webmasters may want to restrict access to, for security or other reasons. 

One such file is /wp-admin/admin-ajax.php, which handles AJAX requests from the front end of a WordPress site. 

The question that arises is whether webmasters should allow or disallow access to /wp-admin/admin-ajax.php in robots.txt. In this article, we'll explore the advantages and disadvantages of each approach, and provide best practices for securing this file.

What is Robots.txt and What is its Purpose?

Robots.txt is a text file that resides in the root directory of a website. This file instructs search engine crawlers which pages and files to crawl and which ones to ignore. 

The file is used to manage how search engines access and index content on a website. Also, the robot file can prevent access to certain areas of a website.

The Impact Allowing or Disallowing /wp-admin/admin-ajax.php in Robots.txt

Search engine robots typically access and index pre-rendered HTML content, which is generated by the WordPress platform and served to users when they visit the site.

The robots do not typically access or index dynamic content that is generated through AJAX requests, including content served through /wp-admin/admin-ajax.php.

Therefore, in most cases, allowing or disallowing access to /wp-admin/admin-ajax.php in robots.txt is unlikely to have a significant impact on the indexing and ranking of your website in search results.

How to Disallow /wp-admin/admin-ajax.php from robots

Several approaches can be used to allow or disallow access to /wp-admin/admin-ajax.php in robots.txt, depending on a website's specific needs. 

One option is to use a custom function to allow or disallow the indexing of the file. Here's an example of how to use them to disallow access of the robots to /wp-admin/admin-ajax.php. You have to go to /wp-content/themes/functions.php and add the following lines of code.

function disallow_ajax_noindex() {
   global $wp_query;
   if( !empty($wp_query->query_vars['action']) && $wp_query->query_vars['action'] == 'my_ajax_action' ) {
       header("X-Robots-Tag: noindex, nofollow", true);
   }
}
add_action('init', 'disallow_ajax_noindex');

Another option is to use the robots.txt file itself to allow or disallow access. Here's an example of how to disallow access to /wp-admin/admin-ajax.php in robots.txt:

User-agent: *

Disallow: /wp-admin/admin-ajax.php

Best Practices for Securing /wp-admin/admin-ajax.php

While allowing access to /wp-admin/admin-ajax.php in robots.txt doesn't have any real effect on SEO or indexing. Anyway, you should protect this file against CSRF attacks.

One way is using Nonce method:

Nonces are random values that are added to AJAX requests. These values are validated on the server side to ensure that the request is legitimate.

Another measure is to implement IP address restrictions to ensure that only authorized users can access /wp-admin/admin-ajax.php.

Conclusion

It is clear that allowing or disallowing access to /wp-admin/admin-ajax.php in robots.txt is unlikely to have a significant impact on the indexing and ranking of your website in search results. This is because Google's robots typically access and index pre-rendered HTML content, not dynamic content generated through AJAX requests.

That being said, securing /wp-admin/admin-ajax.php is important, regardless of whether you choose to allow or disallow access to it in robots.txt. Nonces and IP address restrictions can be used to prevent unauthorized access to the file.

Ultimately, the decision of whether to allow or disallow access to /wp-admin/admin-ajax.php in robots.txt will depend on the specific needs and goals of your website, as well as the security considerations. It is important to carefully consider these factors and make an informed decision based on the unique requirements of your website.

You also might be interested in reading:

Guide to WordPress Security