If your browser can open a WordPress page but Yandex cannot find it, do not immediately replace the entire robots.txt or remove every SEO directive. Choose a real public URL and check these four layers in order: robots.txt, HTML code, HTTP headers, and WordPress or SEO plugin settings.
After completing this article, you will have a checklist for a sample URL, know which layer is blocking the bot or requesting that the page not be indexed, and have a way to fix and recheck the issue while preserving a rollback path. SaibABC.Com is included for illustration only; this article does not claim that the website is connected to, verified in, or has any actual status in Yandex.
Limitation: Removing a blocking directive does not guarantee that the URL will be indexed or appear immediately in search results. This article only helps identify and address technical directives within the scope of one URL.
Distinguishing crawling restrictions from requests not to index a page
| Control layer | Indicator | Primary effect |
|---|---|---|
| robots.txt | Disallow | Instructs bots not to crawl a path or area. |
| HTML | <meta name="robots" content="noindex"> | Requests that search engines not include the page in their index. |
| HTTP header | X-Robots-Tag: noindex | Delivers indexing directives in the server response, including for non-HTML files. |
| WordPress or SEO plugin | A setting that prevents indexing or prevents search engines from indexing the site | May generate a meta robots directive or modify robots.txt. |
robots.txt is not a privacy-protection mechanism. Content that must be restricted from viewers should be protected with login requirements, access permissions, or other appropriate access controls—not solely with directives intended for bots. Yandex also notes that a page blocked in robots.txt may still be discovered or appear in some circumstances (according to yandex.com).
Prepare the URL and inspection log
Where to work: your browser and the website’s administration panel. You need a real public post or page, such as [URL bài viết công khai]. Do not turn this placeholder into a SaibABC.Com URL before verifying the official address.
- Open the URL in a private window while logged out of the admin account.
- Record the final address after any redirect, including
http/httpsand whether it haswww. - Record the inspection date, the tools used, and the evidence collected.
- If you plan to make changes, copy the existing value or configuration before modifying it.
URL kiểm tra: [URL bài viết công khai]
Ngày kiểm tra: [YYYY-MM-DD]
robots.txt: [Cho phép / Bị chặn / Chưa xác định]
meta robots: [Không có noindex / Có noindex / Chưa xác định]
X-Robots-Tag: [Không có noindex / Có noindex / Chưa xác định]
WordPress/plugin: [Đã kiểm tra / Chưa kiểm tra]
Kết luận và hành động tiếp theo: [...]Check robots.txt
Open the website’s correct file
Where to work: the browser’s address bar. Replace the sample domain with the official version you have verified:
https://ten-mien-cua-ban.example/robots.txtThe file is usually located in the root directory. If the URL redirects, requires login, returns an error, or displays content that differs from the configuration you know, record the current state instead of making changes immediately.
Look for groups such as:
User-agent: *
Disallow: /Disallow: / within a group that applies to all bots, this indicates that the website is blocking crawling of all paths. However, check the correct User-agent, path, and source generating the file before reaching a conclusion. A plugin, CDN, or server may be generating robots.txt dynamically.
Use Yandex Robots.txt analysis
Requirement: the website has been added to and verified in Yandex Webmaster. Where to work: Yandex Webmaster, in the section with a name equivalent to Tools → Robots.txt analysis; the section name may vary by language or version.
- Select the correct website and match the protocol and prefix
www. - Read the robots.txt content received by the tool.
- Enter the sample URL in the URL testing field.
- Record the result for that URL verbatim.
A “not allowed” result is evidence that the URL matches a blocking directive in the file being analyzed. Save a screenshot of the actual interface if you have permission to use it; do not use an AI-generated image as evidence. Do not replace the entire robots.txt file with a template found online, as this could delete the Sitemap Sitemap line or reopen an area that was intentionally excluded from crawling (according to yandex.com).
Check for noindex in the HTML
Where to work: a browser in a logged-out state.
- Open the sample URL in a private window.
- Right-click and select View page source or View page source.
- Search for
noindex,nofollowandrobots. - Check the tag in the
<head>.
<meta name="robots" content="noindex">
<meta name="robots" content="noindex, nofollow">
<meta name="yandex" content="noindex">nofollow does not mean noindexnoindex. Record the exact value you see. If the tag appears only when you are logged in as an administrator and is absent from the public source code, that is not evidence that an external bot receives the directive.
Check the X-Robots-Tag in the HTTP header
Where to work: the browser’s DevTools. Open the public URL, press F12 or Inspect, select Network, reload the page, select the main document request with the type Document, then open Headers.
Look for X-Robots-Tag. If you see noindex or none, record the complete value and the response URL. Do not select a CSS, JavaScript, or image request by mistake. For PDFs and files without an <head>HTML section, the HTTP header may be where the indexing directive is set.
If you are not familiar with checking headers, ask your hosting, CDN, or server administrator for help. Do not modify the theme to address a directive being added at the server layer.
Check WordPress and SEO plugins
Site-wide settings
Where to work: WordPress → Settings → Reading. Look for the equivalent of Discourage search engines from indexing this site. If the option is enabled on the public website, record that before making changes. WordPress may generate noindex,nofollow when this option is enabled; confirm it in the public HTML rather than relying only on the setting checkbox (according to WordPress.org).
Do not conclude that every URL is open simply because this option is disabled. The theme, SEO plugin, staging plugin, CDN, or custom code may set its own directive.
Post- or page-level settings
Where to work: the sample URL’s editing screen and the settings area of the active SEO plugin. Look for options such as “allow indexing,” “show in search results,” “Robots meta,” or “Advanced robots.” Menu names vary by plugin and version.
Make changes only after identifying the correct URL and understanding the previous value. After saving, refresh or clear the cache according to the website’s procedure, then check the public source code and HTTP header again. If the plugin provides a preview, treat it only as a suggestion; the actual HTML and header are what must be verified.
If you need to review how Yandex displays warnings and choose the area to inspect, see the series article on reading the Yandex Webmaster dashboard.
Diagnose before making changes
| Indicator | Source to check | Safe action |
|---|---|---|
| Robots.txt blocks the URL | Disallow the path or bot group matches. | Save the current file, identify what generates it, and then edit the correct line. |
HTML contains noindex | WordPress, an SEO plugin, the theme, or custom code. | Check the site-wide settings and the individual URL. |
Header contains X-Robots-Tag: noindex | The server, CDN, security layer, or file-type configuration. | Have the administrator check the correct layer. |
| No blocking directive found | The URL may be incorrect, redirected, returning an HTTP error, served from cache, or not yet reflected in Yandex data. | Check the final URL, the server response, and the status in Webmaster. |
| The website is still in a staging environment | A staging setting or noindex option may still be in place. | Compare the environments before removing the directive. |
A page opening in a browser does not prove that a bot receives the same response: a CAPTCHA, login requirement, IP restriction, server error, or different redirect may produce a different result.
Fix, verify, and roll back
- Record the current state: save robots.txt, the meta robots directive, the header, the URL, the time, and the person who performed the check.
- Edit the correct source: WordPress Reading settings, the SEO plugin, the file that generates robots.txt, or the corresponding server configuration.
- Change only one layer at a time: if you edit several locations at once, you will not know which change affected the result.
- Refresh the cache according to your procedure: do not bulk-delete security plugins or CDN settings just because one check failed.
- Check from outside: reopen robots.txt, run the URL again in Robots.txt analysis, view the private source code, and check the header of the main document.
- Record the status: “fixed, but not yet reevaluated by Yandex” if you have only removed the directive. Do not write “indexed” merely because the block has been removed.
Rollback: restore the exact robots meta value, file, or setting you saved earlier. Do not bulk-delete DNS records, security plugins, or server configuration to address an unconfirmed warning.
If you encounter access errors, redirects, or different responses between the browser and the bot, see the procedure for finding the exact point of failure when verifying Yandex instead of disabling all security controls.
Exercise: check one public URL
Choose a real post on a website you manage. If you do not yet have a verified URL, leave [URL bài viết công khai] unchanged in the log and stop at the preparation step.
- Record the final URL after opening it in a private window.
- Open robots.txt and run the URL through Robots.txt analysis if the website is ready in Yandex Webmaster.
- Find
noindexandnonein the HTML source. - Check
X-Robots-Tagin the main HTML response. - Open Settings → Reading and the robots meta settings of the active SEO plugin.
- Record each layer separately as “allowed,” “blocked,” “has noindex,” “no evidence,” or “not checked.”
The exercise is complete when you can answer whether the bot is allowed to crawl the URL, whether the HTML contains noindex, whether the header contains a blocking directive, and which setting produced that result. If evidence is missing, record “undetermined” instead of guessing.
Misconceptions to avoid
- The page opens for me, so the bot can access it too: not necessarily, because the bot may receive an error, CAPTCHA, login requirement, or different redirect.
- robots.txt and noindex are the same thing: they are not; one concerns crawling, while the other primarily concerns indexing.
- Removing noindex makes the page appear immediately: there is no guarantee; Yandex still has to access, process, and evaluate the URL.
- An empty report means the URL is blocked: it may simply mean that data is unavailable, the wrong website or URL was selected, or the filter is inappropriate.
- robots.txt protects private data: is not secure; private data requires access controls.
If robots.txt does not block the URL but the URL still has an issue, check the server response and the URL status in the relevant Yandex tool instead of repeatedly editing the file (according to yandex.com).
Next step
The result of this exercise is a checklist for one URL, not evidence that the URL has been indexed. Keep the URL, inspection date, status of each layer, supporting evidence, and previous configuration. The next step is to use this checklist to review the URL’s specific status in Yandex Webmaster:
How to Check Whether Yandex Has Indexed an ArticleReference source
- Using robots.txt — Yandex Webmaster.
- Robots.txt analysis — Yandex Webmaster.
- How do I prohibit indexing of a site or pages? — Yandex Webmaster.
- How do I check if a site is accessible to users and indexing robots? — Yandex Webmaster.
- Settings Reading screen — WordPress.org Documentation.
- wp_robots_noindex() — WordPress Developer Resources.
