sitemap 404 for google bot

Ryan S. shared this question ago

Answered

Our Rank Math sitemaps always load in a normal browser, but Google Search Console received HTTP 404 HTML responses for /sitemap_index.xml, /page-sitemap.xml, and /post-sitemap.xml. External testing also produced intermittent 404 responses with a Googlebot user-agent while browser requests returned 200 XML. Rank Math’s internal /?sitemap=1 endpoint consistently returns valid XML. Please inspect the Apache access/error logs, .htaccess sitemap rewrite rules, ModSecurity/WAF rules, rate limiting, and bot protection. Please ensure verified Google crawler IPs can access all sitemap XML files consistently.

The same question

Comments (2)

Gabrielle P. ● 20 days ago

We have our developers looking into this further and will advise when we have an update.

Thanks!

Reply URL

Corey H. ● 20 days ago

Please try re-adding the sitemap. I've had it take a few tries before it'll return 200 instead of 404. I unfortunately do not have a different solution to this. This is an ongoing issue with Google Bot there seems to be no official fix for. This has come up before and re-adding the sitemap a few times in search console usually does the trick. Please let me know if you still have issues.

Reply URL

Ryan S. ● 20 days ago

I've reloaded it a couple times.

03a410a49b6c2b3dca3eac18123a0d20

URL

Corey H. ● 20 days ago

can you use the Live URL tester inside search console and see what that says for the URL? The ongoing issue is usually the Live URL tester returns no errors but then you go to add it to the sitemap section and it all of a sudden has an error.... If the Live URL tester is failing as well I can take a look at the server again.

URL

Ryan S. ● 20 days ago

2e8e2fb38c029ba15ceec57efc652b78

URL

Corey H. ● 19 days ago

Okay this is the scenario I was describing. The URL tester is returning that the page is able to be crawled and fetched, either or both of these would fail if the server wasn't delivering the page properly. However Google contradicts itself when you go to add it to the sitemap and all of a sudden there is an error on that page.

The noindex is normal on sitemap URL's because you wouldn't want a user getting your sitemap URL as a search result thats not helpful. its content gets indexed but not the sitemap itself.

URL

Ryan S. ● 19 days ago

How do I fix it? I've uploaded it multiple times. I guess I should just wait a few days to see if it gets resolved after each reupload?

URL

Corey H. ● 19 days ago

I've run into this about 50 times i'd guess. I usually just try a few days in a row till it finally sticks.... I wish i had a better answer for you. You can also try and use the actual links on the sitemap_index page instead of the _index page itself and sometimes that has a little more luck. Try adding https://www.jumpers4you.com/page-sitemap.xml and see if that has any better luck.

I've had 1 website that I simply could not get to add after months of trying (I think it had something to do with them being massive 5000+ pages). Google will eventually find all the pages on your site naturally without a sitemap it just takes longer, so if that ends up happening just be patient and the problem will solve itself slowly.

URL