Whenever I trigger a reindexing through the DatoCMS API (/reindex) I get a success in the build trigger log, however, the number of indexed pages IS ALWAYS 1
any explanation on why this happens?
I believe this is a duplicate of Get only one page after search spidering - #2 by m.finamor
But as the solution hasnāt been published to the community, I am asking again here
thanks.
Hi @mohamed.younes, welcome to the forum and sorry about that! This is the kind of thing weād need to investigate on a case-by-case basis. Could you please email us at support@datocms.com with the site URL youāre trying to index (or just post it here if you donāt mind sharing it with the public)?
Weāll look into it for you after that. Thanks!
OK, now I got informed through Email (thanks for quick support) that the origin of the issue was a subtle redirect.
In fact, I set https://www.my-website.com/en and it was redirected to https://www.my-website.com/en/ and that trailing slash required an 301 and āblockedā the crawler from working
My suggestion is:
Sorry, it looks like I got back to this thread late and Marcelo already helped you there, so Iām not sure how it was set up previously. Do you mean our crawler failed to follow a legitimate 301 that shouldāve taken it to the right place?
ā¦or did you mean that the crawler successfully followed a 301, but that took it somewhere else and it couldnāt find your sitemap after that?
I donāt think āignore 301s to the same domainā is a sensible rule, because sometimes people do use that to redirect within the same site (just a different page).
Maybe Iām misunderstanding what you meant?
well right now the missing trailing slash invokes a 301 but also causes the crawler to stop/fail and ends up only indexing a single page
what I wouldāve expected is that it follows the redirect as keeps indexing as long and the redirected resources is a subroute of the initial route
Gotcha, thanks for clarifying! Iāll report it to the devs.
The devs looked into it and said that we do follow redirects, as long as itās within the same domain/subdomain. They believe that in your case, it was redirecting from my-example.com (no www) to www.my-example.com (with the www), which is a bit different than what you said in post #3 (adding the www would be different than just modifying the trailing slash).
Can you please confirm if that was indeed the case (i.e., whether you also redirected to the www subdomain)?
Technically, my-example.com would be a different host from www.my-example.com in most implementations, including ours, and thatās probably not something we would change, because there are some cross-site security concerns here. However, if you redirect from www.my-example.com/page1 to www.my-example.com/page2, we should be able to follow that.
I hope that clarifies this behavior? If we are mistaken and you werenāt redirecting across hosts, please let us know!