Hello,
On our site, we have both sitemap and robots.txt available.
I wanted to prevent some page from being crawled, but adding their addresses under DatoCmsSearchBot in robots does not seem to work at all - they are still being crawled as I can see in the Dato logs.
Hey @kamil.socha , can you email us at support@datocms.com with your robots.txt url, the page url that is being wrongly indexed, and your DatoCMS project URL? Thank you!
Hello @kaylijenner234 welcome to the community!
Can you detail your questions so we can know what you are looking for?