Hello community
We are in the process of building a site search. What we would like to control more is the indexing. I have seen that with using a robots.txt file I can control the indexing. In the robots.txt we would like to allow specific pages and subpages and nothing else. But in the example I only saw not allowed pages. Is this possible? Or how could we achieve our goal?
Thanks a lot.
Best
Fabrice