The 'Robots Exclusion Protocol' (https://www.robotstxt.org/orig.html) documents a set
of standards for allowing or excluding robot/spider crawling of different areas of
site content. Tools are provided which wrap The rep-cpp https://github.com/seomoz/rep-cpp
C++ library for processing these `robots.txt`` files.