Free-Range Spiderbots!

Conference object English OPEN
Boruta, Luc;
(2018)

Free-range what!? The robots exclusion standard, a.k.a. robots.txt, is used to give instructions as to which resources of a website can be scanned and crawled by bots. Invalid or overzealous robots.txt files can lead to a loss of important data, breaking archives... View more
Share - Bookmark

  • Download from
    ZENODO via ZENODO (Conference object, 2018)
  • Cite this publication