Web Harvesting Services - Library of Congress

The Library of Congress is seeking a contractor to provide web harvesting services. The contractor will be responsible for harvesting content from the web, based on instructions from Library staff, and providing temporary access to the content and required crawl reports for quality review. The contractor will also be required to transfer the content to the Library for preservation and public access. The Library uses an in-house curator tool, called Digiboard, to manage seed URL nomination, permissions gathering, and the scoping and preparation of a seed list to send to the contractor. The Library's tool creates plain text seed lists with scoping instructions in SURT format. The contractor will be required to provide various reports, including a comprehensive list of approved scopes in SURT format for each crawl, a custom Host data report, and a count of WARCs written for the crawl. • The contractor must be able to harvest content from the web at scale. • The contractor must be able to provide temporary access to the content and required crawl reports for quality review. • The contractor must be able to transfer the content to the Library for preservation and public access. • The contractor must be able to provide various reports, including a comprehensive list of approved scopes in SURT format for each crawl, a custom Host data report, and a count of WARCs written for the crawl.