Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.
https://github.com/internetarchive/heritrix3.git
