crawl
Crawl starts a depth-first traversal of the Web at the specified URLs. It stores all JPEG images that match the configured constraints. Crawl is fairly fast and allows for graceful termination. After terminating crawl, it is possible to restart it at exactly the same spot where it was terminated. It also keeps a persistent database that allows multiple crawls without revisiting sites.
| Tags | Internet Web Indexing/Search |
|---|---|
| Licenses | BSD Original |
| Operating Systems | POSIX |
Recent releases
- 17 May 2003 19:42
Changes: Crawling is more reliable, and crawl performance has improved for large crawls.
- 28 Jan 2002 22:47
Changes: A complete rewrite for higher performance, including support for all media types, asynchronous DNS lookups, and optional wait time between hosts. The configuration file specifies permittable size of media objects depending on media-type, etc.
- 12 Dec 2001 21:36
Changes: This release has portability fixes, and supports downloading of media types other than just images.
- 03 Jul 2001 16:18
Changes: This release fixes a bug where crawl would stop early when encountering errors on subsequent connections. The verbosity level and number of concurrent connections are tunable now
- 18 Jun 2001 16:09
No changes have been submitted for this release.
