Make a cache of robots.txt #16

Open

opened

on Aug 28, 2024

Currently the crawler pings robots.txt everytime, even for repeated URLs. For every hostname, make a cache of allowed and disallowed paths to make the crawling faster. Speed increase: 25-30%

to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Assignees

No one assigned

Labels

No labels

No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests