Creepy crawlies
- You've probably heard me complain about the “AI crawlers” before, but now I actually have some hard numbers I can put up to show their impact.
- In a few words, it's bad enough to create a constant “background radiation” of system load, permanently tying up a chunk of capacity spent on producing output that is only useful for a single purpose — feeding a learning model.
- TL;DR: we spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones.
Unverified
- You've probably heard me complain about the “AI crawlers” before, but now I actually have some hard numbers I can put up to show their impact.
- In a few words, it's bad enough to create a constant “background radiation” of system load, permanently tying up a chunk of capacity spent on producing output that is only useful for a single purpose — feeding a learning model.
- TL;DR: we spend more CPU cycles rendering commits for scrapers than we spend on all other kinds of legitimate access, including git clones.
Sources: Kernel