Reddit isn’t making it easy to get its data anymore.
Blocked: The platform is preventing crawlers from automatically scraping its content by implementing more robust robot.txt files.
The reason? Some crawlers ignore robot.txt files, allowing some companies to lift Reddit posts to feed their AI tools—essentially training them on stolen content.
If you’re using robot.txt files to prevent your own content from getting scraped… well, you might want to take stricter measures to control your traffic.
Because clearly these files are not enough.
