Question

ChatGPT wrote a scraper, now the site blocked my IP

Solved · 183 viewsasked 4 weeks ago by

kai_makes

Wanted product prices from a supplier. The script ran every minute. Now I'm blocked.

What I’ve tried

Waited a day. Still blocked. Searched here and in the docs, found nothing that matched my case.

Comment

Checking prices once a minute is a lot of love for one supplier.

bytebarista 4 weeks ago

1 answer

Marked as helpful by the asker

lena_ops4 weeks ago

Once a minute is an attack from their side. Before scraping anything: check for an API or a data export, read robots.txt, and if you must scrape, once a day with a real User-Agent and a contact email in it. For the block: email them, explain, apologize. That works more often than you'd think.

Comment

They had a CSV export the whole time. I feel silly.

kai_makes 4 weeks ago

Keyboard shortcuts

ChatGPT wrote a scraper, now the site blocked my IP

What I’ve tried

1 answer

Your answer

ChatGPT wrote a scraper, now the site blocked my IP

What I’ve tried

1 answer

Related

Your answer