/robots.txt … huh?!

So, you’ve got a great idea! Maybe it’s an app, or perhaps some time series analysis. But… you need data.

It’s likely your data will not exist on Kaggle, or in a clean, updated ‘.CSV’ within easy reach. So, what do you do?

Photo by Glenn Carstens-Peters on Unsplash

You SCRAPE!

Assuming you already know how to set up a scraper (or use a third-party API), AND you decide to follow the website scraping policy, what can you…