Alright, let’s talk about this ‘deal generator’ thing I put together. It wasn’t some grand plan, really. I just got tired of missing out on good online sales or spending ages hunting for them myself.
So, first thing, I sat down and thought, okay, what do I actually want this thing to do? I needed something to automatically check a few websites I regularly buy from. You know, the big ones, maybe a couple of specific stores for electronics or clothes.
I started simple. Picked one website to begin with. I needed to figure out how to get the product information – specifically the price – off their pages. Spent a good chunk of time just looking at the website’s code using my browser’s developer tools. You right-click, hit ‘inspect’, and just poke around. Had to find where the price was actually stored in all that mess of HTML.
Once I had an idea of where the data lived, I started writing a small script. Just a basic thing to fetch the webpage content. This part was fairly straightforward, lots of examples online for grabbing a webpage.
Then came the tricky part: pulling out just the price and the product name. Websites don’t make this easy. Their structures change. Sometimes they have multiple prices (like list price vs. sale price). It took a lot of trial and error. I’d run my script, it would break, I’d look at the page source again, tweak the script, run it again. Repeat, repeat, repeat.
After getting it somewhat working for one site, I realized just knowing the current price wasn’t enough. Is it actually a deal? So, I needed to store the price history. Started super basic, just writing the product name and price to a simple text file each time the script ran. Crude, but it worked for testing.
Expanding and Refining
Okay, one site down, maybe two. But manually running a script is a pain. I wanted this automated. So, I set up a task scheduler on my computer to just run the script automatically, say, once a day early in the morning.
Now I was getting data, but it was just raw data in files. Not very useful. I needed to compare today’s price with yesterday’s price. Added some logic to the script: load the old price, get the new price, if the new price is lower by a certain amount (say, 10%), then flag it as a potential deal.
The text files were getting messy fast. Decided to move to a simple database. Nothing fancy, just something basic to store product name, date, price, and the website it came from. Made managing the data and comparing prices much easier.
Then I added more websites. Each one was its own little challenge. Different layouts, different ways they tried to block simple scripts. Had to make the script act a bit more like a real browser sometimes, adding pauses, changing how it identified itself. Some sites were just too difficult or changed too often, so I dropped them.
- Figured out website structure.
- Wrote script to grab page content.
- Extracted specific data (name, price).
- Added price history storage (first files, then database).
- Built logic to compare prices and identify discounts.
- Automated the script to run daily.
- Handled issues with website changes and blocking attempts.
- Added more sites, refining the process each time.
Finally, I needed a way to see the deals without digging through the database. Set up a simple email alert. If the script found any items meeting my discount criteria, it would bundle them up into a quick email and send it to me. So I’d wake up, check my email, and see a list of potential deals from the sites I cared about.
It’s not a commercial product or anything super polished. It breaks sometimes when websites overhaul their design. I have to go in and tweak the scraping logic now and then. But it does what I originally wanted: it saves me time and helps me catch deals I might have otherwise missed. It was a fun little project, lots of fiddling around, but satisfying to get it working.