Looking Back for Inspiration Part 1: Gigsniffr

Perhaps my fondest memory in all my time spent coding was the all-out dash to build a complex PHP web-app for my undergrad dissertation. It was the biggest project I'd ever attempted, and was the starting-gun I'd been waiting for, the opportunity to really dig into something.

Prototype Homepage

Man, I was so psyched about my idea - it was a web-scraper named Gigsniffr which aggregated a set of concert ticket websites. The killer feature was that when it saw a ticket come onsale for a user's favourite band, it alerted them via a tweet (back when Twitter was still conceived of as a computer-human API platform). At the time, nothing much like this yet existed (2007 was also when Songkick launched), and I knew for sure that it could be big if I could just get it launched.

This was before any of these music sites had a published API for me to use, so I had to really research how to put together a reliable web-scraping algorithm that could repeatedly spider the ticket sites.

The project was incredibly fun: I was developing into (for me) the complete unknown - I wasn't amazing at PHP, I had no experience of managing a long-running web-scraper service, I had no permission to scrape these sites, I had a hard deadline to hit, and nothing to distract me.

There were many problems to solve: how can you work out the names of a user's nearest cities, without Google's Map API; how do you build an engine to accurately scrape many different websites, abstracting away from the quirks of each (answer: invent own XPath tricks); how do you diff the data collected with that taken an hour previously?

I soon found that the first pass of data gathering was easy - but updating and correcting this dataset was actually much harder: I was attempting to keep track of gig cancellations, date changes, and variances in band names (Red Hot Chili Peppers vs RHCP etc), and it was a rathole without end. Much fun to solve!

I did manage to get a prototype working in time for the deadline - an achievement which to this day I'm proud of. At the presentation were two local entrepreneurs who took an interest in my idea and took me out for lunch to discuss it.

Alas - I ground to a halt somewhere between finishing the proof-of-concept and launching a user-ready product. After I'd delivered the proof-of-concept and completed the course, the pressure was off and it was time to relax - I'd hit the deadline and could now continue developing Gigsniffr at my leisure, right after I've been to these music festivals and learned Ruby and started a job and...

.. and it was gradually forgotten. The would-be investors were interested only in funding marketing, and only once the app was finished - they were not interested in financing even the peanuts I'd need to sit down and complete it. I was also messed up after a nasty breakup. I found myself in a job that absorbed 10 hours of coding stamina per day, which I then quit to take a Masters - without any real gap after the undergrad. If I'm honest, I'd got what I wanted from the project, and was moving on - so Gigsniffr was shelved.

I've a profound admiration for those running startups which actually launch, especially those who succeed longterm - it's an amazing feat. People say that ideas are cheap, and that building them is the hard bit. What they don't mention is that building the interesting parts of a project is pretty easy, but FINISHING it is nearly impossible in comparison. Finishing a product really is hard. But the burst of real energy and creativity I summoned for Gigsniffr is something I've been chasing ever since.

Prototype Admin panel

Prototype scraping control panel