Well, it certainly has been an incredible past 3 weeks. It's almost hard to believe, but it has only been 20 days since we got the initial call from CNN, and the entire team here has been through a firedrill the likes of which I've never seen. In three short weeks, we had to completely revamp and fix a number of core scalability issues with the Technorati service as well as roll out a new site redesign ahead of schedule, not to mention the new politics site as well. I'm amazed that we weathered the storm. Site traffic grew by 8x to 10x over or previous levels, the site is now valid XHTML 1.0, is completely CSS driven, more cross-browser friendly, and we added 60 more servers to handle the increased database demand. We now have far fewer single points of failure, which will allow the service to grow more smoothly in the future.
Unfortunately, we also broke a bunch of things. First off, user logins got broken when we switched to a new load-balanced front-end system. This new system allows us to put as many web servers out there as we wanted, but in the rush, we broke the code that performed user sessions - which meant that user logins don't work correctly. We put in a new caching subsystem which wasn't fully tested, and sometimes service results went awry. Weblog claims aren't working right for a small segment of our members, a nagging problem that we've got to get working as well. And worst of all, email watchlists for some of our paying subscribers have been broken, and we haven't been on the ball in fixing it.
I apologize.
For as much progress that we've made in the last few weeks and months, with all the increased visibility and traffic, we've been so focused on the trees, that we've lost sight of the forest - which are you guys, the bloggers and users of the service. We've gotten offtrack, thinking about new features and enhancements before handling and fixing the known issues we already had (gosh darn it, there are so many cool features to do). We lost sight of the fact that we're here to be of service to you.
There's no good excuse for that.
All I can say is that we're committed to fixing the issues, and to building a rock-solid infrastructure and bug-free service. The first thing we're doing is fixing the session/login issues, so those of you who weren't able to log in will be able to log in. Responsiveness and customer support is also a top priority.
Over the last few days, I've gotten a lot of email and pointers to blog entries that commented on the service - both successes and frustrations. Thanks for the criticism and feedback. Some of it was brutal, but that's the best kind - the stuff that hits closest to the heart. Like how we were being unresponsive. That's just unacceptable. And we're going to fix that. We're taking this week to go back and take a serious look at what we did right, what we did wrong, and how we can better serve all of you going forward. Thanks again for using Technorati, and for all of the time, attention, and feedback. We're committed to making things right, and providing a great service to all of you.