Technorati turns 3, rolls out a major update

After months of work and weeks of testing, we just released an updated version of technorati.com. I hope after spending some time with the updated service, you’ll join me in celebrating its release.

For those of you with ADD, you can jump to the bottom of this post to get a summary of the changes and new features. For some basics, you can read our FAQ. Or, you can view the screencast created by our product manager, Liz Dunn, by clicking on the video below:

Three years ago this month, Technorati was officially incorporated and became a “real” company, moving out of the "science project" phase of its life. Since then, so much has happened – not only has Technorati grown and changed, so has the live web. When I first started Technorati, my primary concern was how to be of service to a growing universe of bloggers, which then numbered in the thousands. Today, we’re tracking about 50 million blogs and we’re increasingly including other forms of citizen media, including photos and video.

While we got a lot of positive feedback from bloggers about the core Technorati service, we were increasingly hearing that we weren’t being of equal service to folks who are new to the live web. And because our mantra is to be of service – we set about renovating the Technorati site to make it easier to use for all people, whether they regularly create stuff on the web or not.

In short, we sought to make Technorati easier and more intuitive to search, discover and organize the World Live Web. What we released today is a big step in that direction.

First, in addition to a simpler and brighter design, we've continued to improve our core search technologies that index new information within minutes of being posted to the Web. We listened hard to your feedback - and redesigned the search results pages to include much less intrusive (and well-marked) ads, more results above the fold, and better sorting options - including sorting by language, authority, and freshness. If you're confused, you can get a deeper explanation of keyword results and of URL results. In addition, our link-counting mechanisms have also been dramatically improved. If you're a blogger, you should notice that your blog is being counted much more regularly, and that your rankings and authority information is much more accurate and up-to-date. If you've claimed your blog on Technorati, make sure to Sign In - we've worked hard to personalize your experience around the site to help you keep track of who is linking to you and what you care about, especially on the Technorati homepage.

Part of this work is also to make things simpler and more stable, so that we can understand and respond to all of your support requests. Hopefully, it'll also reduce the confusion! I know that some folks have sent in support tickets that we haven't answered. If you're in that camp, I'm sorry. I hope that with the back-end and front-end changes, you'll find Technorati more usable and easier to understand. Give us another try. I hope our revamped help section will answer any remaining questions you have, but if not, drop us another line.

Second, we've worked hard to make serendipitous discovery of interesting stuff easier and more intuitive. Our new “Discover” section, drawing on a commonly understood newspaper metaphor, organizes the blogosphere into easily recognized categories, within which some of the most interesting recent posts are highlighted. From news and tech, to sports, business and life, scanning what’s hot in the global conversation at any given moment in time is now only one click away. In case you're wondering how we did this, Liz has a screencast for later this week that will explain. Suffice it to say that if you want your posts included in the discover section, you'll first want to claim your blogs and then configure them with information on the things you regularly write about.

We've improved and updated our Popular section, putting all of the most interesting information on one page, including top searches and tags, The Technorati Top 100, the Most Favorited Blogs, Videos, Books, News, and Movies.

Third, Technorati’s Favorites feature has also been improved, enabling you to select and organize those blogs you like the most, and to see the most recent posts from your favorites each time you return to the homepage (you just need to be signed in!) The service now makes it easier to organize your Favorite blogs by simplifying the process of “favoriting” a blog in multiple ways, including right from your browser bar (drag that link to your browser's toolbar).
If you've ever wanted to just get a better handle on who a blogger is and what they write about, have a look at our revamped blog directory. Simply put in a blog's URL into that searchbox and you'll get an at-a-glance view of what that blog is, how influential it is, and what the blogger writes about - even some interesting stats like their reach (thanks to Alexa), posting frequency, and more.

Finally, we’ve worked hard to do a much better job of highlighting the people behind the global phenomenon – the writers, the poets, the journalists, the commentators, the artists, the prognosticators, the grand-standers, the filmmakers, the justice-seekers, the photographers, the simple diarists and everyone else who publishes their expressions on the Web. I hope you’ll enjoy seeing the many ways in which we try to highlight the people who are the real power driving everything we do – we've put a lot more of these member photos on the site and I hope you enjoy learning more about the folks you love following most.

We've also been doing lots of advertising, syndication, and partnerships of late as well - you may have noticed that you can now see Technorati integrated into the Wall Street Journal Online, joining partnerships with The Washington Post, Newsweek, Der Spiegel, The Associated Press, and many other major media sites. If you're interested in advertising or a partnership, drop us a line.

Please let us know what you think and how it is we can continue to be of service to you as we seek to be the major intersection on the World Live Web.

So, in summary:

• Technorati is 3 years old! What an amazing trip so far.
• The World Live Web is all about people - We're here to help make sense of all the interesting stuff we do in real-time.
• Technorati's rolled out a major update to our site and to our back-end systems.
• We've made some major speed and accuracy improvements in core search.
• Link counting is a lot more accurate and timely.
• We've personalized the homepage so you can get a look at all the stuff you care about on one page.
• While we love expert bloggers, we've also spent a lot of time making Technorati understandable to normal people.
• We've added in lots of features to help you make sense of the blogosphere, including Discover, which is topic-based, Favorites, which gives YOU the power to pick your favorite blogs, and Popular, which algorithmically derives the most linked-to items in the last few days.
• We've made some big changes to blog profiles - allowing you to get stats about any blog that Technorati tracks, including the tags used, posting frequency, traffic, and Technorati ranking.
• We've made things easier for advertisers and partners, and we've been overwhelmed with demand. We're building out our capabilities, and if you're interested in advertising, drop us a line.
• This is just a start - there's more to come in the coming weeks and months, including better charts, more real-time spam detection and elimination, more real-time media indexing, microformats integration, and additional localization and language support.

If there's one thing I'm sure of, after 20 years of building internet services and web businesses, it's that we haven't got it all right. We're going to spend a lot of time listening to you, and fixing, tuning, and tweaking. What have we got wrong? What is still broken? Believe me, I have a list about 3 feet long of those! What have we missed?

I'd appreciate it if you can use the tag technoratifeedback on your posts, or send a trackback or comment on this post. I can't promise I'll get back to everyone with an immediate answer, but I promise that I'll read them all.

I'd also like to send out a big THANK YOU to the entire team at Technorati for all the hard work they put into this, and also to all of our great alpha and beta testers, who gave us such great feedback prior to the launch. THANKS!

Technorati Tags: birthday, blogosphere, blogs, blogsearch, discover, favorites, feedback, givingthanks, launch, linkcounts, microformats, update, scaling, search, search engine, serp, tags, technorati, technoratifeedback, top100

Guy Kawasaki Interview / Link count update

Guy Kawasaki has put up an interview with me that we did over email a couple of weeks ago. Guy was the first person I'd ever heard called a technology evangelist, and I've followed his career since he was at Apple in the 1980's and I was in high school in New York, hearing about magical places called Mountain View, Cupertino, and Palo Alto. His book The Macintosh Way changed my life, and molded the way that I think about products and the companies that build them. Guy was also one of the people that I'd always wanted to meet, but for some reason, we'd never had the chance until earlier this year, when I had the opportunity to give him a signed Technorati t-shirt (I learned from the master) when he entered the Technorati 100.

By the way, you may notice that the Top 100 may jump around a little this week, or that some non-blogs (or blog hosting tools) may occasionally sneak in for a short time - we're doing some maintenance on that code, and I apologize in advance if you see anything wonky. We're doing a major rollout of our link counting system this week which will make it more accurate and show updates more quickly. Because link counts had gotten stalled for some people in the past, we're clearing out a lot of the backlog over the next week or two, and this may cause a bit of strangeness with some of our applications like the Top 100.

The goal is to make sure that Technorati is always accurate, responsive, and comprehensive. Sorry if you see some mess over the next few weeks as we fully roll this new system into production.

Technorati Tags: blogsearch, guykawasaki, interview, linkcounts, scaling, search, statistics, stats, technorati, top100

State of the Blogosphere, April 2006 Part 2: On Language and Tagging

Late last month, I gave a high-level overview of the growth of the blogosphere, covering the overall size of the data sets that Technorati tracks, the number of new blogs created each day, the number of posts per day, and the issue of splogs or spam blogs.

To recap, here's the highlights of Part 1:

• Technorati now tracks over 35.3 37.3 Million blogs
• The blogosphere is doubling in size every 6 months
• It is now over 60 times bigger than it was 3 years ago
• On average, a new weblog is created every second of every day
• 19.4 million bloggers (55%) are still posting 3 months after their blogs are created
• Technorati tracks about 1.2 Million new blog posts each day, about 50,000 per hour

Strong International Growth

Back in April 2005, Technorati started automatically tracking the primary language of each blog that we tracked. We did this so that we could easily allow people to filter out posts in languages other than their native language. This is available in a pull-down menu on every search results page. We also wanted to get some idea of where the worldwide growth of blogging was taking place, and what trends we could glean from the data.

There are three very important caveats in the data sets that I'm going to describe below. The first is that we are using automated language analysis software (based on languid), and it may have bugs, thus over or undercounting a particular language or group of languages. We're going to be continually improving the capabilities of this software, but we are pretty confident in its ability to work reliably, especially over the large data sets that Technorati tracks (over 35 million blogs at this time, and over 1.2 million posts each day). Second, we believe that we are grossly undercounting the Korean blogosphere, mostly due to the fact that the largest Korean blog and hompy services (like Cyworld or Planet Weblog) are not being indexed by Technorati at this time. In addition, we believe that we're somewhat undercounting the French blogosphere, in particular because our indexing of skyblog is poor. We'd love to rectify this - if anyone at these (or other) blogging services is interested in being indexed, please drop me a line. Last, Japanese bloggers appear to write shorter posts more often. This could be a result of blogging from mobile phones, and may be skewing the results, given that we are tracking the total number of posts in this analysis.

Another key point to remember is that language breakdown does not necessarily imply a particular country or regional breakdown. For example, Spanish and English are spoken in a large number of countries around the globe - and this analysis doesn't attempt to determine from which country a blogger is writing from - only the primary language of her post.

The following charts show the relative volume of blog posts based on the primary language of the post, on a month by month basis:

Here's a more detailed breakdown of the last 6 months of data:

Something that may come as a surprise (at least to the English-speaking world) is that English isn't the biggest language of the blogosphere. In fact, English isn't even the primary language of one third of all posts that Technorati tracks anymore. Another interesting finding is that the Chinese blogosphere, which grew significantly in 2004 and 2005 (launches of MSN Spaces in Chinese, Bokee.com saw a peak of 25% of all posts in Chinese in November 2005) seems to be slowing down somewhat this year.

One of the further topics for research is to investigate the language breakdown of posting activity based on blog hosting site or software type. My hypothesis is that various language communities have often grown on one service or another, often for viral or historical reasons, showing a disproportionate language breakdown for that service. For example, livejournal.com hosts a large number of Russian language journals/blogs, and MSN Spaces hosts an overrepresentation of Chinese language blogs.

Tags and Categories

Tagging, the act of categorizing posts with simple words or phrases, continues to grow, and the number of posts with tags or categories has grown past the 100 Million mark since Technorati began tracking tags in January of 2005.

Nearly half (47%) of all blog posts have an author-generated category or set of tags associated with the post. For this analysis, Technorati excluded generic or default categories, like "General" or "Diary", which some services put into each post if the author doesn't specify a particular tag or category. We only counted posts that used a non-default tag or category.

Many bloggers use this tagging capability to help get their content found by people who are searching for a particular topic, even if that topic isn't listed as a keyword in the post. Of course, one of the remaining open questions is whether or not that will lead to massive gaming of the system, but current trends seem to present evidence that large-scale gaming is not occurring. In fact, my belief is that because tags are built as hyperlinks inside the document, and thus visible to the reader, that a strong social pressure to use appropriate tags (or at least to not use inappropriate tags) manifests itself, especially with bloggers who want to cultivate influence and readers.

Clarification: I had a number of questions from people to clarify the tagging statistics. 47% of daily blog posts that Technorati tracks (about 560,000 posts out of the 1.2 Million postings per day) have one or more tag or category associated with the post. Obviously that number fluctuates somewhat given the day and the number of postings tracked that day. Hope that clears things up!

In Summary:

• The blogosphere is multilingual, and deeply international
• English, while being the language of the majority of early bloggers, has fallen to less than a third of all blog posts in April 2006.
• Japanese and Chinese language blogging has grown significantly.
• Chinese language blogging, while continuing to grow on an absolute basis, has begun to decline as an overall percentage of the posts that Technorati tracks over the last 6 months
• Japanese, Chinese, English, Spanish, Italian, Russian, French, Portuguese, Dutch, and German are the languages with the greatest number of posts tracked by Technorati.
• The Korean language is underrepresented in this analysis
• Language breakdown does not necessarily imply a particular country or regional breakdown.
• Technorati now tracks more than 100 Million author-created tags and categories on blog posts.
• The rel-tag microformat has been adopted by a number of the large tool makers, making it easy for people to tag their posts. About 47% of all blog posts have non-default tags or categories associated with them.

Technorati Tags: blogging, blogosphere, blogs, blogsearch, charts, international, language, microformat, microformats, posts, search, search engine, sotb, sotb2006, statistics, stats, study, tags, technorati, technoratitag, weblog, weblogs

State of the Blogosphere, April 2006 Part 1: On Blogosphere Growth

Yes, another quarter has passed, and it is time to take a look at the numbers!

For historical perspective, you can see earlier State of the Blogosphere reports from February 2006, July 2005, from March 2005, and from October 2004.

The State of the Blogosphere is strong.

I continue to marvel at it, but the blogosphere continues to grow at a quickening pace. Technorati currently tracks 35.3 Million weblogs, and the blogosphere we track continues to double about every 6 months, as the chart below shows:

The blogosphere is over 60 times bigger than it was only 3 years ago.

New blog creation continues to grow. Technorati currently tracks over 75,000 new weblogs created every day, which means that on average, a new weblog is created every second of every day - and 19.4 million bloggers (55%) are still posting 3 months after their blogs are created. That's an increase both absolute and relative terms over just 3 months ago, when only 50.5% or 13.7 million blogs were active. In other words, even though there's a reasonable amount of tire-kicking going on, blogging continues to grow as a habitual activity.

In addition to that, about 3.9 million bloggers update their blogs at least weekly. Here's a chart of the number of new blogs created each day, from January 2004 to April 2006:

Spam, Splogs and Spings

Spam blogs and their cousins Spings (which I described in January's report) continue to present infrastructure providers like Technorati a challenge, as more people rely on understanding the real-time web There has been an increase in the overall noise level in the blogosphere during 2006, but aside from a few notable spam storms ("sporms"? Just how far can you take this naming system?) noted in red in the chart above, the high level of interesting, original content being created greatly outweighs the fake or duplicate content listed on splogs.

Posting Volume

A better indicator of the growth of the blogosphere than simply the number of new blogs created each day is the rate of postings to those blogs. Daily Posting Volume tracked by Technorati is now over 1.2 Million posts per day, which is about 50,000 posts per hour. The blogosphere also reacts to world events. I've pointed out a number of the spikes in posting volume that have accompanied major news events in the chart below of posting volume:

I wasn't able to identify all of the spikes, but I did find some of the notables. For example, it certainly appears that technology product launches attract great interest in the blogosphere - seems that we just can't restrain our inner geekiness when products like the iPod Video and the Intel Macintoshes were launched. Posting volumes on those two days even eclipsed blog coverage and commentary of the Superbowl and the 2006 State of the Union speech.

In summary:

• Technorati now tracks over 35.3 Million blogs
• The blogosphere is doubling in size every 6 months
• It is now over 60 times bigger than it was 3 years ago
• On average, a new weblog is created every second of every day
• 19.4 million bloggers (55%) are still posting 3 months after their blogs are created
• Technorati tracks about 1.2 Million new blog posts each day, about 50,000 per hour

Next: The growth of tagging, and the Blogosphere broken down by language

Technorati Tags: blogosphere, blogs, postingvolume, posts, postvolume, pr, scaling, search, search engine, sotb, sotb2006, spam, spamblog, statistics, stats, technorati, weblog, weblogs

Working offsite

Back in 2002 I cofounded a company called Sputnik, and the idea was to enable people to easilt share their WiFi access with each other and with strangers who wanted to "sip" off of their bandwidth, and in return, they would share their bandwidth as well. Back in 2002, though, that was not a very popular vision - folks were going out of business and selling their assets to Starbucks and T-Mobile.

We ended up pivoting the business to focus more on managed wifi for businesses and event locales, and Sputnik has been doing great. But I always held a special place in my heart for the original business model.

So it was with great pleasure that I met Martin Varsavsky a few months back at Les Blogs in Paris. Martin is an amazing guy, successful entrepreneur, philanthopist, and a force of nature. I'm rooting for the Fon guys.

I'm reminded of all of this because I'm sitting here doing my work in a Starbucks today, with WiFi provided by T-Mobile. Here's a picture of my little "mobile office":

By the way, that little cup is actually an espresso cup - I got addicted to espresso when I went to visit Rome for the first time a couple of months ago. Unfortunately, there just isn't anything like Italian espresso here in the states, but I'm getting by with the very bitter Starbucks espresso, which is a far approximation from the rich, smooth, full-flavored espresso I had in Italy...

Technorati Tags: fon, lesblogs, mobile, wifi, technorati, varsavsky

Tom Raftery podcast interview of me

Tom Raferty of Podleaders.com interviewed me a couple of days ago, and the podcast is now up! Tom is a great interviewer. He guided the interview well, nudging it in the direction he wanted it to go, while still making me feel at home. BTW, it was Tom's questioning that got me off my butt and got me to publish full feeds...

Technorati Tags: blogging, blogosphere, blogs, podcast, technorati

Going to Software2006 today

My old friend MR Rangaswami, who has forgotten more about Enterprise Software than I've ever known, is throwing his big conference down at the Santa Clara convention center today. I'll be heading over to check out some of the sessions in the morning, and hopefully get some blogging in as well. If you're in the world of Enterprise Software, you've got to take the day off and go to MR's conference, it is really a don't-miss event!

And if anyone is in the Santa Clara area and wants to go grab lunch or coffee drop me an email at dsifry AT technorati DOT com...

Upgraded to Full-text RSS feeds

I was talking to a popular blogger this week, and bemoaning the lack of time that I have had to blog recently, and he asked me, "Why is it that your blog doesn't have full-text feeds?"

I looked at him funny (over the phone, of course).

The real reason was that I've just been too busy to change it from the MT default that came installed.

But tonight, I made the move. If you're reading this via an Feedreader, hopefully you're seeing everything, and you're seeing it in all the rich color, with pictures and all the rest included.

If you're not subscribed to the blog, here's some easy ways:

RSS 2.0 Feed
Atom 0.3 Feed
RSS 1.0 Feed

Please let me know if you find any bugs! For all I know I've just made the blog show up in Spanish...

Technorati Tags: atom, blogging, feedreader, rss

For something completely different

I finally got around to uploading all my old pictures from my various digital cameras (going back to mid-2000) to flickr. Thanks Stewart and Caterina! I now feel much more comfortable that a disk crash on one of my servers won't cause a major family meltdown because all the pictures are now safely in the Yahoo! cloud.

The other nice thing is that I've been going through the old pictures looking for gems. I've got a few from back in 2001 when the family went to the Four Corners for a week in an RV and I took lots and lots of pictures with my old Nikon E990. Even though I've since upgraded to a Canon EOS10D for "serious" shooting and I keep my PowerShot SD450 with me nearly wherever I go, that even the old cameras, when you used a tripod, could take some great pictures.

Here's some pictures from that trip, you can see more in my old favorites set.

Meeting Guy Kawasaki

I got to meet Guy Kawasaki and give him a signed Technorati t-shirt, as he had just made it to the Technorati Top 100.

I've always admired Guy - I remember reading "The Macintosh Way" back when I was in high school, and it changed my life. So, it was great to finally meet him, and to swap some stories.

He's got a great blog - I always knew that Guy would become a great blogger - and it was really fun to be able to evangelize Technorati and blogging to one of the master evangelists. Man, I love my job!