Skimr: The Inside Story

I met my two Skimr co-founders, Josf and Davd, at my current employer – Seznam.cz. It’s basically the Czech Yahoo – email, maps, news, search, classifieds, etc. We are part of a small team (skunk works or Google X, if you will) where we explore new product ideas, create prototypes, experiment … some of these make it into production.

BTW, my colleagues’ real names are Josef and David. The shortened versions, Josf and Davd, come from VentureBeat’s Dylan Tweney. He thought my name was Peter and I had shortened my name in the same way we shortened the Skimr name (instead of Skimmer). So then we applied that to my colleagues’ names. We think it’s pretty funny, so we keep referring to them as Josf and Davd.

I’ve been using Google Reader for a long time. Basically since the day it launched. I then moved to the popular Reeder app. My favorite subscriptions have always been TechCrunch, Techmeme, etc. However, due to the time difference between Europe and US, the majority of articles were published during the night. So in the morning, I had a gazillion of new unread posts – impossible to read them all.

So I was thinking of a way to literally skim through the headlines and quickly see what’s new. I realized the title alone was sometimes not enough, but a combination of the title plus the first one or two sentences was typically enough to get the basic story. If it was interesting enough, I could click on it and read the full article.

After I realized this, I asked Josf if we could take Techmeme’s RSS feed and display it on a web page in this format. A few days later, I was using this web page all the time. I showed it to my friends and they all asked if they could add their own feeds.

I put together some mockups, and Josf started coding them. Then, I asked my other colleague Davd, who is a great designer, to help me make the mockups look nice. You know, fonts, colors, proportions, etc. At this stage, we were still building a small tool for ourselves and our friends. So the backend was very basic, only able to handle tens of unique RSS feeds, max.

A few weeks later, Skimr was up and running and all our friends were using it. Daily. Many times a day. It was unbelievable. It encouraged me to post it on a few sites and see what other people had to say. And that’s when it started to get complicated …

We didn’t really know any journalists or bloggers outside of the Czech Republic who we could ask to write a review. So, I went to Quora and found a few posts there that were listing websites where people could submit their projects. After submitting Skimr to approx 20 websites, I decided I was not going to continue this way because it had no effect at all.

There was one site left on my list – Web.Appstorm.net. So, I pushed myself and submitted Skimr there too. Five minutes later, I hade an email from their editor, Matthew Guay, saying he was just looking for something interesting to write about for the upcoming weekend and out of nowhere, Skimr landed in his mailbox. He liked the service, so he immediately wrote a review.

And then on Monday, more reviews started to pop up.

I was extremely happy. Davd, too. Only Josf was furious. His backend was absolutely not built for a larger user base. Fortunately, we were running Skimr on AWS, so there was always an option to rent more instances. Also, we have learned there is a devastating conversion rate between tweet-article-visitor-registered user.

Let’s say a blogger writes an article and promotes it to his 10,000 followers on Twitter. 10% of them will click on the link and read the article. That’s 1,000 people. 10% of them will click on the link within the article and actually visit your site. That’s 100 people. And then 10% of them sign up, that’s 10 people.

So, a few weeks later, we had a small user base of people who really liked the service, were sending us emails about features they wanted us to add and telling their friends their friends about Skimr. They were adding a few unique feeds an hour, so no big deal.

But then, Google announced they were going to shut down their Reader. All of a sudden, people all around the world started looking for alternatives. Instead of a few unique RSS feeds being added to Skimr every hour, there were 100 unique feeds added every minute. And then 1,000 feeds every minute. We kept upgrading our AWS instances, launching more and more download robots, renting more storage, etc.

It was fascinating. But we knew it was only a matter of time when our tiny system would break under such a heavy load.

What’s more, journalists became very responsive. I started hearing back from them. Even Walt Mossberg replied to my email to him. Coming from a small country, it was amazing.

Fortunately, over the weekend, people calmed down a bit. We immediately contacted my long time friend Radm who is an excellent backend programmer to help us out. The following paragraphs come from him:

The current version is build on LAMP technology + JavaScript based front end. This solution works perfectly as a proof of concept and is able to handle traffic up to a certain degree.

After Google announced the end of its Reader, the traffic grew significantly and we handled this situation by adding new AWS servers. This solution had one big disadvantage – it was pretty expensive and the PHP backend was not really using the servers efficiently. To solve this challenge as a first step we decided to migrate the site from AWS to our own dedicated servers (based in Prague). This saved us approximately 2/3 of the operational costs.

As a next step we worked on a rewrite of the site to a more efficient technology – to the same technology we have been successfully using for several years now at www.agentka.cz (a popular local job site). Usually, one of the bottlenecks on heavily used sites is the DB which is under high load. This is a paradox as most of the sites are not really using advanced (sometimes even basic) SQL database features the way they were designed – heavy loaded databases are usually denormalized and serves only as a persistent storage. The situation is even worse when it is required to perform full-text searches over all stored data.

In the past, this observation led us to drop SQL database and use a special indexer that stores documents with a very loose structure and provides very fast full-text search features and, of course, provides support for atomic operations. This indexer is the heart of the site and it is used for storing all data, it also allows us to scale the site the way we need to. After we migrate the current Skimr implementation off LAMP to the described one we will be able to provide a lot of nice features to our users – for example full text search in the feeds. This work is in progress and we expect to have a new version in several weeks’ time.

Launching a web project always brings this dilemma – launch quickly and risk the system breaking if it becomes popular; or build it properly first and be ready. If we had chosen the second approach, we would have completely missed the window of opportunity that appeared when Google anounced the Reader closure.

Let’s go from the backend to seeing how the frontend works. It has two basic parts:

1. Static pages generated on the server via PHP, e.g. http://skimr.co/about
2. Dynamic pages generated on the client in javascript, e.g. http://skimr.co/techcrunch

Interesting is the second part – dynamic pages. The core of the application is just a vanilla JavaScript build with static object literals (single modules for homepage, articles, updater, etc). We don’t use any big JavaScript library like jQuery. We want to have full control with a focus on speed and simplicity.

Rendering of HTML is done strictly by Dust.js with precompiled templates on the server. JavaScripts don’t contain any HTML code or any direct manipulation of the DOM. We just supply data from the server via the JSON API, process them and send them to a render function that generates HTML for an active page.

Working with URLs is done by HTML5 function pushState() with a fallback to hash. This is very important and the main key to a really “responsive” application, because you are able to “switch” pages instantly.

All in all, the frontend is quite simple and completely responsive – works nicely on desktops, tablets and smartphones, out of the box. Everything is done just in CSS, we don’t have any application logic to support responsive function of the application. We are also using LESS in style sheets. We can definitely recommend LESS for any kind of web or application.

Regarding design, I wanted Skimr to be as simple as possible. I’ve always thought traditional RSS readers were a bit too complicated for the average users. But I liked the concept of pulling in content via RSS and showing it in a unified form. Yet, I thought it was fair for us to only show a small part of the original content, not the whole article. If people liked it, they could click and read it on the publisher’s website.

The biggest problem of traditional websites is the number of elements people try to put on them. All the menus, options, contextual content, etc. I find them quite distracting. The point of any website should be to solve a specific need and then let go.

When it comes to news sites, they should tell their users what’s new and that’s about it. So I made sure there was a clear purpose as to why Skimr existed – to quickly see what’s new. After a few iterations, I’ve realized there was a simpler way than a traditional RSS reader – just a simple feed. Latest entries on the top, older below. Nothing more, nothing less. It sounds so obvious now, but back then, it meant ditching the majority of features a good RSS reader would have.

Another interesting aspect of building Skimr is that of native vs. web apps. I had an interview with ZDNet’s Dave Morgenstern on this topic.

Basically, I think very soon, people won’t be able to tell the difference between native and web apps. The latest developments in CSS, JS and HTML 5 enable app designers to create almost identical user experiences to traditional native apps.

So, I thought web app would be totally enough. But I was wrong.

One of the main requests from our users are native apps. They think native apps offer better performance. Since it’s basically a distribution channel, we have decided to build these for them.

Speaking of new feature requests, folders are an interesting story. A long time ago, I used to be subscribed to many websites in Google Reader. Over time, I reduced the number to approximately 10. On that principle, we’ve designed Skimr for users with few feeds. It turned out very quickly that there are many people all around the world who still consume hundreds or maybe even thousands of feeds. In this scenario, folders are necessary to organize these feeds. So, lesson learned: we’ve developed folders for our users.

We have many ideas as to how to make Skimr even better for our users, for example native apps, search (within a user’s RSS feeds or even all the RSS feeds in the system). But it is very important to always think twice before implementing a user request.

Since we are not building a traditional RSS reader, some of our users (with all due respect), don’t quite get this, so they ask for features they are used to on Google Reader. Our path is different from Google Reader, Feedly, etc., so we might not make everyone happy. The benefit of this approach, however, is that the majority of our users are actually very happy we are offering something different. We have found a niche, where people are fed up by the complex readers out there and are happy with the simplicity Skimr offers. They are even sending us emails asking us to keep Skimr as is and not add any new features at all. Amazing!

Going forward, we can expect some spikes in traffic around June, when Google will definitely sunset Reader. We will do our best to prepare for them properly, mainly rewriting the backend to the no-DB stuff.

But, in general, we are more interested in organic growth, rather than overnight success. It helps to create a bond with the users, listen to what they have to say and carefully improve the product. After all, Skimr is a hobby project for us. We all have full time jobs.

Having said that, there are obvious business models we could try, especially Freemium (some special features for a small fee).

So far, the Skimr project has been full of surprises, so who know what comes next?