Realtime Data Processing

Realtime environment have to play with data to make connections, be able to provide smart searching that does not only depend on full-text comparison. Imagine that they have to tokenize text input to post-process what’s going on with the message. Almost requires O(n) on CPU. Twitter has about 1 million users, let’s assume averagely every user enters a new twit once in 3 days and average entry is 100 characters long (doesn’t sound realistic, but let’s be optimistic). It converges to 350k posts a day. 350k times tokenizing the 100 character-long input = 35 million characters are processed during a day.  I tried to tokenize a 100 char string 350k times and it took only 41 seconds since I was using the same string over and over again. With helps of caching, CPU minimizes the memory I/O and it made a huge difference, so my 41 seconds were far from being accurate. But besides tokenizing, there are other operations you have to run. And once you fetch it from memory, you’re done. Therefore, I believe it’s not really an extra load to tokenize the input on the server-side.

But, what would you do if you have to post-process terabytes of imagery? I’m not sure if you are aware of Microsoft’s Virtual Earth 3D but, it is more like Google Earth running on your browser.

A very long time ago, I was making a demo to my friends and showing the Mt. Rainer in WA distinct in 3D mode. Virtual Earth 3D fetches higher quality imagery for forefront. Since there is no colour balance adjustments at VE imagery, many people thought  the level differences on the scene is sort of a corruption and not good looking. I decided to talk to engineers that we can solve this problem with relative histogram equalization (fancy name but easy method). I didn’t sound perfectly realistic, cause our imagery were tens of terabytes and it was very risky to process them all for such a tiny improvement.

Luckily, I recognize client power for the first time and the power it brings out. I was going to add a improvement step to equalize imagery once it is loaded as “it is” to user machine. User’s itself was about to play a role to balance images on the front-end.

Future of the Web: More Realtime Data Needs More Client Power

Twitter example is quite lightweight, 35 million characters/day is not huge as you may guess. As millions started to use realtime equipment passionately and generate a hundred thousands times more content than this, we may have to take advantage of our user’s machines mostly for pre and post processing of data. All of the new client technologies are about the provide a rich environment and framework to let the users contribute to the overall system with scalability. Future is going to be all about making a distribution decision between clients and servers. If you are a server guy who has no idea on the user machines, study your lesson before this balance dominates the world.

Edit: Sorry for quoting VE’s 3D mode with Silverlight, thanks for Jon O’Brein (a Microsoft MVP) for fixing my deadly fault. I was high or something.