Saturday, July 30, 2011

[xedwbnth] Open source cloud Mandelbrot

You download some open-source software of two components.  The first component, you make some modifications, compile, and install on your local computer.  The second component, you make some modifications, and upload to your private machines in the cloud.  Then you perform computations using both, with the local computer probably handling the UI and the cloud handling the heavy lifting.

If it weren't for wanting to permit modifications, i.e., if it could be purely closed source software, other paradigms, such as software as a service, could work.  But it is open source that makes things exciting.  People can modify!  Software can become better!

We need some infrastructure to make this work easily and well.  Some means of renting a cloud machine per computational transaction, rather than per integer hour.  (Google App Engine and Amazon Beanstalk sort of do this, EC2 does not.)

I imagine each transaction consists of the entirety of the source code you want the cloud to execute and the input data.  However the cloud provider maintains a complicated server-side cache (kind of the opposite of client side caches on web browsers) of source code it has already seen, has already compiled.  Use tricks of a tree of hashes to decide if only a portion of the source code needs to be newly uploaded in comparison to what the server already has.

The motivating example is to do nearly instantaneous deep zooms of the Mandelbrot set fractal for the large 2560 by 1440 monitors that exist these days.  A back of the envelope calculation: 2560x1440 supersampled 8x8 points per pixel for antialiasing, average 1000000 iterations per point, each iteration is 5 floating point operations, so 1.18 peta flop per image, so a computer of 1.18 petaFLOPS to compute it within a second.  But you wouldn't need that amount of power continuously.

No comments :