Archive

Tag Archives: paas

For a quick recap of last years Deploycon & related talks, check out my “Day #3 => DeployCon && Enterprise && Data Gravity” entry from last year.

PaaS Systems aren’t always effectively distributed. Heroku has fallen over every time east-1 has gone down at AWS. Not that I’m saying they’ve done bad, just pointing that out. With Cloud Foundry, there’s several key SPOFs (Single Points of Failure), and with all PaaS Systems the data tier is often the neglected pairing of the system. I’ve been wanting to write about this for a few months now and Deploycon has lit a fire for me to do just that.

Deploycon – “Platform Services and Developer Expectations” **

I’m on a panel at Deploycon titled “Platform Services and Developer Expectations” and this leads right back around to that. This SPOF issue is concerning to me as PaaS Providers talk up the offerings more and more with little light actually shone on this issue. In some ways each is moving away form their respective SPOFs, but overall they’re all pretty prevalent throughout. For security, each has a non-distributed database, which technically needs backed up still – no clear replication or other mechanisms setup to ensure data integrity in a failure situation. Of course, the huge saving grace with a PaaS, is that if the overall system goes down or a SPOF blows up, all the existing deployed applications will generally continue to run. Unless of course the routing and networking are also SPOF. This is the largest glaring concern with PaaS Systems that I see today.

One of the other things about PaaS that has always led to a ton of questions is “what about my PostGresql/mysql/Riak/mongodb/database thing and how do I do X, Y, Z with it to ensure scalability in my PaaS.” In almost every case it ends with a simple and unfortunate answer, ”…when it comes to data, a PaaS doesn’t really do a damn thing for ya…” This is obviously not very helpful. The entire reason to put a PaaS into place is to simplify life, the sad fact that it barely does a thing for the data tier isn’t very helpful.

Now, hold on a second before you start screaming at me about “but a PaaS does X, Y and Z and isn’t even supposed to touch that aspect of things…” let me elaborate a bit more. The panel at Deploycon states “…Developer Expectations” and when things are getting simplified in the way a PaaS does, developers assume that if it does all this fancy magic for an application it ought to simplify the data side of things too! Right? Well no, and it isn’t going to for the foreseeable future. But no matter what, it doesn’t change the fact that developers often have that expectation.

Now, I could write at length about all the reasons that PaaS doesn’t really do anything for the data tier. I could wax poetic about how a distributed database (re: Riak, Cassandra, etc) just doesn’t lend itself to a cookie cutter approach to deployment under a PaaS or an RDBMS has umpteen different configurations for stability, scaling, hot swappable services, and other such complexities around the data tier. But instead I’m going to skip all, maybe cover some of those things another day, and jump right into some of the things that are actually moving forward to fill this gap.

BOSH, Cloud Foundry, OpenShift & fixing the data tier…

The most obvious reason there isn’t a simple turn key solution to the data side of things with a PaaS ecosystem is that data is complex and extremely diverse. There’s distributed key/value stores (Riak, Cassandra), there’s sort of kind of distributed databases (Mongo), graph databases (Neo4j), the age old RDBMS (DB2, SQL Server, Oracle’s Stuff, etc) and the million solutions around that, there’s key/value in memory styled databases that are insanely fast, like Redis. Expanding just slightly you have software that works around these systems such as Hadoop & Riak CS & the list goes on. All of it focused on the data tier and maintaining one, two or some form of the three points around CAP Theorem (http://en.wikipedia.org/wiki/CAP_theorem), atomicity and other key capbilities.

All of the PaaS Systems, including public and private often have some sort of plug-in style architectures for data. Whether it is Apprenda which is closed to community and closed source or an ongoing open to community PaaS like OpenShift or Cloud Foundry, things still fall almost entirely to the developers or database team to build an architecture around the data. When looking at solutions to simplify data in PaaS Systems the closed source solutions we have no idea what they’re up to in this regard. The one’s that are open source or in large part public and involved in the community PaaSes, like EngineYard, Heroku, Cloudbees and others we can really see the directions and efforts around creating real PaaS style solutions to the data tier problem.

BOSH, Vagrant, etc…  One of the best solutions I’ve seen so far is the ability of Bosh, which was created by the Cloud Foundry team while at VMware, to spool up an environment that includes such things as a Riak Cluster (or other cluster). Currently Brian McClain & Dr Nic have worked to put together such Bosh + Vagrant scripts & get things rolling. I myself will be spending some considerable time on just that. But beyond that this is a good start in enabling data tier back ends.

How to close the gap, between absurdly simple application deployment and still arduous and difficult data tier deployment? For the next several years I think we’ll have cumbersome deployment practices around the data tier. There won’t be anything as elegantly simple as Cloud Foundry’s single line deployment or AppFog’s one click deployment of a web application. The best we can do at this time, is to streamline around pieces and architectures, and at least get them into a kind of simple 3 step deployment.

Please drop a comment or two on how you think we might simplify the data side of the PaaS toolchain. Also drop a few tweets in the twitterverse too, I’m sure that’ll be exploding as usual. I’m @adron, ping me.

Cheers, happy data architecting.

** the Deployconpanel will be at 4:30pm in Santa Clara on April 2nd. Come check it out.

So today we’re super excited to release Thor release candidate from the furnaces of the Iron Foundry. We’ve had number of people working not he project and core Objective-C Coder Benjamin van der Veen @bvanderveen (Twitter), @bvanderveen (Github) and site tearing through tests, implementation, refactoring and UI hacking non-stop these last few weeks. I’ll admit, I think he’s slept some, but nobody knows.

With this new release, the features around…  well…  check out the video.

For a more complete list of the features check out Github, Github Issues & the Iron Foundry Blog.

NOTE: All of this code is available at my Github Project “Remembering” (https://github.com/Adron/Remembering). Feel free to fork it, share it, or send me corrections or pull requests.

While working on the Thor Project there have been numerous situations where I need to fire off an asynchronous callback in C# while maintaining good responsiveness in the actual user interface. Benjamin (@bvanderveen) has been using Reactive Extensions with subscriptions to do this in the Objective-C code for the Cocoa based OS-X Thor user interface. See my previous blog entry for an example of what he’s doing.

For a good example of asynchronous calls against Cloud Foundry I setup the following project using the Iron Foundry Project VCAP Client Library. The first thing I setup was a static class with a few constants to use across the examples for the URI, username and password for the Cloud Foundry Account.

public static class YourSecrets
{
    public const string Username = "youremail@someplace.com";
    public const string Password = "AnAwesom3HardPassw0rd!";
    public const string Uri = "http://api.yourpaas.com";
}

Next step was to setup the delegate and method I’d use for calling out to the Cloud Foundry environment and retrieving data in parallel to my active console or user interface. That code snippet looked like this. I also added a private variable _finished for use in tracking when the request was completed in the begin and end invoke example below.

private bool _finished;

IEnumerable TheMethodToConnectThatWillTakeLongTime(string uri)
{
    var client = new VcapClient(uri);
    client.Login(TheSecretBits.YourSecrets.Username, TheSecretBits.YourSecrets.Password);

    _finished = false;

    return client.GetApplications();
}

delegate IEnumerable MethodDelegate(string uri);

Once I had that setup I was ready to create my baseline method that would make a synchronous call. A synchronous call is one that makes the call as if it just called the method directly. There’s no real reason to create one like I’ve done here, but I was just using it to provide a basic example of calling the delegate.

public void SynchronousCall()
{
    var starting = DateTime.Now.ToLongTimeString();

    var delegateMethod = new MethodDelegate(TheMethodToConnectThatWillTakeLongTime);
    var returnedBits = delegateMethod(TheSecretBits.YourSecrets.Uri);

    var ending = DateTime.Now.ToLongTimeString();

    Console.WriteLine(string.Format("The delegate call returned \n\n{0}\n\nstarting at {1} and

ending at {2} which takes a while of waiting.",
        returnedBits, starting, ending));

    _finished = false;
}

That gets us a baseline. If you run a synchronous call against anything with a console application or a windows app, WPF or whatever it will lock up the calling thread while it is waiting for a response. In any type of user interface that is unacceptable. One of the best options is to fire of an asynchronous callback. The way I did this, which is an ideal way to make calls with the Iron Foundry Client Library against a Cloud Foundry Environment, is shown below.

This is my asynchronous call.

public void DemoCall()
{
    Console.WriteLine("Callback:");
    var delegateMethod = new MethodDelegate(TheMethodToConnectThatWillTakeLongTime);

    var callbackDelegate = new AsyncCallback(MyAsyncCallback);

    Console.WriteLine(" starting...{0}", DateTime.Now.ToLongTimeString());
    delegateMethod.BeginInvoke(TheSecretBits.YourSecrets.Uri, callbackDelegate, delegateMethod);
    Console.WriteLine(" ending...{0}", DateTime.Now.ToLongTimeString());
}

Now the simple callback.

public void MyAsyncCallback(IAsyncResult ar)
{
    Console.WriteLine("Things happening, async state calling.");

    var delegateMethod = (MethodDelegate)ar.AsyncState;

    Console.WriteLine(" called...{0}", DateTime.Now.ToLongTimeString());

    var returnedBits = delegateMethod.EndInvoke(ar);

    Console.WriteLine(" end invoked...{0}", DateTime.Now.ToLongTimeString());

    foreach (Application application in returnedBits)
    {
        Console.WriteLine("Application {0} is in {1} state...",
            application.Name, application.State);
        Console.WriteLine(" with {0} running instances, {1} memory per instance, {2} disk allocated...",
            application.RunningInstances, application.Resources.Memory, application.Resources.Disk);
        Console.Write(" hosted at ");
        foreach (var uri in application.Uris)
        {
            Console.Write(uri + " ");
        }
        Console.WriteLine(".");
    }
}

That’ll get the call running on a parallel thread and when it is wrapped up it returns the data.

The User Interface Interaction Issue

This is all fine and dandy for the command console. But if you want to give control back to the UI thread in a UI application and make sure that the background thread can actually update a control when fired off, do the same thing as I’ve discussed here except set the control up to invoke the dispatcher, so that the “threads don’t cross” when trying to return information to a control that needs updated. In order to do this take the control that needs updated and set the Dispatcher Invoke method as shown below.

private void Write(string updateText)
{
    UpdatingTextBlock.Dispatcher.Invoke(
    System.Windows.Threading.DispatcherPriority.Normal,
        new Action(delegate
    {
        UpdatingTextBlock.Text += updateText;
    }
    ));
}

For more on the Iron Foundry Project and the library I’ve used here, check out the Iron Foundry Blog & Site. For more information on Thor and to follow or get involved check out the Thor Project Site (Hosted with Cloud Foundry at Tier 3!).

All of this code is available in my Github Project “Remembering” (https://github.com/Adron/Remembering). Feel free to fork it, share it, or send me corrections or pull requests.

I was speaking with Krishan Subramanian (@krishnan) and Adam Seligman (@adamse) today. I love talking to these guys. They’re both smart, intelligent and upbeat guys. They see the positive things we’re all working toward and accomplishing in the technology space, specifically around PaaS, Cloud Computing and around the cultural implications of stronger technology communities, involvement of individuals. We all can see the positives, of how the industry is moving forward so that corporations aren’t the only enablers that are juxtaposed against developers or consumers but instead act to serve consumers based on the progress that individuals make themselves. There’s so much to do and so much progress to be made, the venders can simply follow the community and step up to provide points of leadership.

Absolutely great talking with these guys…

On that topic, what is it that we discussed that has me so motivated? Well there’s a few things that I’m done with and I’m going to make every effort to just throw away the trash. Here’s a few of these things that we discussed and I challenge everybody out there, drop the trash talk and let’s move forward because there is a LOT of awesome things to accomplish. Here’s the two things I’m just dropping…  cold. No reason to discuss them anymore.

  • Toss the language and framework religious wars. It is far simpler than it is sometimes perceived. We have a polyglot industry now where we can easily use the right tool for the job, the right framework, or the language that handles our particular domain the best. There is literally no reason to argue about this anymore. Of course we can talk semantics, debate best use cases, and of course we’ll talk accomplishments and what various things do well. That’s exactly what the focus should be on, not the harping on my X is better than your Y nonsense.
  • The culture war is basically over. Sure there are the hold outs that haven’t gotten a clue yet. But it’s an open source world at this point. Even the dreaded and horrible Oracle has generally conceded this and is frantically waving its marketing arms around trying to get attention. But at the core, mysql, java and the other things that they’ve purchased they’re keeping alive. They’re active participants in the community now, albeit in a somewhat strange way. Considering that even Oracle, Microsoft, Apple and so many others contribute back to the open source community in massive ways, that war can be considered won. Victory, the community and every individual in that community!
  • Lockin is basically dead. The technological reasons to lock in are gone, seriously. There’s some issues around data gravity that are to be overcome, but that’s where a solid architecture (see below) comes in. Anything you need can be contributed to and derived from the development community. Get involved and figure out how technology can be a major piece of your business in a positive way. If you design something poorly, lock in becomes a huge issue. Use the rights tools, don’t get into binding contracts, because in the polyglot world we’re in now there’s no reason to be permanently locked in to anything. Be flexible, be where you need to be, and make those decisions based on the community, your support systems, and your business partners. Don’t tie yourself to vendors unless there is mutual reasons to do exactly that. Lock in is a dead conversation, just don’t, time to move on.

So what are the key conversations today?

  • Ecosystem Architecture - If you’re deploying to AWS, Heroku, Tier 3, AppFog or Windows Azure it all boils down to something very specific that will make or break you. Your architecture. This is where the real value add in the cloud & respective systems are, but there are many discussions and many elements of the technology to understand. This is a fundamentally key conversation topic in the industry today. Pick this one up and drop the other trash.
  • Movement & Data Gravity – How do you access your data, how do you store it, where and how do you derive insight from that data? This is one of the topics that came up in our discusssion and it is huge. The entire computer industry basically exists for the reason of insight. What should we eat today, how do I shift my investments, how is my development team doing, what’s the status of my house being built, where is my family today and can I contact them! All of these things are insights we derive from computer systems. These are the fundamental core reason that computers exist. As an industry we’re finally getting to a point were we can get some pretty solid insightful, intelligent and useful information from our systems. The conversation however continues, there is so much more we can still achieve. So again, drop the wasteful convo and jump on board the conversations about data, information and insights!
  • Community Involvement – I’ve left the key topic for last. This is huge, companies have to be involved today. Companies aren’t dictating progress but instead the community is leading as it should. The community is providing a path for companies to follow or lead, but the community, the individuals are the ones that are seen and known to be innovating. This is so simple it’s wild that it is only now becoming a known reality – companies don’t innovate, people do. Companies don’t involve, people do. Individuals are the drivers of companies, the drivers of Governments, they’re the ones driving innovation and progress. The focus should now and should have always been on the individuals and what they’re working toward to accomplish. So get involved, get the companies involved as a whole and keep the semantic ideal of individuals and the progress they can make core to the way you think of communities. The idea of the “company” innovating is silly, let’s talk and build community with the people that are working around and innovating with these technologies.

Of course there are more, I’d love to hear your take on what the conversations of today should be about. What do we need to resolve? How do we improve our lives, our work and the efforts we’re working toward on a day to day basis?

Morning!

I put together a video late last night based on setting up a Cloud Foundry instance with Iron Foundry extensions. It’s a little long, since I’ve mostly *not edited* it. The instructions have been available for a while but I thought, why not make a video and publish it. In the very near future we’re going to start laying out specifics on how to install individual elements of Cloud Foundry and Iron Foundry. We’ll also be providing deeper detail of exactly what these parts do, what is needed, and how to scale a non-micro environment.

Cloud Foundry + Iron Foundry from Adron Hall on Vimeo.

Cloud Foundry + Iron Foundry for a full framework list of options. This video shows the steps to get a Cloud Foundry Micro instance up and running, then get an Iron Foundry instance up and running, connect them and then view a full list of frameworks.

NOTE: This video is barely edited, so it is a little long because everything is mostly done in real time.

This shows how easy this really is. If you’re using any other PaaS software, especially if you’re trying to setup local development environments, I’d love to hear about your processes and also any ideas on how we could make this process easier. In the near future I want to put together a video showing Stackato too. They have a nice offering to get a similar environment setup with more UI elements, making it a bit easier to keep track of all the applications and such.

midnight mystery ride

at midnight, we ride.

Riding on Roadways

Writing on Riding on Roadways

Not Rich Yet

It's going to happen. Gotta find something to do until then.

Portland Transit Lane

The Untold stories of Portland Transit

craftedincarhartt

Carhartt Women's Blog

heydev

For the love of code

Nathan Evans' Nemesis of the Moment

My nemesis of the moment

Open Source Bridge: Presentation Proposals

Snippets, software architecture, lean, agile, management, and leadership bits.

Captured Refractions

A collection of my latest adventures, past reflections and other photos.

for the love of Nike

for the love of Nike

The Cloud Dev

Developing {for/ on/ the} Cloud...

Project Manager in a Cloudy IT World

Thoughts, comments and ideas from experiences as a Project Manager in IT

iBikeuBike

If I can bike... So can you!

MAX FAQs

Portland Light Rail

UX Success

User Experience Design, Agile Development, Lean UX, Start Up

The lost outpost

a weblog by Andy Piper about technology, photography, and life

SaintGimp

Agile development, software craftsmanship, continuous improvement - Eric Lee's blog

Clang and Clamour

pardon the construction noises while we build the internet

Kristen Mozian

social {good} design + experience

RightScale Blog

Cloud Management News & Conversations

Follow

Get every new post delivered to your Inbox.

Join 5,464 other followers

%d bloggers like this: