Back in the Bosh Bunker

In the last post on the topic of Bosh I put together a simple Cloud Foundry environment using the tools & repos of Stark & Wayne. Even though the bootstrap is a great way to get an environment up and started, it doesn’t explain a lot of things about Bosh. So let’s take a look at what we’re dealing with here.

Bosh – What is it?

Bosh handles deployment and upgrades of Cloud Foundry environments. However, it isn’t particularly limited to just Cloud Foundry. It’s been used to launch Riak Clusters, setup Redis, Cassandra, CouchDB and other services that don’t just fit neatly in the Cloud Foundry services design.

It is a very important tool in regards to keeping a Cloud Foundry environment up to date with the latest bits, security fixes, bugs and related elements. Bosh is broken down into several key components that work together to handle these deployment and maintenance tasks.

To put it another way, Bosh aims to give ops or devops the ability to throw together an entire stack to deploy. Bosh starts with stemcells, packages and jobs as the core concepts of how it works.

Bosh is used, within Cloud Foundry and prospectively for whatever anyone would want to use it for, to launch instances, change out the instances, change networking values, IPs and other configuration information. Overall it kind of rolls a lot of other tooling (chef, puppet) together into one tool. How well it does this is up for debate, but I’m not arguing what it is here, just going to get some definitions here.

The Pieces of BOSH

Stemcells

A stem cell or stemcell is something that is a bit hard to track down a definition for. I’m taking a stab at it with what I know a stem cell is, so if you have any corrections please comment below – I’ll be more than happy to add a correction or three. Overall I understand a stem cell to be a complete framework stack built on some sort of virtual image. It can be thought of as the recipe for building an operating systems that will act as an active member of a Cloud Foundry environment. In some situations, such as with a distributed database like Riak, it becomes not so much a member of the Cloud Foundry environment itself but an active node available to a distributed database cluster. This can then be used as a distributed database that is managed by Bosh and accessible within the Cloud Foundry ecosystem.

Packages

A package is sourec with the appropriate scripts for building it into usable binaries. Think of this as a package in the Node.js NPM, Gems (Ruby/Rails), or Nugets (.NET) worlds. It’s something that Bosh will pull in and compile on demand.

There are a few key parts to a package, referred to as package specs. These are: name, dependencies and files. Of the specs, the name and files are really the only required parts. The dependencies are an optional list of other packages this package would depend on.

Jobs

This is pretty self-descriptive. The jobs within Bosh spool up, start servers and services and other miscellaneous responsibilities as needed.

Relavent Sites, Documentation & Key Content
  • The Cloud Foundry Bosh Repo => This is the actual code repository on Github. If you’re in need of really diving into what it does, there’s always the possibility of reading the code!

  • Cloud Foundry Documentation => This has links to documentation related to Bosh that is pivotal (no pun intended).

  • Bosh Documentation => This is the Bosh documentation. It’s almost a good idea to start on the “Running Cloud Foundry” part of the documentation. This documentation can use your help (it’s super sparse at the moment), so if you get going and using Bosh, please contribute with examples and other material.

  • Stark & Wayne Repositories => I already mentioned them, but they’re likely some of the best material out there.

  • Bosh DB => This is a site & repository that Brian McClain @brianmmcclain put together to keep track of bosh stem cells and other repositories related to launching certain tools, services, servers and other things in Cloud Foundry environments via Bosh.

  • Dr Nic’s intro to Bosh => This page serves as an into and description of what’s going on in Bosh. I read this a while back for my own kick off with the Bosh Tool.

Summary

This is what I’ve found and put together as a good starting point. I still think there’s a bit of confusion around what Bosh is, how it works, how to get started with it and having it clearly defined on the web. Documentation is getting better, but still needs a lot of work (remember, you too can contribute). For systems outside of Cloud Foundry it also is a bit difficult and sometimes sketchy to use Bosh as the primary means of deployment, maintenance and upgrading. But just like the documentation that is also getting better. I’ll have more coming in the near future regarding what Bosh is, how it works, and things you can do with it – until then check out Dr Nic’s material for the most up to date how-to and related documentations and videos. He’s done some great work with the tooling and continues to knock it out of the park.

Keep reading and I’ll have more definitions, outlines of what is what, and the entire inception that Bosh is.

Thor HAMMA! OS-X Cocoa UI for Cloud Foundry

So today we’re super excited to release Thor release candidate from the furnaces of the Iron Foundry. We’ve had number of people working not he project and core Objective-C Coder Benjamin van der Veen @bvanderveen (Twitter), @bvanderveen (Github) and site tearing through tests, implementation, refactoring and UI hacking non-stop these last few weeks. I’ll admit, I think he’s slept some, but nobody knows.

With this new release, the features around…  well…  check out the video.

For a more complete list of the features check out Github, Github Issues & the Iron Foundry Blog.

SITREP – Stoking Iron Foundry, Thor Hammering & Joining… ?

What have I been up to? Here’s a quick recap. You may want to get involved with some of these projects!

Iron Foundry & Tier 3 Web Fabric

Iron FoundryBack when I left the kick ass team at Russell Investments in Seattle I stepped directly to bat as team lead at Tier 3. My job, get a PaaS built on Cloud Foundry and extending that with Iron Foundry. It was an ambitious effort that would provide the most extensive framework and language support available from any PaaS Provider on the market.

Well, we did it, thanks to the capabilities of Cloud Foundry Community, the great minds of Jared Wray @jaredwray, Luke Bakken, Eric Lee @saintgimp, Cale Hoopes & the rest of the Tier 3 Team! I was able to add this to my list of successes. We had some bumps, some collisions, a brick wall or two and other scheduling problems – ya know, the standard things that happen on a project. But in spite of it all, we got the Web Fabric released – and it continues to be the only PaaS available with such a wide framework and language support. It ranges from Ruby on Rails, Erlang, Node.js & Java to .NET! If you’d like to check out the open source PaaS of Cloud Foundry & Iron Foundry both projects are always looking for participation & contributions!

Thor Brings the Hamma!

After the release of the Tier 3 Web Fabric I started the search for a wicked smart and capable OS-X / Cocoa Coder – it seems their availability is pretty limited these days! Well I finally lucked out and found Benjamin van der Veen (@bvanderveen, thanks for the intro Selena @selenamarie!) to help me get started on the Thor Projects. There’s the Cocoa Thor Project & the Windows 7 WPF Metro based Thor .NET Project that we’re wrapping up with v1 releases coming really soon. To check out more on these projects that I’ve lead & coded on check out the code bases & information, all linked on ironfoundry.org. The projects are open source, so feel free to jump in and help out or fork & submit pull requests. The team will be happy to review & discuss ASAP.

While we’re wrapping these projects up right now, I’ll actually be continuing on and supporting the projects: Thor & Thor .NET. I will continue to be involved, as I was saying, in a number of ways in the PaaS space. So don’t think I’m disappearing form that realm!

Basho Sings my Song

I’ve been keeping track of Basho for a while now. Riak caught my interest many months ago as a really well built, well thought out & advanced distributed database. As you might guess, being into the whole “cloud computing” industry, I’m just ever so slightly interested in distributed systems. The other thing that I’m a huge fan of, which Basho does, is heavily support and involve itself in the open source software community and movement. The icing on the cake, was their diverse use of systems and language use around Erlang. All things that are massive wins.

Bailey's Taproom

Bailey’s Taproom

Well during a random conversation with Eric @coderoshi at Bailey’s Tap Room & then attending the RICON 2012 Conference (article here and pictures) I spoke to some of the team and found out they were looking for some particular skill sets. Well it just happened that I was keenly interested in meeting those skill set requirements! So December 1st I’ll be joining the Basho team full time as developer advocate, evangelist, messenger or such for the northwest working with a few people you may know such as Mark Phillips @pharkmillups, Andy Gross @argv0 (thanks for the intro James @wattersjames), Eric Redmond @coderoshi, Shanley Kane @shanley, Casey Rosenthal @caseyrosenthal and many others. Simply, I’m freaking stoked.

How This Helps You Help Me Help You

Alright, so it’s great but how can I help you in your day to day? What data do you work with? Do you work with a data scientist? Are you a data scientist? Do you work with huge sets of data, many objects, large objects? I want to know about your data usage and data problems, because there’s a good chance we’ll have more than few things to discuss. Here’s some ways I can help you, help me, help you. Ping me if you’re interested in…

  • talking about your data usage at the monthly Riak user group.
  • coding, pairing & otherwise learning Erlang and the monthly Erlang group.
  • interested in coding, deploying and inventing new paradigms and patterns of data storage.
  • interested in pairing up to learn how to deploy, migrate, upgrade or otherwise use NoSQL solutions – namely Riak.
  • interested in Ruby on Rails, Node.js, Map Reduce, .NET, Java, PHP and how these things can and do work against data in everything from relational databases to the new echelon of NoSQL databases.

I hope to hear from you soon and see you at an upcoming user group, cheers!

Ways to Interact Asynchronously with C#

NOTE: All of this code is available at my Github Project “Remembering” (https://github.com/Adron/Remembering). Feel free to fork it, share it, or send me corrections or pull requests.

While working on the Thor Project there have been numerous situations where I need to fire off an asynchronous callback in C# while maintaining good responsiveness in the actual user interface. Benjamin (@bvanderveen) has been using Reactive Extensions with subscriptions to do this in the Objective-C code for the Cocoa based OS-X Thor user interface. See my previous blog entry for an example of what he’s doing.

For a good example of asynchronous calls against Cloud Foundry I setup the following project using the Iron Foundry Project VCAP Client Library. The first thing I setup was a static class with a few constants to use across the examples for the URI, username and password for the Cloud Foundry Account.

public static class YourSecrets
{
    public const string Username = "youremail@someplace.com";
    public const string Password = "AnAwesom3HardPassw0rd!";
    public const string Uri = "http://api.yourpaas.com";
}

Next step was to setup the delegate and method I’d use for calling out to the Cloud Foundry environment and retrieving data in parallel to my active console or user interface. That code snippet looked like this. I also added a private variable _finished for use in tracking when the request was completed in the begin and end invoke example below.

private bool _finished;

IEnumerable TheMethodToConnectThatWillTakeLongTime(string uri)
{
    var client = new VcapClient(uri);
    client.Login(TheSecretBits.YourSecrets.Username, TheSecretBits.YourSecrets.Password);

    _finished = false;

    return client.GetApplications();
}

delegate IEnumerable MethodDelegate(string uri);

Once I had that setup I was ready to create my baseline method that would make a synchronous call. A synchronous call is one that makes the call as if it just called the method directly. There’s no real reason to create one like I’ve done here, but I was just using it to provide a basic example of calling the delegate.

public void SynchronousCall()
{
    var starting = DateTime.Now.ToLongTimeString();

    var delegateMethod = new MethodDelegate(TheMethodToConnectThatWillTakeLongTime);
    var returnedBits = delegateMethod(TheSecretBits.YourSecrets.Uri);

    var ending = DateTime.Now.ToLongTimeString();

    Console.WriteLine(string.Format("The delegate call returned \n\n{0}\n\nstarting at {1} and

ending at {2} which takes a while of waiting.",
        returnedBits, starting, ending));

    _finished = false;
}

That gets us a baseline. If you run a synchronous call against anything with a console application or a windows app, WPF or whatever it will lock up the calling thread while it is waiting for a response. In any type of user interface that is unacceptable. One of the best options is to fire of an asynchronous callback. The way I did this, which is an ideal way to make calls with the Iron Foundry Client Library against a Cloud Foundry Environment, is shown below.

This is my asynchronous call.

public void DemoCall()
{
    Console.WriteLine("Callback:");
    var delegateMethod = new MethodDelegate(TheMethodToConnectThatWillTakeLongTime);

    var callbackDelegate = new AsyncCallback(MyAsyncCallback);

    Console.WriteLine(" starting...{0}", DateTime.Now.ToLongTimeString());
    delegateMethod.BeginInvoke(TheSecretBits.YourSecrets.Uri, callbackDelegate, delegateMethod);
    Console.WriteLine(" ending...{0}", DateTime.Now.ToLongTimeString());
}

Now the simple callback.

public void MyAsyncCallback(IAsyncResult ar)
{
    Console.WriteLine("Things happening, async state calling.");

    var delegateMethod = (MethodDelegate)ar.AsyncState;

    Console.WriteLine(" called...{0}", DateTime.Now.ToLongTimeString());

    var returnedBits = delegateMethod.EndInvoke(ar);

    Console.WriteLine(" end invoked...{0}", DateTime.Now.ToLongTimeString());

    foreach (Application application in returnedBits)
    {
        Console.WriteLine("Application {0} is in {1} state...",
            application.Name, application.State);
        Console.WriteLine(" with {0} running instances, {1} memory per instance, {2} disk allocated...",
            application.RunningInstances, application.Resources.Memory, application.Resources.Disk);
        Console.Write(" hosted at ");
        foreach (var uri in application.Uris)
        {
            Console.Write(uri + " ");
        }
        Console.WriteLine(".");
    }
}

That’ll get the call running on a parallel thread and when it is wrapped up it returns the data.

The User Interface Interaction Issue

This is all fine and dandy for the command console. But if you want to give control back to the UI thread in a UI application and make sure that the background thread can actually update a control when fired off, do the same thing as I’ve discussed here except set the control up to invoke the dispatcher, so that the “threads don’t cross” when trying to return information to a control that needs updated. In order to do this take the control that needs updated and set the Dispatcher Invoke method as shown below.

private void Write(string updateText)
{
    UpdatingTextBlock.Dispatcher.Invoke(
    System.Windows.Threading.DispatcherPriority.Normal,
        new Action(delegate
    {
        UpdatingTextBlock.Text += updateText;
    }
    ));
}

For more on the Iron Foundry Project and the library I’ve used here, check out the Iron Foundry Blog & Site. For more information on Thor and to follow or get involved check out the Thor Project Site (Hosted with Cloud Foundry at Tier 3!).

All of this code is available in my Github Project “Remembering” (https://github.com/Adron/Remembering). Feel free to fork it, share it, or send me corrections or pull requests.

In My Dreariness, I Installed Cloud Foundry and extended it with Iron Foundry

Morning!

I put together a video late last night based on setting up a Cloud Foundry instance with Iron Foundry extensions. It’s a little long, since I’ve mostly *not edited* it. The instructions have been available for a while but I thought, why not make a video and publish it. In the very near future we’re going to start laying out specifics on how to install individual elements of Cloud Foundry and Iron Foundry. We’ll also be providing deeper detail of exactly what these parts do, what is needed, and how to scale a non-micro environment.

Cloud Foundry + Iron Foundry from Adron Hall on Vimeo.

Cloud Foundry + Iron Foundry for a full framework list of options. This video shows the steps to get a Cloud Foundry Micro instance up and running, then get an Iron Foundry instance up and running, connect them and then view a full list of frameworks.

NOTE: This video is barely edited, so it is a little long because everything is mostly done in real time.

This shows how easy this really is. If you’re using any other PaaS software, especially if you’re trying to setup local development environments, I’d love to hear about your processes and also any ideas on how we could make this process easier. In the near future I want to put together a video showing Stackato too. They have a nice offering to get a similar environment setup with more UI elements, making it a bit easier to keep track of all the applications and such.

How Software Should Get Done, Continually Delivering!

Tonight I spoke at the PADNUG Meetup in Hillsboro, a suburb of Portland, Oregon. The ladies and gentlemen of PADNUG are a great crew, so I actually go out of my way to the suburbs to speak there. Tonight was an exceptionally good experience with a great talk, lots of back and forth between everyone there and great conversations continued late into the night at the local suburban watering hole. All in all a good topic of conversation and one that needs brought to more teams.

Continuous Delivery

How does this fold into my work on PaaS (Platform as a Service) and IaaS (Infrastructure as a Service)? Easy, with the cloud computing capabilities of PaaS and IaaS it makes continuous delivery a no brainer. At least 50% of the effort to get continuous delivery setup is already done with these technologies. Over the next few weeks I’ll be writing a lot about these technologies and the enablement of continuous delivery through these technologies. Just as important as the technology, I’ll also be talking about the processes, ideals and lean thinking that have birthed this tech.

In my presentation I covered a lot of these ideas and efforts. For now, here’s my slide deck with all the information to contact me. If you’d like me to pop into your town and present on these topics, just let me know and we’ll see about me getting onsite.

Coming up on the 20th I’ll be presenting some of this material plus a very hands on demo at the Software Craftsman’s meeting is Seattle titled “Coding in the Cloud, Kick Ass Continuously“. So if you live in the Seattle or are just in the area, drop in!

The Bad, The Ugly and The Good Bits :: Sexism, VMworld 2012 & Smart Cool People

The Divide in Technologists…

Sexism & Those That are Building Tech

There seems to be a pretty distinctive divide in the technology industry today. There are the young, open minded, devop oriented, free-thinking individuals and then there are the old guard of IT. This later group still brings the “booth babes” and finds an incessant need to assume all women aren’t technologists (which I might add is utter bullshit). This is when I’m going to rant for a minute.

[rant=on]

Ok guys, pull your heads out of your collective asses. I’ve spoke to 11 ladies that are hard core technologists, that would take your old guard IT and replace your sorry ass with a shell script plus some cloud computing and leave you to the dogs. They’re programmers, devops pros, hackers and entrepreneurs  Simply, they kick as much ass as anybody, so shove off.

This however brings up the question, “Why the hell does the conference still perpetuate this bullshit with booth babes and mindless dribble?”  Seriously, can we focus on the technology, the reason we’re here? To learn, to build, to maintain, create and extend our services and capabilities that we work with? Can we not have a mass of “talent” come and stand around just so aging IT guys can ogle their breasts with roaming eyeballs?

Don’t get me wrong, beautiful people are great, and when done tastefully things can be fun. One of the ladies I work with mentioned it’d be great if Thor showed up and hung out at the conference (cuz ya see, we have a product called Thor, and this data company had Data attend. (Brent Spiner)).

I could go on. Simply put, companies and conference organizers need to own up and get with the times. For those of us that are a little evolved past nuckle dragging we should stand up to this time of nonsense. There’s a reason we’re at a conference and it damn well shouldn’t be to devalue people as objects and ogle various body parts.

[rant=off]

Ok, back on track with the successful bits. There were, after all a lot of successful bits and the sexism is a small, yet very sad and noticeable part of the event. The other good news is the amount of women’s groups that are getting together these days to code  (and I also find it unfortunate that to create a positive environment, women usually have to entirely disengage with men, and it is generally men’s fault)  Yup I said code. Rails Girls, Code n’ Splode and many others. So if you’re reading this and are female, check these groups out and get hacking & devoping.

The Big Move, PaaS is Starting to Rock!

VMware made serveral announcements around Cloud Foundry, which is pretty huge. The momentum is still growing, the community is still growing, and the energy is contagious. There’s been some egregious accusations and suggestions that the Cloud Foundry ecosystem is going to collapse. This is, however one of the more absurd notions I’ve heard in months. This definitely falls into the category of FUD flinging with no concrete notion. Lucas (@cardmagic) from AppFog lays out a bit of reality though, and the 20k people at VMworld and the thousands using and hundreds coding to Cloud Foundry give a resounding shout of,

“HELL NO”

Cloud Foundry is not collapsing, VMware is not taking an unfair advantage, and they’re in a position to win along with all the rest of the advocates of PaaS and open source. The thing is, things can indeed be win-win. They don’t have to be win-lose and the later thinking is negative to the industry and counter to the reality of open source.

Either way, toss any ideas this is going away out of your minds. I know most of you already have.

Smart People, Networking and a Few Rounds

The greatest thing about these conferences is the ability to network and meet face to face with hundreds of people that I do business with everyday. These range from people I hack code with, to people I help implement Cloud Foundry or people that simply are involved in the community too. To me, the most valuable ROI is the networking at a conference. Just to throw a few out there, I got to catch up with…

  • Dave McCrory @mccrory – This guy is awesome, if you get to work with him you’re a lucky soul. He’s heading up WMG as SVP of Platform Engineering now to get some cool things built and build out a solid team, which I look forward to hearing about!
  • Andy Piper @andypiper – Andy is VMware’s Cloud Foundry Dev Advocate of Great Britain. I got to meet Andy a while back and got to team up with him and many others to catch up on Cloud Foundry, see were things are heading, talk through some ideas and generally cause mischief around San Francisco.
  • James Watters @wattersjames – I always love running into this guy. Top notch smart, snarky and always ready to go through who’s who and who’s doing what in cloud technology. He’s the VMware Director of Ecosystem for Cloud Foundry and they’re damn lucky to have this guy!
  • Brian McClain @brianmmcclain – When I was originally writing Brian’s name out, I mispelled it “brain” and almost just left it this way. Brian is all over the Cloud Foundry realm working with BOSH, pushing forward with Cloud Foundry in an enterprise environment, and generally always ready to dive into the tech heavy deep end. Always great chatting with Brian about the details and whatever random code adventures come up!

…and there were dozens of others I got to catch up with. Mark Kropf, Ken Robertson, Daine Mueller, Jeremy Voorhis and almost got to catch up with Derek Collison too. Well, there’s always the next trip to San Francisco! If you’re into the Cloud Foundry space, into PaaS technologies, or just interested definitely reach out, follow these guys on twitter, and make an effort to meet them.

VMware’s VMworld Summary

VMworld was good times, for sure. There were the hiccups as I pointed out, but overall a great experience, the organizers did a solid job (still would help if they could crack down on the companies that perpetuate sexism and BS over content on the booth/show floor, but otherwise, kudos on a job well done). It was great catching up with the brain power in the industry and finally meeting many people I’d been wanting to. I even wrote more than a few lines of code and tested out a few deployment ideas based on the conversations. This, in the end, is exactly what the conference is truly about.  Cheers!