Cloud Speak

Deploycon – “Platform Services and Developer Expectations” **

I’m on a panel at Deploycon titled “Platform Services and Developer Expectations” and this leads right back around to that. This SPOF issue is concerning to me as PaaS Providers talk up the offerings more and more with little light actually shone on this issue. In some ways each is moving away form their respective SPOFs, but overall they’re all pretty prevalent throughout. For security, each has a non-distributed database, which technically needs backed up still – no clear replication or other mechanisms setup to ensure data integrity in a failure situation. Of course, the huge saving grace with a PaaS, is that if the overall system goes down or a SPOF blows up, all the existing deployed applications will generally continue to run. Unless of course the routing and networking are also SPOF. This is the largest glaring concern with PaaS Systems that I see today.

One of the other things about PaaS that has always led to a ton of questions is “what about my PostGresql/mysql/Riak/mongodb/database thing and how do I do X, Y, Z with it to ensure scalability in my PaaS.” In almost every case it ends with a simple and unfortunate answer, ”…when it comes to data, a PaaS doesn’t really do a damn thing for ya…” This is obviously not very helpful. The entire reason to put a PaaS into place is to simplify life, the sad fact that it barely does a thing for the data tier isn’t very helpful.

Now, hold on a second before you start screaming at me about “but a PaaS does X, Y and Z and isn’t even supposed to touch that aspect of things…” let me elaborate a bit more. The panel at Deploycon states “…Developer Expectations” and when things are getting simplified in the way a PaaS does, developers assume that if it does all this fancy magic for an application it ought to simplify the data side of things too! Right? Well no, and it isn’t going to for the foreseeable future. But no matter what, it doesn’t change the fact that developers often have that expectation.

Now, I could write at length about all the reasons that PaaS doesn’t really do anything for the data tier. I could wax poetic about how a distributed database (re: Riak, Cassandra, etc) just doesn’t lend itself to a cookie cutter approach to deployment under a PaaS or an RDBMS has umpteen different configurations for stability, scaling, hot swappable services, and other such complexities around the data tier. But instead I’m going to skip all, maybe cover some of those things another day, and jump right into some of the things that are actually moving forward to fill this gap.

BOSH, Cloud Foundry, OpenShift & fixing the data tier…

The most obvious reason there isn’t a simple turn key solution to the data side of things with a PaaS ecosystem is that data is complex and extremely diverse. There’s distributed key/value stores (Riak, Cassandra), there’s sort of kind of distributed databases (Mongo), graph databases (Neo4j), the age old RDBMS (DB2, SQL Server, Oracle’s Stuff, etc) and the million solutions around that, there’s key/value in memory styled databases that are insanely fast, like Redis. Expanding just slightly you have software that works around these systems such as Hadoop & Riak CS & the list goes on. All of it focused on the data tier and maintaining one, two or some form of the three points around CAP Theorem (http://en.wikipedia.org/wiki/CAP_theorem), atomicity and other key capbilities.

All of the PaaS Systems, including public and private often have some sort of plug-in style architectures for data. Whether it is Apprenda which is closed to community and closed source or an ongoing open to community PaaS like OpenShift or Cloud Foundry, things still fall almost entirely to the developers or database team to build an architecture around the data. When looking at solutions to simplify data in PaaS Systems the closed source solutions we have no idea what they’re up to in this regard. The one’s that are open source or in large part public and involved in the community PaaSes, like EngineYard, Heroku, Cloudbees and others we can really see the directions and efforts around creating real PaaS style solutions to the data tier problem.

BOSH, Vagrant, etc… One of the best solutions I’ve seen so far is the ability of Bosh, which was created by the Cloud Foundry team while at VMware, to spool up an environment that includes such things as a Riak Cluster (or other cluster). Currently Brian McClain & Dr Nic have worked to put together such Bosh + Vagrant scripts & get things rolling. I myself will be spending some considerable time on just that. But beyond that this is a good start in enabling data tier back ends.

How to close the gap, between absurdly simple application deployment and still arduous and difficult data tier deployment? For the next several years I think we’ll have cumbersome deployment practices around the data tier. There won’t be anything as elegantly simple as Cloud Foundry’s single line deployment or AppFog’s one click deployment of a web application. The best we can do at this time, is to streamline around pieces and architectures, and at least get them into a kind of simple 3 step deployment.

Please drop a comment or two on how you think we might simplify the data side of the PaaS toolchain. Also drop a few tweets in the twitterverse too, I’m sure that’ll be exploding as usual. I’m @adron, ping me.

Cheers, happy data architecting.

** the Deploycon panel will be at 4:30pm in Santa Clara on April 2nd. Come check it out.

Riak is… A Whole Big List of Things

January 11, 2013

Cloud Speak, Distributed Things, Riak

2 Comments

What is Riak? Who builds it? Who maintains it? Can I download it? How does it work? What are the features?

Here’s the start of answers to these questions and more.

First, the basic high level description:

“Riak is an open source, highly scalable, fault-tolerant distributed database.”

That’s the first line you’ll read when checking out the product via the Basho product link. It provides good information, but here I’m going to add more to the definition without the need to dig around yourself. Maybe I can save you some time & provide some links directly to solid information in the docs. Kind of a “Cliff Notes” of Riak. Let’s take this feature by feature which will in turn get us to a definitive definition of what exactly Riak is.

Riak is Open Source.

Riak is built and contributed to by the community, with Basho being the steward and an active member that extends, builds and provides support for additional products. The avenues to reach the Riak Open Source Community members is pretty straight forward, following known avenues of communication. Hit us up on the email list, especially feel free to contribute & ask questions via the Github Basho organization, there is the Basho Riak Blog, the weekly recap and jump into the IRC chat room #riak on freenode. Oh, and there’s a twitter feed @basho.

So what exactly does this get you, when you become a user or contributor of Riak? The entire community is behind you, will help you get started using Riak and provide help whenever you run into problems. If you want SLAs or 24 hour support Basho can provide this for you. But for bugs, issues, queries, searching and all sorts of other related development questions there is the community. An open source community like this is passionate, which means you’ll have support like no closed source company will ever provide you, and absolutely no closed source product’s community will provide you. We’re talking about a different level of interest, passion and levels of personal involvement.

Riak is a key value based database store.

Riak is a key value store. What exactly is a key value store? It’s pretty simple and you’re probably already familiar with what a key value store is. A key value is made up of two pieces of data, the first is the identifier for the second element within the data structure. This gives a system or developer using key value storage a schema-less way of working with data.

Riak is designed for highly distributed environments.

This type of distributed isn’t the “we put one database over here and one database over here and you gotta figure out how they work together” type of distribution. So this isn’t some of that oddball pretend stuff Oracle keeps hoisting on people. This is the honest to goodness distribution of the sort, when one node goes down you don’t blink, you don’t stop eating dinner, you don’t sweat it. You just continue onward with life knowing full well that you’ll just spool up another node when you need to.

Riak is master-less, with no single point of failure.

This is one of self explanatory features. But what does a master-less system provide us? One thing is no single point of failure. Being that all nodes can act autonomously to work around the loss of one or more nodes it also helps add to the high availability of the system.

Riak is fault tolerant, like a disk drive you wish was real.

Ever have a backup disk drive? What? You don’t have one of those? Ugh. Ok, so imagine you had a backup disk drive that had an unfortunately high failure rate. Well, why, because you know, they have an oddly high failure rate. If you do backups like good practices dictate, eventually you’ll end up with some dead drives.

RAID, both software and hardware, are built specifically to deal with this type of failure. With a distributed system like Riak, it bumps the level of abstraction above software or hardware RAID, enabling another level of even greater fault tolerance. Not to remove the relevance of RAID capabilities, but with a multi-node system like Riak, you can easily remove nodes and swap them out as needed, keeping costs down by using simple drives in simple machines. If you want to, you could indeed get higher I/O machines and faster drives, but it isn’t necessary to insure fault tolerance in a Riak Database System.

Riak scales, with hot swappable nodes enabling zero downtime.

The ability to commit hot swappable changes while in the midst of operating starts at a very low level for Riak. The language used to build Riak, Erlang has the ability to change pieces of an application system in realtime built into the precepts of the language. This provides, at the core, the inherent capability to change out systems, and by proxy of architectural design, the ability for nodes in Riak to be changed out simply by removing them from a cluster ring. Once that is done it is just as simple to add another node or nodes back into the cluster ring, enabling a number of additional practices around upgrades, hot swaps for failures, or even version changes.

Riak can be used as a building block for distributed (aka cloud) infrastructure.

The concepts and contractual components that Riak Database is built on are available for use via the Riak Core Project. If you’re looking into starting a project around distributed systems this is a great place to get start. Also be sure to do a general web engine (re: google) search for “riak core” and you’ll find lots of material around the project, and projects people have started with the project as a base. I’m currently in the process of putting together one of these projects myself.

Riak is eventually consistent.

The term eventually consistent is becoming more and more common place. Riak is one of the many systems, that inherently often apply to distributed systems, that use the concepts of eventual consistency. The idea, is that even though all nodes may not immediately receive a new piece of data, or updated piece of data, they eventually will receive that update and by synchronized with the cluster ring of nodes. This goes back to the equality of nodes and removal of the master-less concepts, providing the availability and other capabilities, with some trade off in the synchronization of data through eventual consistency.

In Summary

That’s round one for the many features of Riak. I’ll be adding more in the future, but for now this is a good starting point in knowing about and knowing what Riak is, what it can be used for, and how it might help you extend, maintain or invent the next great piece of technology.

Adam & Krishan Got Me Motivated Today… to toss the trash conversations

September 6, 2012

Cloud Foundry, Cloud Speak, Ideas, Iron Foundry, Platform (as a Service), Presentations and Speaking, Thor Project

1 Comment

I was speaking with Krishan Subramanian (@krishnan) and Adam Seligman (@adamse) today. I love talking to these guys. They’re both smart, intelligent and upbeat guys. They see the positive things we’re all working toward and accomplishing in the technology space, specifically around PaaS, Cloud Computing and around the cultural implications of stronger technology communities, involvement of individuals. We all can see the positives, of how the industry is moving forward so that corporations aren’t the only enablers that are juxtaposed against developers or consumers but instead act to serve consumers based on the progress that individuals make themselves. There’s so much to do and so much progress to be made, the venders can simply follow the community and step up to provide points of leadership.

Absolutely great talking with these guys…

On that topic, what is it that we discussed that has me so motivated? Well there’s a few things that I’m done with and I’m going to make every effort to just throw away the trash. Here’s a few of these things that we discussed and I challenge everybody out there, drop the trash talk and let’s move forward because there is a LOT of awesome things to accomplish. Here’s the two things I’m just dropping… cold. No reason to discuss them anymore.

Toss the language and framework religious wars. It is far simpler than it is sometimes perceived. We have a polyglot industry now where we can easily use the right tool for the job, the right framework, or the language that handles our particular domain the best. There is literally no reason to argue about this anymore. Of course we can talk semantics, debate best use cases, and of course we’ll talk accomplishments and what various things do well. That’s exactly what the focus should be on, not the harping on my X is better than your Y nonsense.
The culture war is basically over. Sure there are the hold outs that haven’t gotten a clue yet. But it’s an open source world at this point. Even the dreaded and horrible Oracle has generally conceded this and is frantically waving its marketing arms around trying to get attention. But at the core, mysql, java and the other things that they’ve purchased they’re keeping alive. They’re active participants in the community now, albeit in a somewhat strange way. Considering that even Oracle, Microsoft, Apple and so many others contribute back to the open source community in massive ways, that war can be considered won. Victory, the community and every individual in that community!
Lockin is basically dead. The technological reasons to lock in are gone, seriously. There’s some issues around data gravity that are to be overcome, but that’s where a solid architecture (see below) comes in. Anything you need can be contributed to and derived from the development community. Get involved and figure out how technology can be a major piece of your business in a positive way. If you design something poorly, lock in becomes a huge issue. Use the rights tools, don’t get into binding contracts, because in the polyglot world we’re in now there’s no reason to be permanently locked in to anything. Be flexible, be where you need to be, and make those decisions based on the community, your support systems, and your business partners. Don’t tie yourself to vendors unless there is mutual reasons to do exactly that. Lock in is a dead conversation, just don’t, time to move on.

So what are the key conversations today?

Ecosystem Architecture - If you’re deploying to AWS, Heroku, Tier 3, AppFog or Windows Azure it all boils down to something very specific that will make or break you. Your architecture. This is where the real value add in the cloud & respective systems are, but there are many discussions and many elements of the technology to understand. This is a fundamentally key conversation topic in the industry today. Pick this one up and drop the other trash.
Movement & Data Gravity – How do you access your data, how do you store it, where and how do you derive insight from that data? This is one of the topics that came up in our discusssion and it is huge. The entire computer industry basically exists for the reason of insight. What should we eat today, how do I shift my investments, how is my development team doing, what’s the status of my house being built, where is my family today and can I contact them! All of these things are insights we derive from computer systems. These are the fundamental core reason that computers exist. As an industry we’re finally getting to a point were we can get some pretty solid insightful, intelligent and useful information from our systems. The conversation however continues, there is so much more we can still achieve. So again, drop the wasteful convo and jump on board the conversations about data, information and insights!
Community Involvement – I’ve left the key topic for last. This is huge, companies have to be involved today. Companies aren’t dictating progress but instead the community is leading as it should. The community is providing a path for companies to follow or lead, but the community, the individuals are the ones that are seen and known to be innovating. This is so simple it’s wild that it is only now becoming a known reality – companies don’t innovate, people do. Companies don’t involve, people do. Individuals are the drivers of companies, the drivers of Governments, they’re the ones driving innovation and progress. The focus should now and should have always been on the individuals and what they’re working toward to accomplish. So get involved, get the companies involved as a whole and keep the semantic ideal of individuals and the progress they can make core to the way you think of communities. The idea of the “company” innovating is silly, let’s talk and build community with the people that are working around and innovating with these technologies.

Of course there are more, I’d love to hear your take on what the conversations of today should be about. What do we need to resolve? How do we improve our lives, our work and the efforts we’re working toward on a day to day basis?

How Software Should Get Done, Continually Delivering!

September 5, 2012

Cloud Foundry, Cloud Speak, Iron Foundry, Platform (as a Service), Presentations and Speaking

Continuous Delivery

How does this fold into my work on PaaS (Platform as a Service) and IaaS (Infrastructure as a Service)? Easy, with the cloud computing capabilities of PaaS and IaaS it makes continuous delivery a no brainer. At least 50% of the effort to get continuous delivery setup is already done with these technologies. Over the next few weeks I’ll be writing a lot about these technologies and the enablement of continuous delivery through these technologies. Just as important as the technology, I’ll also be talking about the processes, ideals and lean thinking that have birthed this tech.

In my presentation I covered a lot of these ideas and efforts. For now, here’s my slide deck with all the information to contact me. If you’d like me to pop into your town and present on these topics, just let me know and we’ll see about me getting onsite.

Coming up on the 20th I’ll be presenting some of this material plus a very hands on demo at the Software Craftsman’s meeting is Seattle titled “Coding in the Cloud, Kick Ass Continuously“. So if you live in the Seattle or are just in the area, drop in!

The Bad, The Ugly and The Good Bits :: Sexism, VMworld 2012 & Smart Cool People

August 30, 2012

Cloud Foundry, Cloud Speak, Coding Adventures, Platform (as a Service), Rants

4 Comments

The Divide in Technologists…

Sexism & Those That are Building Tech

There seems to be a pretty distinctive divide in the technology industry today. There are the young, open minded, devop oriented, free-thinking individuals and then there are the old guard of IT. This later group still brings the “booth babes” and finds an incessant need to assume all women aren’t technologists (which I might add is utter bullshit). This is when I’m going to rant for a minute.

[rant=on]

Ok guys, pull your heads out of your collective asses. I’ve spoke to 11 ladies that are hard core technologists, that would take your old guard IT and replace your sorry ass with a shell script plus some cloud computing and leave you to the dogs. They’re programmers, devops pros, hackers and entrepreneurs Simply, they kick as much ass as anybody, so shove off.

This however brings up the question, “Why the hell does the conference still perpetuate this bullshit with booth babes and mindless dribble?” Seriously, can we focus on the technology, the reason we’re here? To learn, to build, to maintain, create and extend our services and capabilities that we work with? Can we not have a mass of “talent” come and stand around just so aging IT guys can ogle their breasts with roaming eyeballs?

Don’t get me wrong, beautiful people are great, and when done tastefully things can be fun. One of the ladies I work with mentioned it’d be great if Thor showed up and hung out at the conference (cuz ya see, we have a product called Thor, and this data company had Data attend. (Brent Spiner)).

I could go on. Simply put, companies and conference organizers need to own up and get with the times. For those of us that are a little evolved past nuckle dragging we should stand up to this time of nonsense. There’s a reason we’re at a conference and it damn well shouldn’t be to devalue people as objects and ogle various body parts.

[rant=off]

Ok, back on track with the successful bits. There were, after all a lot of successful bits and the sexism is a small, yet very sad and noticeable part of the event. The other good news is the amount of women’s groups that are getting together these days to code (and I also find it unfortunate that to create a positive environment, women usually have to entirely disengage with men, and it is generally men’s fault) Yup I said code. Rails Girls, Code n’ Splode and many others. So if you’re reading this and are female, check these groups out and get hacking & devoping.

The Big Move, PaaS is Starting to Rock!

VMware made serveral announcements around Cloud Foundry, which is pretty huge. The momentum is still growing, the community is still growing, and the energy is contagious. There’s been some egregious accusations and suggestions that the Cloud Foundry ecosystem is going to collapse. This is, however one of the more absurd notions I’ve heard in months. This definitely falls into the category of FUD flinging with no concrete notion. Lucas (@cardmagic) from AppFog lays out a bit of reality though, and the 20k people at VMworld and the thousands using and hundreds coding to Cloud Foundry give a resounding shout of,

“HELL NO”

Cloud Foundry is not collapsing, VMware is not taking an unfair advantage, and they’re in a position to win along with all the rest of the advocates of PaaS and open source. The thing is, things can indeed be win-win. They don’t have to be win-lose and the later thinking is negative to the industry and counter to the reality of open source.

Either way, toss any ideas this is going away out of your minds. I know most of you already have.

Smart People, Networking and a Few Rounds

The greatest thing about these conferences is the ability to network and meet face to face with hundreds of people that I do business with everyday. These range from people I hack code with, to people I help implement Cloud Foundry or people that simply are involved in the community too. To me, the most valuable ROI is the networking at a conference. Just to throw a few out there, I got to catch up with…

Dave McCrory @mccrory – This guy is awesome, if you get to work with him you’re a lucky soul. He’s heading up WMG as SVP of Platform Engineering now to get some cool things built and build out a solid team, which I look forward to hearing about!
Andy Piper @andypiper – Andy is VMware’s Cloud Foundry Dev Advocate of Great Britain. I got to meet Andy a while back and got to team up with him and many others to catch up on Cloud Foundry, see were things are heading, talk through some ideas and generally cause mischief around San Francisco.
James Watters @wattersjames – I always love running into this guy. Top notch smart, snarky and always ready to go through who’s who and who’s doing what in cloud technology. He’s the VMware Director of Ecosystem for Cloud Foundry and they’re damn lucky to have this guy!
Brian McClain @brianmmcclain – When I was originally writing Brian’s name out, I mispelled it “brain” and almost just left it this way. Brian is all over the Cloud Foundry realm working with BOSH, pushing forward with Cloud Foundry in an enterprise environment, and generally always ready to dive into the tech heavy deep end. Always great chatting with Brian about the details and whatever random code adventures come up!

…and there were dozens of others I got to catch up with. Mark Kropf, Ken Robertson, Daine Mueller, Jeremy Voorhis and almost got to catch up with Derek Collison too. Well, there’s always the next trip to San Francisco! If you’re into the Cloud Foundry space, into PaaS technologies, or just interested definitely reach out, follow these guys on twitter, and make an effort to meet them.

VMware’s VMworld Summary

VMworld was good times, for sure. There were the hiccups as I pointed out, but overall a great experience, the organizers did a solid job (still would help if they could crack down on the companies that perpetuate sexism and BS over content on the booth/show floor, but otherwise, kudos on a job well done). It was great catching up with the brain power in the industry and finally meeting many people I’d been wanting to. I even wrote more than a few lines of code and tested out a few deployment ideas based on the conversations. This, in the end, is exactly what the conference is truly about. Cheers!

M	T	W	T	F	S	S
« May
					1	2
3	4	5	6	7	8	9
10	11	12	13	14	15	16
17	18	19	20	21	22	23
24	25	26	27	28	29	30

—Composite Code

Snippets, software architecture, lean, agile, management, and leadership bits.

Archive

Cloud Speak

Deploycon, PaaS & the pending data tier gravity fallout…

Deploycon – “Platform Services and Developer Expectations” **

BOSH, Cloud Foundry, OpenShift & fixing the data tier…

Riak is… A Whole Big List of Things

Riak is Open Source.

Riak is a key value based database store.

Riak is designed for highly distributed environments.

Riak is master-less, with no single point of failure.

Riak is fault tolerant, like a disk drive you wish was real.

Riak scales, with hot swappable nodes enabling zero downtime.

Riak can be used as a building block for distributed (aka cloud) infrastructure.

Riak is eventually consistent.

Adam & Krishan Got Me Motivated Today… to toss the trash conversations

Follow “Composite Code”

Deploycon – “Platform Services and Developer Expectations” **

BOSH, Cloud Foundry, OpenShift & fixing the data tier…

Share this:

Like this:

Riak is Open Source.

Riak is a key value based database store.

Riak is designed for highly distributed environments.

Riak is master-less, with no single point of failure.

Riak is fault tolerant, like a disk drive you wish was real.

Riak scales, with hot swappable nodes enabling zero downtime.

Riak can be used as a building block for distributed (aka cloud) infrastructure.

Riak is eventually consistent.

Share this:

Like this:

Share this:

Like this:

Continuous Delivery

Share this:

Like this:

The Divide in Technologists…

The Big Move, PaaS is Starting to Rock!

“HELL NO”

Smart People, Networking and a Few Rounds

VMware’s VMworld Summary

Share this:

Like this:

Follow “Composite Code”