Polyglot Conference in Vancouver, Using Riak to Analyze, Store Data and Survive the Zombie Apocalypse

May 20, 2013

Community, Conferences, Databases, Distributed Things, Riak

First things first - If you haven’t registered for the Polyglot Conference yet, get registered ASAP as it could sell out!

I have a workshop coming up on Friday of this week at the Vancouver BC Polyglot Conf. In addition I’ll be attending the conference itself and looking to have a lot of conversations and open spaces sessions around distributed systems, technologies, platform as a service, other *aaS and generally anything for modern, distributed, scalable architecture and systems. Hopefully I’ll get to converse with all of you about as many topics as possible - it’s always a great time to dig into these topics at conferences like these. It’s where the real next steps in technology come from!

In the coming days before and after the workshop I’ll be blogging material I have put together for the workshop. Previously I’ve put together one blog entry on backing up distributed systems ala the Riak variety. Check it out “Backup Riak – Learning About Distributed Databases :: Issue 001“.

The Workshop, for those of you interested in surviving a Zombie Apocalypse (or just learning about Riak) can sign up still, there’s a few spaces left! Below is a the basic outline of what we’ll be touching upon. There’s a LOT more topics beyond this, but we’ll save those for open space sessions. If there’s anything specific you’d like me to cover, feel free to throw a comment on the blog and I can easily add 1-2 topics of interest if attendees would like!

Introducing Riak, a database designed to survive the Zombie Plague. Riak Architecture & 5 Minute History of Riak & Zombies.

Architecture deep dive:

Consistent Hashing, managing to track changes when your kill zone is littered with Zombies.
Intelligent Replication, managing your data against each of your bunkers.
Data Re-distribution, sometimes they overtake a bunker, how your data is re-distributed.
Short Erlang Introduction, a language fit for managing post-civil society.
Getting Erlang

Installing Riak on…

Ubuntu, RHEL & the Linux Variety.
OS-X, the only user centered computers to survive the apocolypse.
From source, maintained and modernized for humanities survival.
Upgrading Riak, when a bunker is retaken from the zomibes, it’s time to update your Riak.
Setting up

Devrel – A developer’s machine w/ Riak – how to manage without zombie bunkers.

5 nodes, a basic cluster
Operating Riak
Starting, stopping, and restarting
Scaling up & out
Managing uptime & data integrity
Accessing & writing data

Polyglot client libraries

JavaScript/Node.js & Erlang for the zombie curing mad scientists.
C#/.NET & Java for the zombie creating corporations.
Others, for those trying to just survive the zombie apocolypse.

Cheers!

Node PDX – Introducing Adam Baldwin, James Halliday, Ryan Jarvinen, Mike McNeil and Horse JS

May 15, 2013

Awesome Coders, Community, Conferences

Leave a Comment

This is it, last string of introductions. Hope you’re registered.

Adam Baldwin is presenting…

Introducing NodeSecurity.io

Adam Baldwin

Adam Baldwin is a web app hacker, team lead at ^Lift Security and the CSO for &yet. Adam has presented at various security & dev conferences in the past including, DEFCON, Djangocon, Toorcamp and RealtimeConf.

The node.js community is growing at an amazing rate. At the time of writing there was 27,757 modules publised on npm. Have you ever stopped to think just what you are putting into your project when you npm install somebody else’s module? Do you trust that code? This is an insane project to find out the answer to that question.

This talk will introduce the nodesecurity.io project, it’s goals, current results in hopes of inspiring involvement and receiving feedback directly from the node community!

James Halliday is presenting…

beep boop

James

Oh hello. I write too much code. I co-founded browserling. Here are some pretty pictures.

Unix philosopher and methodological reductionist etc.

Learn how to make computer sounds in node and the browser with the same api.

Using just a single function that takes a parameter t, time in seconds, and returns an amplitude between -1 and 1, inclusive, you can create music!

You can use this basic approach to write songs and synthesizers. In javascript. Yay!

Ryan Jarvinen is presenting…

Clustering Node.js on OpenShift

Ryan

Ryan Jarvinen is an Open Platform Advocate working with RedHat’s OpenShift team. He lives in Oakland, California and is passionate about open source, open standards, open government, and digital rights. You can reach him as ‘ryanj’ on twitter, github, and IRC.

Learn how to automate builds, deployment tasks, and application scaling as we use OpenShift’s platform architecture on-demand to build your own git-based release pipeline, including: development, testing, staging, and cloud-scaling production environments for node.js.

Slides:

Posts:

An adaptation of this talk was presented recently at HTML5DevConf in SF - http://html5devconf.com/sessions.html#r_jarvinen

Intro to Sails.js

Mike McNeil is presenting…

Mike

Mike autobiogrophies himself as, “I’m Mike, a developreneur based out of Austin, Texas and connoisseur of fine code. I’m also the creator of Sails.js, the open-source BaaS framework which allows front-end developers to build robust, scalable APIs using only JavaScript.

My first startup was in social television, where I saw the need for more efficient, easy-to-use solutions for realtime social features. Because of that, I got involved in Node.js early on, and after building a few early apps, recognized the need for an MVC solution to normalize patterns. Early last year, I founded Balderdash, a UX-focused mobile and web studio, which has given me an excellent opportunity to build out and utilize Sails.js in production.“

Sails.js makes it easy to build custom, enterprise-grade Node.js apps. It is designed to resemble the MVC architecture from frameworks like Ruby on Rails, but with support for the more modern, data-oriented style of web app development. It’s especially good for building realtime features like chat.

Sails empowers UX and design teams to build hi-fi prototypes in no time without waiting for the back-end to be finished. This means focusing more resources on the user experience, which means better products. One Sails.js project at a time, companies move their legacy architecture over to a simpler, more efficient Node.js cloud. Each new client-side code base is more maintainable, since it’s built using the universal language of the internet: a RESTful JSON API.

Chris Dickinson is presenting…

Implementing Git in JavaScript & the Browser: A Case Study

Chris

Chris describes himself as “I make silly things with JavaScript: I particularly love bit-twiddling and WebGL-based projects. I live in Portland OR and work at Urban Airship as a JavaScript engineer.”

Git is one of my favorite things to hack on. It’s long been my goal to get a working (workable?) implementation of git running in pure JS, in the browser. My first attempt two years ago failed; and for a long time I’ve let the thought bounce around in the back of my head.

Spurred on by the recent interest in js-git, I recently restarted the journey towards an in-browser git, in order to help creationix deliver the best possible js-git. Newly armed with browserify and the small-module ethos, I’ve come much closer to a working git in browser and Node, and in the process have really put browserify and its shims through their paces.”

This talk will be comprised of:

A quick intro to the git object model and transport protocol
How browserify and the small module ethos have enabled great successes in the project.
Difficulties encountered in the process, both with Node.JS itself and with browserify, and how I’ve worked through them.
How I’ve diagnosed and worked through various performance issues.
Where is this project going?

Horse JS is presenting…

JavaScript, This is Confusing

….Horse JS Tweets, nuff’ said. Horsing around…

Are you signed up? BUY YOUR TICKET FOR NODE PDX HERE

Want to learn more? http://nodepdx.org/

Want to know the dates? http://nodepdx.org/

Want to know who else is speaking? Stay tuned here or go check out http://nodepdx.org/!

Node PDX – Introducing Zach Bobb, Paul Jungwirth, Forrest Norvell and Charlie Key

May 13, 2013

Awesome Coders, Community, Conferences

Leave a Comment

…and the fifth iteration of Node PDX Introductions!

Zach Bobb is presenting…

Building a Computer In Your Browser

Zach the TriMet Ticket Man!

Zach is a mobile engineer with GlobeSherpa working hard to bring you the app that will let you buy TriMet tickets on your phone.

Want to learn how computers work under the hood while learning Web technologies? Come learn about the Von Neumann 51, a series of tutorials that will take you through building a computer from the ground up in JavaScript, HTML5, and CSS3.

The talk will start by laying out some of the motivation behind the project and then demonstrate some of the components people will build through the tutorials: logic gates, adder circuits, displays, etc. The end of the talk will demo a prototype of the vN51.

Some of the technologies in this talk are: D3 | A JavaScript library for dynamic data-driven SVG graphics HTML5 Canvas | Pixel manipulation for display output

Paul Jungwirth is presenting…

Handling Errors with Cluster and Domains

Paul

Paul will jump into,

“Node’s callback pattern makes error handling difficult: throwing an exception kills the entire node process, terminating all current requests, and every callback initiates a new stack, so stacktraces are terse and don’t indicate how you got where you died. You can solve these problems using some newer features of Node called clusters and domains. This talk with explore using these tools for better error handling.“

Forrest Norvell is presenting…

Do as I say, not as I do: Node in the real world

Forrest

Forrest is a software engineer who has spent the better part of the last two years working with Node full-time, first working on bringing Singly’s Locker Project up to web scale, and then exploring the wilds of monitoring the performance of Node applications for New Relic. He has learned an unreasonable amount about JavaScript, V8, libuv, and making it all work well in the real world.

Forrest has spent the last year plumbing the depths of Node for New Relic, as he brings support for Node to New Relic. It’s a process that has required ingenuity, guile, and the willingness to break a lot of otherwise very sensible rules. He will take you on a tour of some of the more powerful but dangerous tools in the JavaScript toolbox: monkey patching, working with Node’s internals, and mixing synchronous and asynchronous code without setting your hair on fire.
Charlie Key is presenting…

Building a Multiplayer World for Pillow Pets

Charlie

Charlie is Co-founder of Modulus, a premier Node.js hosting solution. He has spent the last six-years working in the software, where he has created over a dozen production websites and applications for many global brands.

He is also a core contributor on Pulse game engine and one of members of the team who developed Pillow Pets World.

The talk will dive into the design goals, architecture and end result of creating this massive virtual world. Come see how Pulse (a HTML5 game engine) and Node.js were combined to create a fast, expandable, mobile ready world. Targeting a game at 10,000,000 people? Then come and find out how Pillow Pets World was built. Pillow Pets World is a virtual world built for millions of kids. Scalability and performance were aspects that were included from the start.

The technologies included in this talk are:

Pulse | HTML5 Game Engine Node.js | Scalable small servers Socket.io | Real-time communication using Web sockets Redis | Small in memory storage used for pub/sub communication between servers

Are you signed up? BUY YOUR TICKET FOR NODE PDX HERE

Want to learn more? http://nodepdx.org/

Want to know the dates? http://nodepdx.org/

Want to know who else is speaking? Stay tuned here or go check out http://nodepdx.org/!

Node PDX – Introducing Adam Ulvi, Aron Racho, Christopher Meiklejohn, Max Ogden and Brock Whitten!

May 12, 2013

Awesome Coders, Community, Conferences

Leave a Comment

…and the fourth iteration begins!

Adam Ulvi is presenting…

Put a Sensor On It!

Adam Ulvi

Adam is a Portland native (straight out of the Simpsons) with a penchant for software and systems integration. 8-bit gamer, seasoned professional, perpertual noob. Specialization is for insects.

Learn how easy it is to create your own monitoring system! Hobbyist components and a rich ‘maker’ community puts advanced system designs well within the reach of your average software wonk. Stop planning and start building!

Our case study is ‘GroMon’, a solution for monitoring a tiny indoor lettuce garden. Our wireless sensor keeps track of temperature and humidity, if the plants get too hot or too cold then we are notified via text message.

We will discuss the design goals and architecture, as well as component selection, prototyping and debugging steps. With a little bit of programming skill and patience, anyone can build this network. Learn how to easily extend this solution for your own use.

Our stack is Node.js running on a Raspberry Pi. We connect over Bluetooth to an Arduino hosting a single sensor. All components can be purchased off-the-shelf, no soldering is required and the total cost is around $80. Code and bill of materials is available on GitHub, let’s hack!

Aron Racho is presenting…

Jive Purposeful Places SDK – A NodeJS Bromance

Aron

Aron Racho is a Senior Software Developer at Jive Software. Aron’s background has primarily been in Java, about 10 years in. He is a relatively recent convert to Javascript, and server-side Javascript in particular. Though relatively new to NodeJS, he has been smitten by its fluency, flexibility, and superb design. One might even say a Bromance has been started.

Ok, get ready, this description for this sessions is HUGE!

Jive Software’s latest cloud release enables 3rd party developers to easily push data marshalled from external systems of record such as Salesforce into Jive. Our aim is to publish a developer framework and API which makes it drop-dead simple — and fun — to get up-and-going from scratch, or easily integrate into an existing framework. We chose NodeJS precisely for those reasons:

High developer adoption
High velocity development — javascript! no compilation required
Best-in-class IDE support (IntelliJ for example)
Native support for REST and HTTP
Excellent package management system (NPM == maven the Good Parts)
Tons of great libraries
Amazing documentation, well organized, very easy to get started instantly

Our framework is built on Express, and is designed to be programmed by “convention: fill in the blanks with logic specific to your application, and as long as you’ve satisfied the contract, the framework automatically:

Wires up routes required for configuring your integration
Notifies your listeners for integration life cycle events (integration created; destroyed; updated, etc.)
Executes recurrent tasks you’ve scheduled
Handles persistence of required objects. We have support for 3 types of persistence out of the box — in-memory, file, and MondoDB.

The framework is designed for developers who want to as quickly as possible start integrating a 3rd party service with Jive, with minimum setup.

For those interested in integrating Jive into an existing NodeJS Express app, we are going to make the underlying API available to developers, allowing them finer grained control over the setup of their integration. This API will be the same one underlying the mechanics of the aforementioned framework.

For my presentation, I will be describe how we used NodeJS as the basis for this framework and API. Please note at the time of this proposal, we’ve created the framework, and are now in the process of refining the API so that it can be used independently of the framework.

Christopher Meiklejohn is presenting…

An Introduction to Functional Reactive Programming

Chris Meiklejohn

Christopher Meiklejohn is a Software Engineer with Basho Technologies, Inc. where he focuses on building rich web applications for Riak using Erlang and JavaScript. Before joining Basho, he worked at Swipely, a loyalty program startup based in Providence where he maintained critical infrastructure components written in Ruby. Christopher currently serves as one of the maintainers of Rubygems.org.

Chris also knows a thing or three about this show… called The Wire… he could probably speak entirely in quotes solely from The Wire and still make complete sense. So throw a quote out, I bet he’ll catch it.

There is no doubt that todays web applications continue to grow in adoption, replacing their desktop counterparts in all areas of computation. Essential to their growth is their ability to provide near-native performance and rich user experiences. As these applications grow in essential complexity, they also grow in accidental complexity due to the imperative callback processing style found in most web applications. The asynchronous nature of most of these applications also further compounds the issues due to guarantees around message ordering, and a level of indirection required in callbacks handling events.

Functional reactive programming is one approach for mitigating accidental complexity, using a declarative and composable data-flow model. During this talk, we’ll look at the history of functional reactive programming, some JavaScript implementations of FRP, and finally some ClojureScript implementations of FRP and examples on how to get started with using functional reactive programming.

Max Ogden is presenting…

Minecraft.js

Max Ogden – Penciled and colored!

Max used to live here in Portland, and at some point defected to work on noble causes with Code for America. Since then he’s been an Oaktown Coder (Oakland, the other city near San Francisco).

In January of 2013 he started the Voxel.js Project. Since he and contributors have generatored nearly 100 node modules related to 3D game development and distribution. Examples including voxel rendering and first person controls and physics. On the main voxel-engine the project has received over 50 pull requests from 20 contributors. All of this since January. Max is going to dive into this effort and what inspiration people have taken to dive into game development with JavaScript.

Brock Whitten is presenting…

Mighty Messaging Patterns

Brock Whitten

Best know for having co-created PhoneGap, Brock went on to work at Joyent where he created the Public API for the beloved (and now sunset) No.de Platform. He’s now working on the Harp Platform where he’s worked with a team to have created a dead simple publishing platform that uses Dropbox as its deployment mechanism. Much has been learned, he’s read to share.

He describes his course as, “Messaging is the lifeblood of distributed systems yet it is often treated as an afterthought when applications are architected. Few get passed the point of tacking on a message queue to fire and forget tasks with no visibility into what is happening on the system. I’m here to tell you we can do better. That Messaging can do more for us than just put tasks in the background. By combining basic messaging paradigms we can build powerful distributed systems with full awareness of what is happening around the network. And we can do it all in pure JavaScript.

In this talk, I will start with a crash course on the basic messaging patterns push/pull, pub/sub, and request/reply and then show a real example of how we have combined these patterns to build a custom message broker that we have used to build a fully distributed and modular architecture for the Harp Platform. I will share details about what we have learned and common pitfalls to avoid when building a messaging system for your needs.

Basic outline for the talk:

how messaging can be useful
crash course on the basic message patterns
how to get started with zeromq/axon
common pitfalls when in production
proven trade secrets we have learned

By the end of the talk, my hope is that everyone will have a new appreciation for what can be achieved with massaging and will know where to begin when attempting to integrate messaging into their next project. I feel this aspect of building modern web applications is often overlooked and viable techniques need to be shared and discussed.“

Are you signed up? BUY YOUR TICKET FOR NODE PDX HERE

Want to learn more? http://nodepdx.org/

Want to know the dates? http://nodepdx.org/

Want to know who else is speaking? Stay tuned here or go check out http://nodepdx.org/!

Have a last minute request, idea, comment or a speaking proposal? http://nodepdx.org/

Node PDX – Introducing Scott Hanselman, Tracy Abrahms, Matthew Lyons & J Chris Anderson

May 9, 2013

Awesome Coders, Community, Conferences

Leave a Comment

Welcome to iteration 3 of introductions.

Scott Hanselman is presenting…

Azure for the non-Microsoft Person – How and Why?

Scott Hanselman

Scott is a web developer who has been blogging at http://hanselman.com for over a decade. He works on Azure and ASP.NET for Microsoft out of his home office in Portland. Scott has three podcasts, http://hanselminutes.com for tech talk, http://thisdeveloperslife.com on developers’ lives and loves, and http://ratchetandthegeek.com for pop culture and tech media. He’s written a number of books and spoken in person to almost a half million developers worldwide.

For a dose of Scott, check out how to scale to infinity by doing NOTHING!

Join Hanselman as he digs into the open source SDKs of Windows Azure. Let’s access Azure from the command line and deploy and redeploy with Git. We’ll fire up Linux VMs, setup Mongo and run node.js apps in the cloud. We’ll look at things like SendGrid and New Relic. The future of the cloud is open and it’s a hybrid. This very technical session will cover Windows and Mac, .NET as well as pinching pennies in the cloud.

Tracy Abrahms is presenting…

Punching Rocks: my intimate adventure with rock climbing and becoming a programmer

Tracy Abrahms

Tracy tells a story, “A funny thing happened at the rock gym… I kept running into programmers. Rock climbing is a constant challenge. Physical? Hardly. Tired muscles is a concern once you’re two pitches up and can’t figure out the next move. I’m afraid of heights! The psychological and mental tenacity required to complete a wall feels eerily similar to the daily challenges of the Programmer. You will commit yourself to situations that you pretty much HAVE to find a way out of. Sound familiar?

Malcom Gladwell’s 10,000 hours–I’m not that far in. How are people learning to program nowadays? MOOCs, tutorials, workshops, communities(meetup.com), books, standard CS degrees(what of non-tech degrees?), internships, apprenticeships, code schools. What am I doing? What have I done? How many callouses have I build in the process? Experiences shared from my own perspective and others I have met on my journey have shown me a number of great ways to help move forward those willing to take up the challenge.

Finally, what can I do, along with knowledgeable and helpful Node.js programmers, to build the knowledge base and accessibility into the community? How do I get programmers hooked? And how can I convince all of these brainiacs to get out and punch a few rocks?“

Matthew Lyon is presenting…

Let’s Make Music

Matthew Lyon

I got to work with Matthew @mattly for a short period while I was helping out at AppFog a while ago. I also have had the great fortune of catching Matthew talk about JavaScript magic and more!

Also a homebrewer, hacker, artist, photographer Mattew Lyon is coming to give us a lesson on making music. He describes his course as thus, “Music happens over time and so does asynchronous code. Therefore, let’s write music in Javascript! We’ll build a network sound sequence server that will talk to freely-available softsynths and possibly an HTML5 Audio API, covering how to model things such as drum patterns, melodies, oscillators and pattern banks.“

J Chris Anderson presenting…

Hands On Realtime Text Analytics

J Chris Anderson

J Chris just recently returned to Portland because of many different reasons after hacking in the San Francisco area. If you know what livable streets are (re: Bike Portland and if you intend to stay and live in Portland, you probably should get in on this, livable streets are too good to merely pass up), you’ll want to keep tabs on J Chris’ efforts ongoing, I definitely will be! J Chris also refers to himself as a hacker dad, couchbase cofounder & mobile architect.

With “Hands on Realtime Text Analytics” J Chris will present to us as described:

“A common problem in large scale computing, is coordinating workers when they can be scattered across compute nodes. For workloads like this, atomic operators like increment and decrement reduce contention between distributed processes.

In this talk I’ll show a full text analysis tool which ranks words in the Twitter firehose. By storing each token in a key based on its characteristics, we can provide word rankings both globally, as well as over time and space.

We’ll show the running application, and take a tour through the code, as well as discuss cluster sizing and how it is impacted by variations in the input data stream.

For instance a tweet in English from San Francisco might say “Go Giants” so counters for 2012:go and usa-sf:2012-07:giants (among a few dozen others) are incremented. Even using memory like this, the counts from a full corpus of English text would only take a few gigabytes to hold, making this architecture more efficient than a traditional index-and-rollup approach.“

Are you signed up? BUY YOUR TICKET FOR NODE PDX HERE

Want to learn more? http://nodepdx.org/

Want to know the dates? http://nodepdx.org/

Want to know who else is speaking? Stay tuned here or go check out http://nodepdx.org/!

Have a last minute request, idea, comment or a speaking proposal? http://nodepdx.org/

Backup Riak – Learning About Distributed Databases :: Issue 001

May 8, 2013

AWS, Basho, Databases, Distributed Things, How-to, Riak

2 Comments

I’ve got more than a few series in the queue, so why not another one eh! The intent is, I’ll grab a specific topic to break down and add details to related to distributed systems, primarily around Riak. I will however diverge into other distributed databases too, but I’ll primarily be sticking to Riak. Without more introduction, the first topic is…

Backing Up and Recovery of Riak (Nodes)

I’ve been asked approximately 423,983,321.7 zillion times how this is done. So here’s a quick summary and respective links to the best ways to backup Riak, how to recover nodes.

When backing up Riak there are two key things that need copied to the backup storage; the ring and data directories. Each of these things are specific based on the backend used with Riak. In addition to the core backup containing the ring and data, another good thing to backup is the configuration directory. When recovering this comes in useful.

For the locations of data, it depends slightly based on the operating system being used. The two big variances are OS-X and Linux Distros. On OS-X the data path, ring data and configuration are located at the locations listed below:

Bitcask data: ./data/bitcask
LevelDB data: ./data/leveldb
Ring data: ./data/riak/ring
Configuration: ./etc

For each specific distro, there are slight variations on where the locations are, for a full list check out the Basho Riak docs on backups. But on Linux distros the paths are as follows:

Debian and Ubuntu

Bitcask data: /var/lib/riak/bitcask
LevelDB data: /var/lib/riak/leveldb
Ring data: /var/lib/riak/ring
Configuration: /etc/riak

Fedora and RHEL

Bitcask data: /var/lib/riak/bitcask
LevelDB data: /var/lib/riak/leveldb
Ring data: /var/lib/riak/ring
Configuration: /etc/riak

Other Operating System Paths

Freebsd

Bitcask data: /var/db/riak/bitcask
LevelDB data: /var/db/riak/leveldb
Ring data: /var/db/riak/ring
Configuration: /usr/local/etc/riak

SmartOS

Bitcask data: /var/db/riak/bitcask
LevelDB data: /var/db/riak/leveldb
Ring data: /var/db/riak/ring
Configuration: /opt/local/etc/riak

Solaris

Bitcask data: /opt/riak/data/bitcask
LevelDB data: /opt/riak/data/leveldb
Ring data: /opt/riak/ring
Configuration: /opt/riak/etc

When backing things up, it’s important to note that each node could have slightly inconsistent data. The data however is rebuilt by the Riak read-repair system once it is recovered and brought into use.

Backup Jobs

One of the easiest ways to backup Riak is to setup a cron job with your choice of cp, rsync or tar. Then just get those files onto whatever your choice of backup medium. An example tar cron job to backup a Bitcask backend is shown below (snagged from the documentation) just to give you an idea of where to start.

tar -czf /mnt/riak_backups/riak_data_`date +%Y%m%d_%H%M`.tar.gz /var/lib/riak/bitcask /var/lib/riak/ring /etc/riak

For a leveldb back end the most important thing to note is that the node must be stopped. The basic workflow of backing up a node in this manner is to stop the node, backup the data, ring and configuration and then start the node back up.

Backup Recovery / Restoring

When recovering data on a node that is replacing an existing node that has the same name (fully qualified or IP) then follow the steps below:

Install Riak
Restore the old node’s configuration, data & ring.
Start the node

Once you’ve got the node started back up it’s a good idea to do a ping or status against the node to verify it is in a good state.

If node names have been changed there are additional steps.

Mark the original instance down
```
riak-admin down 
```
Join the restored cluster
```
riak-admin join 
```
Replace the original with
```
riak-admin cluster force-replcae  
```
Get the cluster plan built
```
riak-admin cluster plan
```
Commit the changes
```
riak-admin cluster commit
```
Change the -name setting in the vm.args configuration file to match the new name.
Change & verify that the IP reflects the instances IP in the app.config for http and protocol buffer interfaces.

Cluster Backups via Riak Enterprise Multi-Data Center (MDC)

In the above sections I wrote about the traditional backup approaches. This is very similar to the way RDBMS are backed up. However, with a distributed system like Riak there is another great alternative if you’re utilizing multiple datacenters and Enterprise Riak. In this version of Riak, which is basically Riak with additional features and capabilities, one of the possible backup scenarios is to use the Multi-Data Center, or MDC, to replicate a duplicate cluster and use it as an active, real-time and always ready backup.

One workflow that is an exceptionally effective way to provide backups is to setup the “backup” cluster beside the current operative cluster. As an example, if your cluster is operational in AWS and it is running in X region and Y zone then you’d want to put the backup cluster in that same region and zone. Once you’ve setup Riak Enterprise and MDC, then just setup a full sync. Once the full sync is done you can then remove the backup cluster and it provides a point in time backup of the data.

riak-repl start-fullsync

It’s easy to schedule full sync operations to low usage periods and it is also possible to pause and resume full sync operations.

riak-repl resume-fullsync<br />riak-repl pause-fullsync

The variations on backing up data with Riak Enterprise and MDC are pretty expansive. Doing a point in time, maintaining a secondary live copy of the data, using the replication as a data dump to another cluster or even just using the MDC replication to dump all of the data to a single instance.

File System Snapshots

One other technique that is extremely efficient, fast and thorough is snapshotting the file system. The backup workflow for snapshots is extremely easy. First stop Riak, then snapshot, then start Riak again. Of all the methods, snapshotting is one of the easiest of the options. Just like setting up a cron job, automating snapshots based on some pre-defined schedule and meshing that with automated start and stop of Riak provides a very thorough backup.

With these options, have fun strategizing your stratagems into strategies for backups.

Diskettes

One of the oldest, tried and true backups is the old diskette. The bestest way to backup with diskettes is to backup each node on three diskettes each. The send one of each diskettes to a geographically dispersed to a bank lock box or other secure facility. Do this for each node, and if need be use as many diskettes for each node as needed. A particularly useful method is to use the sharded zip strategy to stripe a backup across many diskettes. Once each lock box has a copy of the node for each node in the cluster, you’ll have one of the most secure backups in existence. Nothing compares to the diskette backup!

References:

Node PDX – Introducing Jason Denizac, Thorsten Lorenz and Peter Braden

May 8, 2013

Awesome Coders, Community, Conferences

Leave a Comment

Here’s the second of a series of introductions for the upcoming Node PDX Conference while I train ride up to Seattle today.

Jason Denizac presenting…

Programming With a Purpose

Jason Denizac

I met Jason over a year ago while he was in Portland at a coding conference. Since then we’ve kept in touch, and everytime I’ve enjoyed seeing him speak. This guy is from a place far over the hills and valleys somewhere in that central part of the United States, but I’m stoked he’s out here on the west coast now bringing more brainpower to the region.

Jason described his talk, “As programmers we are wizards. Our job is to manufacture super powers. Like with the Manhattan Project, wielding such great technological power entails moral implications which we ignore at our peril. But we can recognize this power, embrace it, and use it for great good. Node gives us a tool kit to confront great problems and share in solving them. Specifically: radical decomposition, horizontal reuse, and positive community norms around testing and documentation.

Audience participation requested: be prepared to share your expertise in a real world problem space like education, civics, social equity, environmental conservation, healthcare, or your own favorite “intractable” problem. If time and scheduling permits, I’d love to have a series of lightning talks in which people could introduce other developers to the problem domain.“

This should be a good one.

Thorsten Lorenz presenting…

Module Driven Programming

Thorsten Lorenz

Thorsten Lorenz has been creating nodejs projects for over two years mostly in his spare time. The main reason why he can’t (and won’t) stop is, that ideas can be transformed into something that works in a very short amount of time when implemented in JavaScript. Aside from these projects, he also contributes to projects important for the community, one of the recent additions is source map support to browserify. These source maps even allow debugging transpiled languages (i.e. CoffeeScript) right in the browser.

Thorsten threw a bullet point list of the key points for his presentation, which follows:

what constitutes a module
why it is desirable to build smaller modules
challenges and patterns for separating the application into independent modules
process of pulling out a module from an application
replpad case study
how to become module driven phase1 and phase2
browserify and how it enables to even manage your client side modules with npm
quick primer on tools like npm init, pkginit, travisify and npm link that help with module driven development

Peter Braden presenting…

Teaching Robots to See With Javascript

Peter Braden

Peter Braden is the author of the node-opencv bindings that allow node scripts to interface with this powerful computer vision library. Peter related to us that he’s both excited and terrified for a future of seeing drones and robots. Currently he works as a web consultant-of-all-trades at frozenridge.co.

In his talk he’ll be diving into, “As we start to use javascript to control quadrocopters and robots, it becomes important that they can see with images from cameras onboard. In this talk we’ll take a step back and look at the field of computer vision; look at some of the exciting practical problems that can be solved with CV techniques, and look at how we can do this with javascript. Whether it’s detecting and recognising faces, building a picture of surroundings from a video stream, or tracking shapes, we’ll look at how we can make our javascript programs see. And all going well, we’ll have a live demonstration of a drone navigating based on onboard camera data.”

…

…ready to get your tickets?

Are you signed up? BUY YOUR TICKET FOR NODE PDX HERE

Want to learn more? http://nodepdx.org/

Want to know the dates? http://nodepdx.org/

Want to know who else is speaking? Stay tuned here or go check out http://nodepdx.org/!

Have a last minute request, idea, comment or a speaking proposal? http://nodepdx.org/

M	T	W	T	F	S	S
« Apr
		1	2	3	4	5
6	7	8	9	10	11	12
13	14	15	16	17	18	19
20	21	22	23	24	25	26
27	28	29	30	31

—Composite Code

Snippets, software architecture, lean, agile, management, and leadership bits.

Polyglot Conference in Vancouver, Using Riak to Analyze, Store Data and Survive the Zombie Apocalypse

Node PDX – Introducing Adam Baldwin, James Halliday, Ryan Jarvinen, Mike McNeil and Horse JS

Introducing NodeSecurity.io

beep boop

Clustering Node.js on OpenShift

Intro to Sails.js

Implementing Git in JavaScript & the Browser: A Case Study

JavaScript, This is Confusing

Node PDX – Introducing Zach Bobb, Paul Jungwirth, Forrest Norvell and Charlie Key

Building a Computer In Your Browser

Handling Errors with Cluster and Domains

Do as I say, not as I do: Node in the real world

Building a Multiplayer World for Pillow Pets

Node PDX – Introducing Adam Ulvi, Aron Racho, Christopher Meiklejohn, Max Ogden and Brock Whitten!

Put a Sensor On It!

Jive Purposeful Places SDK – A NodeJS Bromance

An Introduction to Functional Reactive Programming

Minecraft.js

Mighty Messaging Patterns

Node PDX – Introducing Scott Hanselman, Tracy Abrahms, Matthew Lyons & J Chris Anderson

Azure for the non-Microsoft Person – How and Why?

Punching Rocks: my intimate adventure with rock climbing and becoming a programmer

Let’s Make Music

Hands On Realtime Text Analytics

Backup Riak – Learning About Distributed Databases :: Issue 001

Backing Up and Recovery of Riak (Nodes)

File System Snapshots

Diskettes

Node PDX – Introducing Jason Denizac, Thorsten Lorenz and Peter Braden

Programming With a Purpose

Module Driven Programming

Teaching Robots to See With Javascript

Follow “Composite Code”

Share this:

Like this:

Introducing NodeSecurity.io

beep boop

Clustering Node.js on OpenShift

Intro to Sails.js

Implementing Git in JavaScript & the Browser: A Case Study

JavaScript, This is Confusing

Share this:

Like this:

Building a Computer In Your Browser

Handling Errors with Cluster and Domains

Do as I say, not as I do: Node in the real world

Building a Multiplayer World for Pillow Pets

Share this:

Like this:

Put a Sensor On It!

Jive Purposeful Places SDK – A NodeJS Bromance

An Introduction to Functional Reactive Programming

Minecraft.js

Mighty Messaging Patterns

Share this:

Like this:

Azure for the non-Microsoft Person – How and Why?

Punching Rocks: my intimate adventure with rock climbing and becoming a programmer

Let’s Make Music

Hands On Realtime Text Analytics

Share this:

Like this:

Backing Up and Recovery of Riak (Nodes)

File System Snapshots

Diskettes

Share this:

Like this:

Programming With a Purpose

Module Driven Programming

Teaching Robots to See With Javascript

Share this:

Like this:

Follow “Composite Code”