ETLs and APIs – A Thought Stream on the Matter, Are Things Really Changing That Much?

Often the tendency has been, and largely still is in Enterprises, to build things really large. Doing big design up front (BDUF) all the way to creating massive JavaScript libraries for generating JavaScript from non-JavaScript langauges, leading to a huge case of analysis paralysis and other such keywords that all point to the mythic man month or software death march. In the end, building big is building to fail – in a non-learning, non-useful, big bank, financial collapse type of failure. So mumbo jumbo aside, what am I saying?

I recently read the article by Anant Jhingran over at apigee titled “From ETL to API A Changed Landscape for Enterprise Data Integration“. In the article Anant points out a number of things around this movement to APIs versus ETL efforts in leading enterprises. I couldn’t help but notice this is a recurring theme in software development and with humans in general.

Observation

With Governments: Central planning through attempting to dictate order (i.e. Soviet Union) will eventually fail  for a less dictatorial plan around keeping things simple and organizing around communities to plan large efforts.

Translation to Tech: Having mainframe/server that dictates order will eventually fail for a less controlled Internet of systems that maintain simplicity and organize around communities of people to create large efforts.

Translation to Software: Building homogenous massive systems of record that dictate order will eventually fail for less controlled service orientation that can maintain their own system of record around departmental domains of people to create and operate large efforts, like enterprises.

Of course, the concern I have is, we humans tend to do things in a cyclic nature, as with Governments we move toward freedom and liberty, then the other direction, then back again. Big control to little control to big control again.

Summary

I see the point in the article, bringing the ETL massiveness under control of the smaller services, but in the end what we all need to do is to keep things in perspective. The extraction, transformation and loading of ETL still needs to occur along with the usage of application programmer interfaces ala the APIs. So where does this really get us, this acknowledged shift? It really just brings into focus a desire to simplify and bring things into a manageable state in the Enterprise. It makes sense that the leading enterprises are doing this, simply because they’ve reached that maturity level. Of course there will still be thousands upon thousands that haven’t reached this maturity point and will have to go through the pain of large scale centrally managed and planned ETL systems.

In the end, we’ll always have to return to the simple things to get true understanding.