The vanishing specialist

No matter how established things seem to be, at some point a truly disruptive innovation will turn everything on its head.  We have seen the consequences of this many times before with entire industries and occupations becoming obsolete and extinct. What we are seeing now are the beginnings of this disruption in enterprise IT.  

The world of IT has long been the domain of specialists, men and women who were responsible for the individual silos of infrastructure that underpin the information system as a whole.

The actual business system or application being delivered is normally integrated to suit the combination of platforms these systems run on, the business needs they fulfil, and the experts that built them. In essence, each one of these business applications becomes a highly customised deployment that is painstakingly built into the IT infrastructure to support the business needs around it.

Today, this application deployment model is in the process of changing, resulting in many of these experts seeing their specialties vanish.  Why?  Because workloads are being abstracted from infrastructure.  

The sound of change

We can see the essence of this change in the way that startups have been building infrastructure over the last ten years.  The dotcom boom saw massive infrastructures filled with these custom application deployments that were established at great cost, often before there was a business to run on them.  Fast forward thirteen years and a startup asking for a million dollars to build IT infrastructure would find it difficult to secure funding.  

But before we dig deeper into what that actually means, let’s switch gears for a moment and use an example from a completely different field: music.

My uncle is a sound producer and composer.  I remember having a conversation with him over a decade ago, where he was telling me about the changing world of session musicians. Traditionally, session musicians were called upon to play as part of a music piece in a recording studio.  Obviously, the more instruments your composition took on, the more expensive and complex the project became.

The inherent challenge is that as production costs came under more and more pressure in fields like advertising and television, it was no longer viable to pay for a dozen session musicians.  Producers turned in another direction: they could have a dozen instruments at the push of a button on a synthesizer…   

According to a New York Times article about this shift, the synthesizer had become irresistible for two reasons: “economy and flexibility.”  

The synthesizers “irresistibility” over-ruled most other concerns, including the individual nuances of the musicians.  Today, my uncle’s assessment that session musicians would be replaced by the synthesizer is the status quo in all areas where production costs are constrained.  

Now back to IT.  Until recently when it came to system management, these painstakingly hand-crafted application deployments were typically rigid, temperamental, and almost treated with a level of care bordering on superstition.  After all, they had taken such a long time to set up, to get right, to shake down and iron out. Now that they were up, whatever you did: Don’t touch them; don’t patch them; and don’t reboot them.

What organisations could not, and still cannot, afford is risk to the availability of their critical systems.  For example, I have seen data centre relocations where the concern was that certain applications, when turned off, may not turn back on again.

Riding the ‘Cloud’

So how are modern startups handling things differently than their peers in the dotcom era?  The cloud has shaken the startup world quite significantly.  Now, a startup will build an application on cloud infrastructure that can cost only dollars a day, and only scale the systems up when the business requirements dictate a need.

Thanks to the cloud, the world is moving in a direction that won’t accommodate the kind of system idiosyncrasies that these custom application deployments are dependent on. When you go into the cloud, typical enterprise IT shops hit two main issues.

Firstly, in a cloud, you see the total cost of ownership every month, from the costs associated with the actual infrastructure, to the guy who waters the pot plants in the lobby of the cloud data centre.  It is a real total cost of ownership – no hidden cost centres, no “Facilities Department” paying for power, no “Corporate” cost centre covering human resources overheads.

Many enterprise IT shops, when they look at the cloud find themselves in a state of “Sticker Shock” because all of the costs are included in the monthly bill – and the CFO sees it all!

Secondly, to compound this cost-based assessment problem, the old way of building and maintaining these precious monolithic inflexible application deployments simply does not work in cloud world. When IT houses on tight budgets can’t reduce costs any further without turning things off, they need to use smaller systems that can scale up and down dynamically to meet peak needs. As we’ve seen, this strategy is diametrically opposed to the present way that enterprise IT does things.

The current state of corporate IT

So what is happening in corporate IT now – we are now living in the world of the “Virtualised Data Centre”.  We have reached the point where the systems are now almost totally abstracted from the infrastructure building blocks. We are moving beyond infrastructure to an application-centric view of the world.  The underlying infrastructure is becoming “plumbing” – in a sense, it’s out of sight; out of mind. In this view, what matters isn’t infrastructure, but the application outcome.

So what does a session musician have in common with an IT person? Like musicians, IT people are specialists who come in and work on a single piece of the whole.  But if the whole is being orchestrated, these specialists will be replaced by the person driving the synthesizer, doing all of their jobs from one place, building systems from commodity pieces brought together to solve a problem.

Soon, people won't want an expensive networking ninja, a separate and expensive storage sage, another separate and equally expensive security guru, not to mention the expensive application aficionado.  

Driving "300 violins at the press of a button"

Enterprise IT, coming under the same cost pressures that music production houses came under a decade ago, will increasingly expect their IT shops to be able to drive "300 violins at the press of a button".  To achieve this, they will need an orchestrator who crosses all the silos and manages the application outcome all from one place. They will spin up applications with lots of small, wimpy, cheap systems made resilient with intelligent application delivery controllers in which ever environment meets their availability, security, and budgetary requirements.

Application changes will be automated and pushed almost constantly with rigorous automated testing. And when demand is not there, the orchestration will spin the application down and turn off unneeded systems to save money. 

So where does this leave people like me, one of the old school "security guys" with some network skills? Unless I start moving my skillset towards that orchestrator role, I will be in a pretty tough place. So I have picked up Python, and have invested heavily in building skills in interfacing with REST-based APIs.  In short, I am able to orchestrate.

So my uncle tells me that there are only 10 per cent of the jobs for session musicians that there were 15 years ago, but individuals are now more enabled to produce a finished product.  That’s exactly why this massive shift will ultimately be a positive one.  

Just like the synthesizer made possible many ambitious creations that would otherwise be unfeasible based on cost with music, this shift to orchestration by abstracting the application from the infrastructure has liberated organisations both large and small.  The cloud will be taken up gradually as enterprise IT cost pressures require the infrastructure design to be refactored.  

Just as with session musicians, there will be significantly fewer IT infrastructure jobs, but more jobs for application orchestrators – the good news is the “music” will be even better in the end.

Aidan Clarke is a Global Consulting Engineer at Riverbed Technology.