The insidious cost of incrementalism — Part 2

March 20th, 2013 | mike.bushong | No Comments

When I said that the networking industry at large is mired in a culture of incrementalism, I was really painting a broader brush than just networking vendors. The costs of incrementalism on the customer side are every bit as hidden and thus every bit as insidious as those on the vendor side.

If vendor issues start with feature development, the issues on the customer side start with application development. A new Line of Business (LOB) spins up. That LOB has objectives, and those objectives require work to be done. Applications are purchased (or in some cases built), problems are solved, the business is rocking. As the business grows, the demands on IT continue to grow. The interactions between the LOBs and IT become more transactional. Each new request enables new revenue. As IT retires requests, they are hailed. Life is great.

Over time, the business grows. To continue to grow, it must scale. And scale requires new productivity applications. The early applications were broadly applicable to all of the LOBs, so they were easy to justify, even in the cost center that is IT (for most companies). But as the company grows, the individual productivity needs become slightly more focused on specific roles, geographies, verticals, whatever. The central IT team cannot service every little request, so you create a prioritization system that leads to a funnel. The process is great – projects are throttled by an IT advisory board, and IT is no longer the culprit.

But the LOBs do need to be more productive. Maybe they hire a contractor or spin up a shadow IT team to build just one application. One application turns into three or four, and those turn into a few dozen. The next thing you know, your IT infrastructure is supporting 14,000 users and 1,100 applications. But employees love their tools. Never mind the fact that these tools are stressing an infrastructure that was not designed with them in mind. This leads to the IT equivalent of software rot: infrastructure rot.

A bunch of the Cat6ks bought years ago are due for a refresh. The first rule of managing an upgrade? Do no harm. So you buy new equipment, but deploy it to do roughly the same thing the old equipment was doing. You use the same protocols and knobs and doohickeys, not necessarily because they are needed but rather because they were on the old stuff. Why? Because in an environment that is this complex, you can’t possibly know the impact of a change, so the best thing to do is keep everything the same. New firewall rules live alongside old firewall rules, and the poor firewall guy just prays that his phone doesn’t light up whenever he makes a change.

The executive team decides to roll out video conferencing equipment to all the major conference rooms. The network team cringes, because the best way to find out the problems in the network is to put a latency- and jitter-sensitive application in the hands of executives. On an executive call, one of the rooms doesn’t work quite right. The next day, the network team gets raked over the coals: what happened? You try to explain about the 1,100 applications and the sprawling infrastructure, but that won’t work. Your only solution? Build a separate network dedicated to the conference rooms.

You hand the project over to the procurement guys to start the bidding process. They pick out an old RFP, you add a few SDN requirements, and voila! A new RFP is born. Of course, it looks like the old RFP, just longer. Well, you know who won the old RFP, so you call the sales guy. He is ecstatic – a new network! Sure, we can bang out those new features in no time, maybe in the next one or two releases.

You sign a contract and you’re off. The features slip (for good reasons), but you are assured they will show up in another quarter. The project is delayed, and so you see cost overruns as contractors continue to work. Two more months slip, the executive team is not happy with progress or budget, and the CIO or VP of networking is suddenly updating his resume and asking for recommendations on LinkedIn.

Why? Because no one ever sat down to ask the question: what do we actually need? Where should we start? The starting point is not always the ending point. You might not need all the same knobs and doohickeys, so why start with the same RFP? Maybe you don’t even need all 1,100 applications, in which case you ought to start with the existing network.

Our IT teams, our network guys, our infrastructure brethren are all burdened with the curse of incrementalism. So worried about doing no harm, we have collectively forgotten to ask the most basic question: what do we need? We never assess what we have and systemically end-of-life applications. And the result is second or third networks, designed for specific things.

Every year you’re faced with parsing a new set of three-, four- and five-letter acronyms from your vendors. This is the vendor’s fault for obfuscating the solutions and selling you minor incremental improvements as new architectures or major features. But it happens for a reason: customers aren’t willing to take bolder steps to solve their complexity problems. SDN is an interesting acronym, because it is a call to action, not a protocol or product. It has the industry cachet to get your boss’ attention, so use it to clean up your own network. When was the last time you got a hall pass like this?

Leave a Reply