"But it sounded like such a good idea!" How many times have we heard this in the midst of battling the chaos monster we created for ourselves? In this article, I'm going to discuss an example of this from my time with a former employer, who shall remain nameless to avoid the shame.

As a network engineer for a former employer, one of my major assignments was the buildout of a large SONET fiber optic network. The SONET network used Lucent Metropolis DMX multiplexers to carry both TDM services (DS1/DS3) and fast and gigabit Ethernet. This was a major upgrade that replaced a variety of older Cisco routers and a web of point to point links in the core of the network. As the buildout progressed, a business decision was made that a portion of the network would not be completed or delayed in completion. This was a logical business decision that triggered a chain of bad technology decisions in an attempt to deal with it.

The nature of a SONET network is that it is intended to be used in a ring. This provides redundancy, in case of one break in the ring, traffic will still continue to flow uninterrupted. In this case, we're left with an incomplete broken ring because of the missing portions. Now it's time for the second chaos inducing business decision, we MUST close the ring and do it with minimal to zero cost.

So now we're at the first try. There was existing fiber in place from an office part way up one "horn" to the end of the second "horn". The fiber was out of spec to run the OC48 SONET links but it was adequate to run 100Base-FX Ethernet. So, the first try was to tie an Ethernet port on both ends together with this fiber link and close the ring. Unfortunately, this wasn't such a good idea. Lucent, in their wisdom, had neglected to implement spanning tree protocol on the DMX Ethernet switch ports (only the virtual port connecting the Ethernet switch to the SONET ring had STP). This resulted in a massive and immediate STP loop and interrupted service for about 65,000 people for 10 minutes.

The second try was to insert a Cisco 2924 Ethernet switch in the link and allow the 2924's STP to break the loop. This idea worked perfectly in testing and worked perfectly on installation.

Unfortunately, sometime later, the network once again experienced an STP loop with corresponding service outage. I attributed this to the fact that the 2924 wasn't aware of the STP network of the DMX Ethernet switches and vice versa and they were making independent decisions that might conflict.

Third try was to chop up the SONET network, using scarce and expensive Ethernet ports on the DMXs and reinserting the old Cisco routers. At this point, I decided a mutual parting of the ways was appropriate, given where I could see this was heading. We had taken an elegant, state of the art, SONET network and mutilated it in an attempt to patchwork it, creating an ill performing and unsupportable monster. To say that this would have been a nightmare doesn't say enough. The configuration was far from anything intended by Lucent.

This illustrates the danger of allowing mutually conflicting business decisions made without adequate consideration of the consequences to drive a a series of desperate patches to critical network systems in an effort to compensate. This caused lost revenue for the company, lost customer good will, and loss of reputation. Ironically, I believe that, in the end, it would have been cheaper to have finished the original project.

The moral of this? Consider carefully the REAL cost of technology affecting decisions, both in terms of money and effect on customer service. Don't make assumptions about the capability of your network to absorb unconventional changes in configuration. Make sure you understand all the possible downsides. Consult with an experienced network engineer.