A big part of my job is responding to network down situations. On any given day I could be dealing with a variety of different types of WAN circuits. T1’s, DS3’s, several different flavors of carrier ethernet, cable, DSL – you name it and I’ve probably had to troubleshoot it recently.
Without a doubt the most frustrating issues I deal with are when we (meaning myself and our Vendor) have troubleshooted a connectivity problem down to a local loop issue. This means that our WAN provider now needs to open a ticket with Verizon (among the circuits I deal with, Verizon is usually responsible for the local loop) to dispatch and fix the issue. I’ll happily admit that most of the time these issues are rectified by Verizon in an efficient and timely manner.
However, when they don’t go OK, these issues can snowball into truly epic levels of craziness.
When this craziness happens, the worst aspect is how helpless I am to do anything. All I can do with Verizon is to keep asking for updates from our vendor and to try to get our issues elevated in Verizon’s support system so the issue becomes more visible on their end. At the same time I’m trying to reassure my users at the sites that we’re doing all we can, and that we do care that they’re unable to get any work done.
As an extreme example, I’m going to share what two of our sites went through a few months ago. These sites are two buildings down from each other in an urban area and have T1s for connectivity.
Day 1
- Mid morning. Site 1 and Site 2 go down. I dial in to the routers, see that they’re powered up and have a down/down on the serial interfaces. I call the vendor who can’t loop the smartjack. Time to call up Verizon.
- Early afternoon. No real updates, Verizon still testing.
- Late afternoon. Verizon decides a dispatch out to the area is needed. Unfortunately, it’s close to 5:00pm now – QUITTING TIME! Verizon schedules a morning dispatch (Side note – After several years of doing this I have yet to have a Verizon field tech try to resolve one of our issues after hours).
Day 2
- Morning – Verizon dispatches out. Not first thing of course, but they get on it around 10:00am or so.
- Noontime – Site 2 back up. Verizon found an issue at the doubler. YAY! However, numerous errors are now accumulating on the serial interface. BOO!
- Afternoon – Site 2 bounces several times. The errors are making connectivity spotty at best.
- Late afternoon – Verizon still working on the errors at site 2. They inform us that the problem at Site 1 is an equipment issue on our end.
WTF? There are two locations, both using T1s and right next to each other. They both go down at the same exact time and you’re trying to tell me that one of them was a network problem and one was equipment related? Grrr…..blood pressure….rising….
Day 3
- Morning – Site 1 still down. Site 2’s T1 bounced all evening. Verizon dispatches out first thing in the morning (amazing!) and decides to switch the cable pairs at both sites (I’m not sure what happened to the equipment issue).
- Late morning – Both sites up and stable! YAY!
Day 4
- Morning – Both sites down. BOO!
- Mid-morning – I find out that when the Verizon tech switched the sites to the new cable pairs the day before, they took two pairs that were in use by another client and knocked that customer offline. Another tech dispatched out this morning and took the cable pairs back thus knocking the two sites off again.
- Afternoon – The smartjack is being worked on at one of the sites. That site is still down at the end of the day.
Days 5+6
Weekend! Woo Hoo! Both sites are down the entire time. You can’t expect a tiny company like Verizon to fix something over the weekend, can you?
Day 7
- Morning – Verizon dispatches out bright and early. They detect a cable issue in the neighborhood. Time to call in the cable crew to fix things up. YAY!
- Late afternoon – No dispatching of the cable crew this afternoon. BOO!
Day 8
- Morning – Cable crew has dispatched out!
- Late morning – No updates….
- Afternoon – No updates….
- Late afternoon – Cable crew has found a patch of bad fiber! YAY! They have to order new fiber to fix it, should be here in a week or so. BOO!
Days 9 – 11
No new fiber, both sites still down.
Days 12+13
Weekend! Woo Hoo! Both sites are down the entire time of course.
Day 14
- Morning – New fiber is in! YAY!
- Afternoon – Cable crew did not dispatch out today. BOO!
Day 15
- Morning – Cable crew is dispatching out! YAY!
- Mid-morning – BOTH SITES COME BACK UP AND STAY UP! YAY!
Is that sequence of events incredible or what? I can appreciate that there’s a lot I don’t understand about the job of a Verizon tech, but I can’t help but get pissed off when there were at least two attempts to just dump my issues (when I was told my equipment was an issue and when they grabbed two “live” wire pairs to transfer my circuit to) without trying to get to the heart of the problem. I don’t think you can deny there was a general lack of urgency on Verizon’s part either.
I’m sure you won’t be surprised to learn that both sites continued to have recurring issues over the next few months. The client eventually decided that enough was enough and discontinued the T1’s and migrated the sites to a cable connection. They haven’t had a problem since.
Damn it felt good to get that off my chest 
! Thanks for letting me vent.
Jason