Network cleanup at $dayjob
Reading time: 8 minutes (1509 words)Author: @pugmiester
Tags: network
Over the last 6 weeks or so, I’ve been running about like the proverbial headless chicken trying to get ready for our network infrastructure replacements at $dayjob. Last week was the big bang where we closed one floor per day to strip out all of the network cabinet patch cables (15+ years of layers on top of layers), strip out the old switches, install shiny new switches and finally patch back in the limited number of access ports we are actually using now.
But I’m getting ahead of myself.
In preparation, we had a third party team come and help with an audit of all of the ports we could still locate and which, if any, were still in use. When I say we could still locate, it’s because there have been an untold number of department reorganisations and moves around the building over time (despite the director in charge of the building refurbishment project back in 2008 saying “This is the last time we will ever move furniture, we’ll only ever move staff from now on”…. As you can probably guess dear reader, that lasted about 6 weeks) and some areas that used to have banks of desks now have nothing or a single meeting table so there’s a lot of possible ports now just coiled up below the raised floor and a lot of those points were still patched in and taking up space in the switches and patch panels.
Using the audit details we decided on the number of replacement switches we needed to buy and got those ordered. Then it was the challenge of finding some time when we could get the work done. Originally we had assumed we would be working overnight to minimise the disruption to the business but as it turned out we were able to get agreement from the business to close one floor at a time (we have 3) and work on it, leaving the other 2 floors available for anyone who wasn’t able to work from home on the day.
So, we had the dates set, Tuesday May 7th through to Thursday May 9th with Friday the 10th as contingency. Now all I had to do was tick off that last few (hundred) little jobs of making sure we didn’t impact the building access systems, or the CCTV, or the WiFi on other floors, or….. you get the idea. As you can probably guess, every one of these tasks generated another dozen things I’d forgotten about or had been hidden over the years and I rediscovered it.
As we got closer to the go-live date, I finally started to feel like I’d not found anything new to add to the list in a while and it was actually getting shorter but it still felt like the pressure was on. And then we come to implementation week.
Tuesday May 7th - Day 1
Official Start: 07:00, actual start about 06:45
Finish: 15:00 for the contractors, I left some time after 17:00
2 Support tickets logged
2 x 44U high cabinets crammed with somewhere around 500 patch cables which were stripped out. 8 x 48 port switches were stripped out. 5 new 48 port switches were installed together with around 230 patch cables.
This was the first day and we had the extra pressure of meeting and orienting the third party team that were here to help with the cable work as well as starting on the busiest floor with the most to remove and reconnect. I would have preferred to start with a different floor but we didn’t have much choice about when we could do each floor because of prearranged meetings and conferences so we worked around them as best we could. We had agreed building access for up to 12 hours and I’d also got contact details for the key control team in case we ran really late.
07:00 and we were underway. It took about 40 minutes to clear enough patch cables to start getting the old switches removed and by 07:50 the first new switch was alive. The remaining patch cables took about 40 minutes to clear so we could start sorting them by length and we had the WiFi access points back online by 08:45 and we started on the connections out to users desks.
By around 11:00 we had a good third of the new connections back in place and took some time to escape for some lunch. By 15:00 we had everything we wanted reconnecting back in place with some amazing cable management work. The bar has certainly been set pretty high now. The guys from the third party team wanted to stay and help with moving all the patch cables down to the next floor but with a 2 hour drive (each way) for them I opted to send them on their way and I’ll deal with the leftovers.
Wednesday May 8th - Day 2
Official Start: 07:00, actual start about 6:35
Finish: 14:00 for the contractors, I escaped around 16:30
4 Support tickets logged, 2 by me on the day
2 x 44U high cabinets (with hardware installed front and back in one cabinet) crammed with somewhere around 400 patch cables which were stripped out. 7 x 48 ports switches were stripped out. 5 new 48 ports switches were installed together with around 130 patch cables.
06:40 started stripping cables to get the switches out of the way. Our contractor team arrived a few minutes later and started giving me a hand to clear the way for the old switch removal and by 6:55 we had all the old switches out and started getting the new ones in. The guys started the cable strip out about 7:00 and by about 8:30 the new switch stack was online. I can’t remember why it seemed to take so much longer, I think I was digging in with the patch cable removal having lost a chunk of time for the guys helping me get the switches out so it seemed like the least I could do. We got all the new connections in place by about 14:00 and the guys made a move. I spent another hour and a half getting all the spares relocated to the ground floor and clearing out the old switches to the data centre ready for reset and recycling.
We had a couple of issues we found during the day for some of the facilities team who were still onsite and happened to be sitting near where we were working on the cabinets so we got those cleared while we were working. Silly little things like 2 printers with no port labels so they were missed off the patching list and one simply misread. Both were 30 second fixes.
Thursday May 9th - Day 3
Official Start: 07:00, actual start about 06:35
Finish: 14:00 for the contractors, I escaped around 16:30
0 Support tickets logged
1 x 42U high cabinet (with hardware installed front and back) crammed with around 250 patch cables which were stripped out. 7 x 48 port switches were stripped out. 5 new 48 port switches were installed together with around 120 patch cables.
I arrived really early again and was in the building and sorting one remaining issue from yesterday, we missed out MD’s printer off the list. Ooops. All sorted by 6:50 when the team arrived. The ground floor was a little different as the cabinets we were using were Dell ones designed for server hardware so there was very little space to work at the front however, a few minutes looking at the situation at the end of Thursday made us realise the vertical rails were adjustable so on as soon as we have cleated the switches and patch cables out of the way we moved the rails back about 10cm and that made such a massive difference to how we could cable manage. We started repatching about 8:30 and by 8:50 had the critical stuff alive again. This cabinet seemed to go even better than the last 2, I think we had found our ryhthm. We had almost everything reconnected before lunch and the guys escaped around 13:30.
The aftermath…
I had said in advance that there was no way were going to be perfect with so many connections to strip out and reconnect but if we only had a couple per floor I would be happy.
So, was I happy? Yeah, I’m happy. I’m also taking a break from rushing about for a bit. I’m getting too old for this shit. Over the last week I’ve been averaging 18k+ steps per day. Over the last four weeks I’ve spent more time on my feet than sitting down and you know what, I was feeling pretty good for it. Implementation week was a heck of a shift and I was physically exhausted on all three days but at no point did I seem to be getting any of my usual neck or back niggles that have returned now I’m sat down again.