It’s late August here in the United States. That means one thing for a lot of people. Football is starting. No offense rest of the world. Your football is my soccer, although I tend to side with you that my soccer should be called football. How often does one kick an American football? A LOT less than we touch the ball with our hands. I’m getting off on a “semantics” tangent though. It is the one sport that predominately resides within North America. Yes, I am acknowledging you too Canada!
Many athletes at all age levels have been practicing for several months and are ready to get started with the football season. Many a Saturday afternoon, Sunday afternoon, and Monday night will be spent watching people knock each other over to carry a piece of pig skin across some lines on the ground and celebrate by dancing as gracefully as one can when covered by all that protective gear. Millions of people will watch all the way up to early next year when the championships are decided by as little as 1 point. For the most part, there are no do-overs. All of you “instant replay” fans just bite your tongue and let me carry this analogy as far as I can. When the game is over, it is over. There are no series of games like baseball, hockey, and basketball have. You have one shot at glory. Miss it, and you’ll have to wait until next season.
There’s a German proverb which says: “To aim is not enough. You must hit!”
I get paid for things that go bump in the night. Whether that thing happens to be a router failing, or a circuit deciding it no longer likes my 1’s and 0’s, my job is to fix it and fix it fast.
I do come to work during the day. I go to meetings and look at configurations of various hardware. I build network diagrams and dispense or seek advice on a number of different things. I participate in the important philosophical discussions like whether or not Anakin Skywalker was a better Jedi than Luke or Yoda(In my opinion, Anakin (aka Darth Vader) was the better Jedi and was robbed of his destiny by his meddling child and his rebel scum friends). I put in change requests for maintenance that must be performed. I can plow through the day to day stuff without hardly any interaction from management. Of course, they care about the quality of the work and if I used these stencils in my Visio diagrams, they might object. However, my overall existence in the day to day network operations life is rather calm.
In essence, I do the things that need to be done during the day, but my REAL job comes in spurts. Kind of like football(From now on, when I say football, I mean American football.) players. My game time comes at odd hours much like the police officer or fire fighter. When trouble happens, I need to perform. I need to be able to ask the right questions and formulate a short list of what the possible problems are. I need to be able to troubleshoot in a logical fashion either working up/down the OSI model or grabbing a packet capture and examining the session flows. When it is my equipment or systems that are at fault, I have to get in there and make the big play. I need a touchdown each and every time. I can’t drop a pass or fumble the ball. I get paid for results and rest assured my management is watching. They have to. All it takes is for someone much higher up on the food chain to ask why they pay the salaries of network people who can’t seem to fix the network. Then, I am out on the street forced to sell my services to the highest bidder, who hopefully doesn’t play Dungeons and Dragons or World of Warcraft with any of my now former co-workers/managers. I would have used a sport like tennis or basketball, but since I am in IT, the odds of that happening are much less than a bunch of technical geeks sitting around in Viking helmets and leather tunics taking part in the raid of an Ogre village on World of Warcraft over a shared broadband connection in an obscure apartment complex deep in suburbia while guzzling Red Bulls and listening to angry death metal music. By the way, for all of you D&D geeks who are shaking your heads in disgust at my mention of Ogre villages, I get it. I saw Shrek. I know they are solitary creatures, but I needed an effective illustration. If I used Elf village, the visual would have been less powerful.
Am I saying that we can’t make mistakes? Well, that depends. There are some places in which you can’t. Ever. Most places will allow mistakes. We’re all human and mistakes will happen. Of course, with enough attention to detail those mistakes can be minimized significantly. What I AM saying is that you need to be able to perform when a crisis hits. Your entire career at a particular company may come to a screeching halt over just a few minutes of doing the wrong thing. It won’t matter how long you have been with company X if your performance is so poor that company X starts bleeding millions of dollars due to an outage that you can’t fix. Problems are going to happen. Outages are going to happen. If your company expects you to fix them, you better fix them. Now I know that some people get in over their heads. It may be the company’s fault for placing an unrealistic demand on you, or it may be your fault for misrepresenting your capabilities. If your company is expecting you to fix and support issues with F5 load balancers and you have never so much as looked at an F5 load balancer, you better let someone know and get up to speed as fast as you can. After all, your job typically is whatever your company says it is. Don’t like that? Tough. Go somewhere else. Life isn’t fair. Sometimes you are the one in the room everyone is counting on to fix the problem, even if it isn’t your equipment that is causing the problem.
In the interest of brevity, let me close with some thoughts on how to ensure your performance is top notch.
1. Know what the scope of your job is. – This may seem a bit simplistic, but you need to be on the same page as management when it comes to your responsibilities. You cannot rely on someone else to tell you if that piece of network gear buried in some rack in a data center is your responsibility. You are going to have to find that out yourself and it needs to happen before the problem occurs. Hopefully your co-workers who have been there longer than you have a good grasp on what things belong to you. For example, if your boss expects you to take care of the wireless network, you better do it or have it handed off to someone else who can take care of it when a problem arises.
2. Develop your skills around your responsibilities. – I’m not advocating you abandon any sort of professional development that is not DIRECTLY related to your job. However, a BIG part of getting a pay check from a company is directly tied to being able to do your job as defined by the company. Good managers won’t load you up with things you are not able to do unless you have managed to con your way into a job by being a bit liberal with your resume. If you are stuck with something you are relatively new to, do the best you can and make sure your management KNOWS you are doing the best you can. Read books, configuration guides, white papers, and other technical documentation. Attend a training class. A career in IT is all about adaptation. None of us are working with the same hardware/software we were 10 years ago. If you are, odds are you either work for the government or a REALLY cheap company. Perhaps there are one or two things that have had a ten year plus shelf life, but for the most part, technology changes so fast that a decade is a lifetime in IT.
3. Be prepared. – Expect the unexpected. Think about different failure scenarios and design the network to remediate any single points of failure. If need be, have some block time purchased with an external consultant or VAR that has considerable experience with your specific hardware/software platforms. Carry maintenance contracts on all your hardware/software that is critical.
4. Raise any red flags early on. – If there are issues you know are going to be a problem, let someone know as soon as possible. Document those issues. Fix those issues. Even if the company says no due to budgetary reasons or some technical issue, at least you have done your homework and tried to make these issues known. If a problem does occur, nobody can come back to you and say that you should have known about this, or that it was your fault, etc. Additionally, it may work out to your benefit as management typically appreciates people who just want to make things better and do what is right for the stability of the network.
5. Stay calm during the outage/problem. – Remember that in a lot of people’s eyes, it is always the network that is at fault. Don’t let that get you down. Stay focused and work on the problem at hand. Ask as many questions as needed to get an idea of what the scope of the problem is. Don’t be afraid to ask very basic questions. One of the best ones to ask is “What changed?” or “When did the problem start?”. Maintain professionalism at all times. I get upset when I am on a conference call and someone won’t stop moaning about why it’s not their fault long enough for me to ask a question or answer one. However, it’s rather immature and unprofessional for me to lash out at them in anger even if I know it’s not my issue. There ARE times when I have had to tell someone to stop talking so that I could either answer a question or ask one of someone else on the call. I hate having to do that, but sometimes in the interest of getting it fixed YOU HAVE TO. If you are dealing with an issue where people are congregating around your desk watching over your shoulder, try and tune them out. You can’t always tell them to get lost or to leave you alone. You have to learn to work under pressure, but if you have taken item number 2 to heart, you should be able to minimize the time these people are hovering near your desk.
6. Be humble. – If people know that you don’t know it all, they tend to cut you a little more slack. If you are condescending and treat people like garbage because they don’t know the difference between a “routed” protocol and a “routing” protocol, they will be very unforgiving of your mistakes. Remember, there is no way possible you can know it all. There are people out there who know far more than you do. Sometimes they are in the same room as you. If you save the day and score a touchdown, good job. You don’t have to do the happy dance in front of everyone if you figure out what caused that routing loop. Your actions will speak for themselves. On the other hand, if you storm into the room demanding people shut up and watch you perform, you better get it right. If you don’t, your stock just went down and at some point, you’ll be looking for work elsewhere.
Ask any athlete how hard they have to work in order to get to their peak performance level and you’ll no doubt hear a recurring answer. You will find that it took a lot of time and effort to get there. There are no short cuts. When the wide receiver catches the ball and runs 80 yards to the end zone for a touch down, you can bet he ran sprints hundreds of times in the months prior. When the quarterback throws the ball for 50 yards and drops it right on the chest of the wide receiver, you can bet he threw that same pass hundreds of times in the months prior. When the defensive end wraps his arms around the running back and slams him to the ground, you can bet he practiced on a tackling dummy hundreds of times in the months prior. The examples go on and on. Peak performance takes time and effort. You practice and refine your skills for what is usually a short performance. Sometimes the performance extends over a couple of days or weeks, but generally issues get diagnosed and resolved in a relatively short time. How you prepare will determine the outcome. If you take shortcuts, expect poor results. If you put in the effort to perform well, good things will come your way. Granted, you probably won’t get a multi-million dollar contract with company X, but how many football players do you know who understand cool stuff like policy routing and VRF’s? Oh, and being able to fix problems on the network quickly leaves you more time to play World of Warcraft.
I have a bit of a problem when it comes to information. I tend to resemble someone on the TV show Hoarders. I have loads of PDF files on my laptop. Some are on my iPad. Some are on my desktop PC. I even have some on a little flash drive I carry around in my pocket. Of course, I have plenty of books. Just for networking related stuff, I have a pile at home as well as a good size collection at work. Then there are the URL’s. Every day I save all of the valuable URL’s I have discovered from Twitter and RSS feeds and put them in their own little folder with the date as the name under my bookmarks in Firefox. If I follow you on Twitter and you post a link, odds are I have looked at it and bookmarked it if it is something that pertains to my interests. If I read your blog, and odds are I do, I will bookmark various posts of yours and at some point go back and reference them. You see, I don’t always have time to read everything during the day. Additionally, if it is a post like this, or this, I will have to go back and read it all when I have a considerable amount of free time.
Therein lies the problem. I never seem to have time to go back and sift through every thing like I had planned. Well, that’s not entirely true. I have the time. I just get caught up in all the new links that are posted on Twitter every day and wind up spending study time skimming new blog posts or digging through websites. There’s a lot of good info out there that people are sharing. I suppose I could limit my intake to just routing and switching, but what fun would that be? Besides, I don’t want to be ignorant of the other things that are out there. After all, it wasn’t that long ago that I had absolutely nothing to do with voice, storage, wireless, and security. Times are changing, and changing fast.
There’s just so much out there that needs to be absorbed. Just when I think I have a handle on most of the Cisco product line, they go and release UCS, and the Nexus 1000V, and the ASR1000’s, and Clean Air. It never ends. There is always a new technology or some new hardware to read up on.
The realization I have come to is that there is no use in collecting information if you are not going to use it. All of those PDF’s, books, and URLs will do me no good if I never use them. At the same time, if I stop keeping up with what is current, I will fall behind and be of less help to my employer. I won’t be able to effectively design anything because I won’t be aware of what the possibilities are.
One of two things has to happen. The first option is that I can really narrow down the focus to just the things that directly pertain to my job. That will alleviate some of the information I have been hoarding. The second option is to start dedicating a bigger portion of my day to information consumption. I think option two is the best one as I can’t see myself ignoring products and technologies that I am not using today due to the fact that I may be using them tomorrow. Besides, it’s more fun when you have a wide range of technologies to keep up with as opposed to a handful.
I don’t know how everyone else handles their technical knowledge maintenance. If you happen to have a tried and true method of keeping up with all things networking, I would love to hear about it.
Have you ever bought a car without all the bells and whistles? You end up with some blank buttons in your dashboard. You’re not really sure what they are for, but there’s that little voice in the back of your mind telling you that you should have bought that feature. Of course, you can drive the car for years and never need that button. Or, you can flip through the driver’s manual and see just what it is that button does on the fully loaded model you didn’t buy.
Perhaps you are a student of automobiles and wouldn’t dream of buying a new car without knowing all the possible options or features. You make sure you buy exactly what you need. Nothing more. Nothing less.
What about features that you never knew existed? My mother drove a 1994 Mazda 626 for about 7 or 8 years. It was a pretty nice car, but it had a feature that I have not seen in any other car. The center vent could oscillate back and forth between the driver and passenger seat. There was a button on the center console labeled “Swing”. Push the button, and the vents “swing” back and forth. Leave it off and the air blows in the direction you have the vents turned. Before I saw this, I had no idea such a thing existed. After I saw it, I looked for it in every car I drove or rode in. Not long after my mother bought her 626, I bought a Mazda Protegè. Sadly, I did not have the “Swing” button as an option on my car. Although I drove that car for a good 8 years or so, I never forgot about the “Swing” button letdown and felt as if my car was inferior. My mother moved on and bought a Mazda Millenia. That was the step up from the 626. The flagship car of Mazda, much like the Toyota Avalon or Chrysler 300. Sadly, the Millenia lacked the “Swing” feature in the AC vents, but it did have blue colored gauges at night on the dashboard. Now the “Swing” feature didn’t seem as cool next to the blue colored gauges and dials. One of my friends had recently bought a Volkswagen Jetta around the same time. He had blue colored gauges and dials as well. Of course, his CD changer was in the trunk or boot, and I was not a fan of that feature at all.
Now that I have exhausted my knowledge of automobiles, let me relate this to what you and I do for a living(or at least I assume you do the same thing as me). Features come and go. Some are neat and have a practical purpose. Others are just there. Eye-candy. Nothing more. Sometimes what we need is not the same as what we want. Sometimes we don’t want something until we find out it exists. Ahem, iPad anyone? Now before any Apple fan-boys or fan-girls jump down my throat, I must admit that I own one. I bought one recently and have decided that if my house were burning down and I had to choose one item to take with me in addition to my wife and kids, it would probably be my iPad. 🙂 Having said that, I was perfectly fine living with a laptop and desktop PC at home prior to the iPad’s debut. Once it was marketed to me, and I must say it was marketed rather well, I needed one. Not wanted. NEEDED.
I’m getting away from what I wanted to focus on and that was features, versus an entirely new product, but you get the point. There are a lot of neat little things out there that one vendor does over another. However, I wonder if those particular features are REALLY something we need. Do I REALLY need something like OTV? Some people will say yes. Others will say no. I would say it depends. What were you doing prior to OTV? Although my main focus is on network hardware and software, the same holds true for features in software and hardware outside of the network space. In the case of security, sometimes features can actually end up being vulnerabilities or additional entry points that you have to lock down.
So what is my point in all of this? Well, I am not going to give you answers because to be quite honest, I don’t have them. Remember, this is a blog about network therapy. A big part of therapy is simply stating the problem or concerns. Here’s what I think. If you are spending a lot of money on something, make sure you need what you are buying. Not want. Need. Yes it takes time to go through everything, but that’s what you get paid for. Don’t buy a Lamborghini if a Kia will suffice. If you need the Lamborghini, make your case and get it. Don’t settle for the Kia. If you absolutely have to settle for the lesser due to decisions made above your pay grade, then put in writing your concerns about why the Kia is not sufficient and move on. I know I said I had no answers, but I do have some suggestions. It’s better than a kick in the head, and it’s free, so take it for what it’s worth.
1. Only buy what you need or will need in the near future. You’ll want to consider future requirements as well (ie expansion, features needed down the road). It is often hard to predict the future, but do the best you can.
2. Careful consideration of how to spend company dollars will ultimately reflect good things about you or your particular group. You don’t want to be known as a money wasting group or person.
3. Careful attention to features will help you navigate the difficult waters of vendor selection. This is one of the harder things to master. If you know what you need and are relatively aware of what the major vendors are doing, product selection along with the right feature set becomes a bit easier. For example, check out this article by Greg over at etherealmind.com. If you ONLY live in the Cisco 3750 world of stackable switches, you might miss the fact that Juniper can do the same thing, but extend the logical switch over a LOT longer distances. This is but one example. There are many more like this out there. Go find them and buy them, but only if you must. 😉
Thankfully, most network hardware comes with a ton of features by default. It’s usually the higher end stuff that we talk ourselves into buying and don’t necessarily need. I’m looking at you VSS. I’m not saying there isn’t a place for it. I use it and think it is some pretty cool stuff, but I wonder if the expense is worth the benefit sometimes.
If you are a consultant, ignore everything I just said(or “wrote” if you want to nitpick). You make your living off of selling services and equipment. You are exempt. However, if you are the reason a 10 user network of office workers have dual 6513’s with Sup720’s, ACE, FWSM, and WiSM, you should be ashamed of yourself. In that case, I can simply quote Jesus: “Go and sin no more.“.
Maybe you’ve experienced this before. You are minding your own business without a care in the world when all of the sudden the phone rings.
Them: There’s a problem and we think the network is the cause. Can you check it?
You: Check what? The network?
You: Which part? What am I looking for?
Them: Any sort of problem.
(Fast forward an hour or so later)
You: Well, I ran a packet capture on the switch port connecting to system XYZ. I see a bunch of TCP resets coming from your server.
Them: Okay. We’ll take a look.
(Fast forward another half hour or so)
Them: It looks like we found the problem. Process blah-blah-blah was failing due to a dependency on process ha-ha-ha. We reset the services and everything is working again. Thanks for your help.
You: Okay. Not a problem.
(Back to life as before)
Sound familiar? If you have been in networking for more than a couple of years, this should invoke all kinds of warm and fuzzy memories. Meals were missed. Plans were canceled. Sleep was lost. All in the name of defending the network’s honor. Oh yes. This is the part about a career in networking that is conveniently left out of the brochure you are given before signing your life over to Cisco/Juniper/Citrix/Aruba/Nortel/F5/Brocade/Alcatel/etc.
I have seen more than my fair share of these incidents. With the exception of a brief stint in consulting and about 2 years doing things in the US military that you’ll never do anywhere else, I have lived my entire IT existence in the “corporate” setting. By that I mean chained to a desk looking over logs and configurations. Slaving away on the same network for years on end. Getting to know the lay of the land in the same way one knows all the sounds an old car or house makes. In short, after you work on a certain network long enough, you can see into the guts of it like Neo can with the Matrix.
If you are like me, you have a certain affinity towards your network. Sure, it may need some help with cabling or a cleaner route table, but you work with what you have. You make changes as you can. You replace hardware as the budget allows. You care for it like a farmer does his corn fields. Is this creeping you out yet? Well it shouldn’t. There are plenty of people out there who love their networks even to the point of showing them off to the world.
Here’s the problem with being a networking engineer/administrator/architect/designer/janitor. You have to understand everyone else’s piece of the pie, but not too many people have to understand yours. Fair? No, it isn’t, but as an officer I once worked for in the military told me: “That’s a burden you have to bear.” He was right, even if I didn’t like hearing it. That is not to say that all other entities within IT or greater corporate America are completely clueless when it comes to networks. Quite the contrary. There are plenty of systems people who understand networks very well. You can give them an IP with a classless subnet mask and they don’t even bat an eye because they know exactly what you mean when you say it’s a slash 26 network. However, when it comes to “applications” people, my experience has been that they only have to know their piece of the pie and can conveniently blame the network when a problem arises. I know what you’re thinking. Did he just paint all applications people with a broad brush? Yes. Yes I did. Of course, if you happen to be an applications person, I meant everyone else. Not you. 😉
That brings me to the title of this mini-rant/post. You can plead your case before everyone telling them that it probably isn’t the network, but they’re not going to believe you. Why? A lack of understanding or a lack of visibility into your world. You see, the network is just a big murky box to them. Maybe if they had access to some monitoring platforms they could be swayed, but unless your monitoring package can go down to the transaction level like Compuware’s Vantage product, you’re still going to have some explaining to do. However, in a way that I cannot begin to explain, people tend to believe packet captures. Don’t ask me why. I can tell you until I am blue in the face that the switches and routers on the network for the most part could care less what your payload is and you won’t believe me. You may not even understand TCP, UDP, and the rest of the acronym soup being tossed around, but for some reason, Wireshark or tcpdump results are more credible than Steven Hawking discussing time travel. If you want some good laughs around things like this, follow this guy on Twitter. He seems to deal with this on a regular basis and has some hilarious tweets to show for it.
Let me end this post with the following suggestions:
1. Get familiar with interpreting packet captures. Wireshark is the most well known packet capture utility for Windows boxes out there. There’s even a good book out there that covers everything in detail. You’ll also need to know about TCP and how it works. There are other protocols like UDP and ICMP that will be good to know, but TCP is by far the most useful protocol to know and understand when dealing with packet captures. For some good info on TCP, see here.
2. Don’t be afraid to run a packet capture early on in the troubleshooting process. I am finding that this tends to solve the problem when all other methods fail.
3. Don’t EVER, and I stress EVER, state emphatically that there is no way possible that the network is at fault. 99 times out of 100 you may be right. Get it wrong 1 time, and everyone will be gunning for you. There’s always the possibility that the network is at fault. Even when everything you know is telling you that it isn’t the network, if you don’t have a packet capture to back it up, you’re wasting your time.
4. Educate your co-workers about the network, or networking in general. Try to do this without condescension. Nobody wants to listen to Nick Burns tell them how stupid they are. The more people know, the less likely they are to hurl unsubstantiated accusations your way that you are manipulating traffic to break their application. It makes every organization a lot stronger when education is provided from the various departments. Please understand that although you and I might get excited when talking about routing protocols, not everyone else will. Oh how I wish my wife and I could have the EIGRP vs OSPF discussion, but it’s just not going to happen. Some people are not going to want to know a whole lot about the network, so try and figure out how much they really want to know and tailor the education to that level.
If nothing else, looking at a bunch of packet captures will help you appreciate what is going on behind the scenes every time you read an e-mail message or look at a website. Although other people might not appreciate it, I find that it helps my wife fall asleep faster when I talk about the various TCP flags and why they are used in data transmissions. At least she will never blame the network. 🙂
***Note: I am asking this question from a corporate IT perspective. I am not asking this from the standpoint of a vendor or reseller.
Most of what I do in the networking world revolves around one vendor’s equipment. Not all, but most. Can you guess the vendor? 😉
Do we buy most of our equipment from vendor XYZ for any of the following reasons?
1. We are comfortable with it.
2. Their products work.
3. The support is good. Documentation is abundant and detailed.
4. They have the most features.
5. Their cost is lower.
6. There is a large talent pool out there that knows their products.
7. They provide a complete end to end solution.
8. They are a financially stable company.
9. They get great reviews from all the trade magazines.
10. No other company has this particular technology/protocol/gadget.
11. They always buy us a great lunch and take us to sporting events for free. (Or some variation of this.)
12. We want one throat to choke if there are problems.
Perhaps some of these apply to you in terms of your relationship with vendor XYZ. I believe that some of those things are very valid reasons to buy from vendor XYZ. Some of them are not.
The problem, as I see it, is that SOMETIMES what we buy isn’t necessarily the BEST solution for the company. Notice that I said SOMETIMES. There are plenty of times in which we buy from vendor XYZ because it is the BEST solution for the company.
There’s a lot to be said for vendor comfort level. I, along with many others, know a decent amount about the Cisco switch and router product line. I know a LOT less about every other vendor’s switch and router product lines. Just for fun, over the past couple of weeks I have looked at other vendor’s switches and routers and tried to compare them to the Cisco line. It has been an interesting experiment to say the least. In the latest Packet Pushers podcast, Greg Ferro of etherealmind.com mentions something similar. Towards the end of the podcast he talks about how frustrating it is that other hardware vendors have the spec sheets for each model as a separate PDF. There’s no easy way to do a side by side comparison. See here for an example. I should point out that Juniper does have a “Compare Family Models” link on the main page of each product family but it is not a full blown separate page.
Let’s take switches for example. If I want to evaluate alternatives to the Cisco 3560 switch, how do I go about doing that? What vendors do I look at? There are easily a dozen vendors that I can look at. At what point do I draw a line in the sand and say that I am only going to look at 5 alternative vendors, or 3? Do I base the decision solely on features? Cost? Market share?
In regards to all of that, I would simply ask: “How much time do you have?”. My experience has been that doing something right takes time. If you don’t take the time to do it right, you’ll cut corners. One of the easiest corners to cut is in the vendor selection process. Just because a name is familiar doesn’t mean that it is going to be the best choice. It’s better to take the time and make the right choice than to buy what is familiar and wind up with bigger problems down the road.
Is it possible to stay vendor neutral? Yes, but it requires a lot of time and effort. Unfortunately, we don’t always have the time. I have pretty strong feelings toward certain product lines. Juniper’s SA line of SSL VPN appliances are nothing short of spectacular. HP’s Network Automation Software (CiscoWorks NCM) is an amazing product as well. There are several Cisco products that I could say the same thing about. Although I feel strongly about them, if someone were to show me a better product that was a better fit(cost,features,support), I would have no logical reason to oppose it. Business is business.
I have to be honest though. I have a certain inclination to lean towards Cisco many times during product selection. This is due to several factors that I listed at the beginning of this post. Two of the biggest reasons are the sheer amount of features their products contain as well as the generally large amount of documentation available for each product. Those two reasons don’t always hold true for all of their products, but more often than not, that is the case. Of course, for any substantial project(WAN optimization, wireless, IP telephony, firewall, network management), I would be foolish not to consider multiple vendors. For the smaller things, it just seems so easy to order a switch or two from Cisco. Is that me cutting corners? Well, as in anything, it depends. 😉
****Note – While I thought about detailing the technical steps necessary for delegation on different pieces of equipment, I decided to go with the more “architectural” or “philosophical” approach in this post. Besides, there are plenty of others out there who do a far better job with graphics and CLI examples.
Recently, I took some steps to make my job a little easier. I delegated access to another group that does not normally have anything to do with the network side of the house. In this particular instance, I was able to give that group access to a Cisco ACE load balancer. Normally, giving non-network people access to equipment would be frowned upon. This is especially true for equipment in a data center that controls data flows for your most important applications. I had to consider the following:
1. Can I give them specific levels of access?
2. Will they be able to perform operations with relative ease?
3. Does it make sense to do this?
Question 1 was easy. Of course we can provide granular levels of access. It is hard to find a piece of equipment on most enterprise networks that can’t do this. Question 2 was a “most likely”, but could have been tough if everything needed to be done via CLI. Question 3 was probably the most important. Generally speaking, most technical problems can be solved given enough time and resources(ie people, money, and equipment). What many of us should ask, and some of us fail to ask, is whether or not we SHOULD do something. I for one love playing with new equipment. Build an Ethernet switch that interfaces with a toaster and I want to play with it. However, is there any use for something like that? Is there a large community of people out there that want connectivity with their toaster?
The point, is that while a lot of things are possible, not everything is necessary. Sometimes giving people access to network equipment can cause more harm than good. While I am a big fan of wanting to provide as much information to others as possible, if that information cannot be interpreted correctly, you are wasting your time. For example, I have been in environments where non-network related groups were given access to Netflow data. While that sounds great on the surface, the reality was that the data was being interpreted incorrectly. When looking at something like a 3Mbps circuit, some people would see full utilization and assume that more bandwidth was required. What they failed to take into account was that the QoS markings of the traffic indicated that a bunch of AF11(what was deemed scavenger) traffic was using the bulk of the bandwidth. Had any additional traffic come over the circuit that was tagged as AF21 or higher, it would have pushed down the AF11 traffic and gradually used more and more of the circuit until it reached the bandwidth limit that was set for that specific class of traffic. More bandwidth was not needed when the Netflow data was viewed in its entirety. Had this particular group understood QoS markings, they would have come to a different conclusion. Could we the network group have provided more in depth training on this particular product? Sure, but how long would that training have to be before the individuals understood QoS well enough to interpret traffic flows correctly? If you are a QoS fan, how long did it take you before you understood things like shaping vs policing? Or L2 vs L3 markings?
Back to the issue at hand. Does it make sense to give another group access to the load balancer? Yes. In this case it did. The typical process for maintenance on a server getting requests via the ACE load balancer was to have the network group pull it out of the active pool. Then, another group would make whatever changes were needed. Once they were done, they would contact the network group who would place the server back into the pool. If you are having to make changes to a dozen servers, this process can take some time. Why not just give the group making changes to the server limited access to the load balancer so they can do everything themselves? Time and resources would be saved by all.
That brings me back to the second question of can we make it easy for them to make changes to the load balancer? In the case of Cisco ACE, yes. We had an instance of Application Network Manager(ANM) running in our data center to help us. While I tend to be a fan of CLI (except in the case of the Cisco ASA), not everyone else is. Sometimes a GUI is far more helpful for people who need to make changes to network gear. That’s where ANM comes in. In a matter of minutes, I was able to create a domain(which is where you define the servers and farms you are giving access to), and role(you can create your own if you don’t like the default ones) for this other group to use. Now they had access to select servers and their corresponding server farms, but not enough access to do any real damage.
After doing that, I just had to create some instructions for the 2 tasks they would need to do. First, they need to know how to remove servers from a load balanced pool. Second, they need to know how to add servers to a load balanced pool. With ANM and the specific domain/role I assigned to their group, this is a piece of cake. I took the appropriate screen shots to walk them through the process of adding and removing a server and put it in a nice concise MS Word document. There are times when I am hesitant to put a lot of pictures in instructions. Sometimes people get offended when you drop it down to an elementary school level. Thankfully this particular group LOVED pictures, so everything worked out. In about 15 minutes we ran through the instructions. Additionally, I asked if they wanted a bit more detail about the Cisco ACE load balancer in general, so we talked about what it does and where it sits in terms of its physical place in the network. Everyone seemed happy with the training, and I think they were truly excited about not having to wait on the network group anymore when they needed to make changes.
Problem solved. Everyone was happy, and I know that outside group is reaping the benefits of being able to make changes on their own. I have jumped on to conference calls several times recently and noticed that servers were being added to and removed from load balanced pools without the network group having to do anything. The group I gave access to was taking care of it.
If you have the means to delegate processes to other groups, I would recommend that you do it provided it complies with any security and administrative policies your company or IT department has. You do have those policies in place right? 😉 If it makes your job easier, makes other people’s jobs easier, and you get to impart some knowledge about the network to external groups, why not do it?