Dealing With Knowledge Gaps

Image courtesy of http://www.freedigitalphotos.net

Inevitably, we are all going to come across things in our jobs that we are deficient in. Maybe we know a little about a certain topic, but we need to know more. Maybe we know absolutely nothing and need a basic introduction to the topic. Regardless, there will come a time in which we need to increase our knowledge and understanding of something in this ever growing world of networking or just IT in general.

The problem as I see it, is how I go about filling in those gaps. When you just start out in the IT world, you may not have a good methodology in which to learn about IT things. If you have been in the industry for a long time, you may already have a good system that works for you. No matter which category you fall into, the fact that you will constantly have to learn is unavoidable. There are NO exceptions to this rule. If you wish to be at the top of your game in IT from a technical standpoint, you must make a habit of constantly learning new things. Failure to do so means that your knowledge will become dated and you will drift off into obscurity working as some corporate slave in a dark and dreary cubicle. This may or may not involve working for the government. 🙂

Now that we have established that static knowledge is a dead end, let’s look at how to ensure we are always at the top of our game. I offer you the 5 step plan. Others have 12 step programs. Maybe some have less. I only have 5. I am all about efficiency…..and my program doesn’t cost you a dime.

1. Examine your current level of knowledge. – How much do you already know about the subject in question? The answer to that question is going to dictate the kind of resources you use. Let’s use BGP for example. If you need to learn about the basics of it, there are a few good books that can handle that. There are also plenty of websites with white papers and blog posts that give a generic overview of BGP. There are some classes out there that will accomplish the same thing. However, there are quite a few books and white papers that will completely blow your mind if you don’t already have a decent understanding of BGP. The service provider side of BGP comes to mind. Enterprises and service providers use BGP in VERY different ways.

2. Find out where the information is. – For starters, you need to identify what kind of learner you are. Some of us are visual learners. Some of us are audible learners. Some of us learn by doing. Perhaps you are a mix of several different methods. Only you know what works best for you. If you need a lot of pictures and the topic is relatively mainstream, maybe a visual CBT(computer based training) course is what you need. If that is the case, I highly recommend you check out CBT Nuggets. If what you are looking for is somewhat more obscure, then I would recommend asking other people who do what you do. There are a variety of resources in which you can ask these questions like LinkedIn, forums, or Twitter. I prefer Twitter because it is a lot quicker. The only possible problem would be having enough people see the request. If you are new to Twitter, or very rarely use it, you may not have many followers who would see your message. Feel free to engage others in a substantive manner and over time your followers will grow. If all you do is tell everyone what you ate for lunch or what the weather is like in your part of the world, you probably aren’t going to get anywhere. If you absolutely refuse to use something like Twitter, then consider posting on Cisco’s forums if your issue is of a Cisco nature or networking-forum.com. There are other forums out there as well as mailing lists(NANOG comes to mind). All of the major vendors have support forums as well. Keep in mind that you may have to sift through tons of information before you finally find the information you are looking for. There is not always going to be a technical paper or book that explains exactly what you are looking for. Sometimes you have to piece it together from multiple sources. Actually, I would recommend that you use multiple sources unless it is some vendor specific thing that you can only get in one place. I have found out that you cannot trust a single source for 100% accuracy. Not that all sources are wrong, but imperfect human beings write books, white papers, and blog posts. Other imperfect human beings double check these same sources. When the content is of a technical nature, things get missed. This is especially true for the deeper technical things.

3. Execute. – You have all of the appropriate resources identified. Now you just need to get that information into your head. There are no shortcuts. While I wish I could learn kung-fu like Neo did in The Matrix, it isn’t going to happen. You have to put in the time required to absorb all of that information. Sometimes it can be done in a matter of minutes. Sometimes it takes weeks.

4. Ignore any distractions. – In the course of your learning, you are bound to come across something else that is interesting or neat. Resist the temptation to get sidetracked and stay focused on the main thing you are trying to learn. If you want to go back at another time and research the other items that pop up, then make a note of them. By focusing on the main thing you are trying to learn, you have a better chance of retaining information then if you start going in 100 different directions with every new thing that appears.

5. Allow the information to digest. – Sometimes it helps to simply think about things. Just go over it in your head. I tend to do this in conjunction with step 3. If I need to absorb a large amount of information, I like to take it in pieces and digest it little by little. By stopping to sort things out in your head, you can really come to terms with what makes sense and what doesn’t. I am very thankful my current employer allows me the freedom to do this. While it may look like I am spacing out on any given day in my cubicle, lots of times I am just thinking about something I just read or watched. It’s my way of performing a “write memory” on my brain. One of the other things I will do is drive to and from work in complete silence. That really helps because all I have to focus on is not crashing the car, which is relatively simple.

**Note – When asking others about a certain technology or product, do yourself a favor and research it first. Try and figure some things out on your own. This isn’t so much a problem with people who have been in the industry for a number of years as it is with those who have only been in IT for a few years or less. It’s not that people don’t want to answer the question. There will always be someone who will just blurt out an answer. The issue with asking without having done any research on your own is that you miss out on a great opportunity to develop your own research methods. There’s a reason that lmgtfy.com was created and is often quoted on Twitter. It has been my experience that those who last in IT are the ones that only need a nudge in the right direction. They don’t want their hand held. They just want a sanity check every now and then. The people who never want to put in the time or effort to figure something out and habitually want you to solve their problems are the ones that won’t make it in the long run. Well, they might have a job, but they won’t be anywhere near what they could be if they put forth some effort.

I am not going to make the bold claim that the 5 steps I laid out will work for everyone. They work for me when I follow them, and I don’t always follow them. I find the instances in which I have tried to cram something new into my head without following these steps ends badly. I forget something and have to start all over again. When I take the time to really dig into something and not rush it, it tends to stay with me at least from a conceptual point of view.

Chasing the “Ah-ha!” Moments

Image courtesy of Wikipedia

Whenever I talk to people who are just getting started in networking, there’s a part of me that wishes I was in their shoes. I say that because I know several of the things they are going to learn or figure out in the next couple of years and I remember having to go through the same process. Before I understood variable length subnet masks(VLSM), the numbers in the subnet mask field of a workstation’s TCP/IP settings didn’t really mean a whole lot to me. If someone used slash notation(ie /24, /16, /27), I had no idea what that meant. Like a lot of people, I relied on someone to tell me what the subnet mask was. However, once I learned about VLSM, it was as if a whole new world opened up. That was one of my absolute favorite “Ah-ha!” moments. You’ve had those yourself haven’t you? It is the point in time in which a certain technical concept just clicks in your head. You go from not really understanding it, to comprehending it. In fact, it’s almost as if that concept is only represented in binary inside your head. You go from a 0 to a 1 with no in between.

As you progress along in networking, more and more of these “Ah-ha” moments come. Unfortunately, over time they become fewer and fewer. That’s not to say that they go away completely. They don’t. They are just harder to come by. I’ve found that I am able to keep a steady stream of these “Ah-ha” moments coming as long as I look at technology without taking anything for granted. What I mean by that is that I don’t assume anything when it comes to trying to understand a protocol or technology. What I “think” I know might actually be wrong. My understanding might only be partial. I have to continually ask “why/what/how/when/where” when dealing with technology.

Let me give you a personal example. I have known for many years that a T-1 is 1.544Mbps in terms of bandwidth. It is comprised of 24 64kb channels. The only problem is that 24×64,000 is 1536000 and not 1544000. Oops. Where did the other 8k go? To further drive this home, a “show interface” on a serial link that is configured as a full T-1 shows the interface bandwidth to be 1536kbps. Why the discrepancy? I could have just moved on and ignored the reason behind the discrepancy. However, by researching the issue and figuring out what the issue with this discrepancy was, I learned a whole lot more about T-1’s. I learned how alarms over the circuit get propagated. I learned what the extended super frame(ESF) actually was. In other words, had I not been curious as to why the math didn’t add up when it came to T-1 bandwidth, I would be far more deficient in the inner workings of the T-1.

In the spirit of chasing the “Ah-ha” moments, take a look at the 4 questions below. Go find the answers if you don’t already know them.

1. Why is MPLS faster than conventional IP based routing?
2. What are the differences between a multi-layer switch and a router?
3. Why do you need different antennas for wireless access points and where would you use each antenna type? Sure, this is rather open ended, but what I am getting at is the radiation pattern of each antenna.
4. How does traceroute really work? Not just the TTL mechanics, but look at the various ICMP type codes as well.

Can you remember the last “Ah-ha” moment you had? If not, why? If so, does it make you want to go out and find more of those moments?

ACE Boot Camp – Days 3 and 4

Days 3 and 4 did not disappoint! I don’t know if I stated this in the earlier posts, but the days basically consisted of lecture in the morning and labs after lunch. I REALLY, REALLY enjoyed the lecture portion. Again, I have to state that the instructor was fairly knowledgeable in regards to ACE, so he was able to actually teach instead of regurgitate a slide deck like other classes I have been in. That makes all the difference in the world. As for the labs, I guess they do some good if you have not had much experience with the ACE CLI. We did not do any labs using the built in GUI or ANM. The problem I have with labs is that they are a very canned and controlled environment. You end up just going through the motions without actually soaking up what it is that you are doing. Ideally, the labs would need to be tailored to your environment to have the greatest effect. This of course, is not realistic. Having said that, I am sure there are some people who get something out of it. My opinion was shared by others in the class in regards to the effectiveness of the labs, so I am not the only one who feels this way. However, the effectiveness of the lecture portion completely overshadowed any shortcomings of the lab portion.

In the interest of brevity, I am going to touch on the things I thought were the most interesting, but I don’t want this post to be so long it requires a coffee break to finish.

Route Health Injection – On a simplistic level, RHI allows the ACE to inject a host route into the network. You would use this to advertise the VIP(virtual IP) that clients use to connect to a server farm. If the server farm is not available due any number of issues, the host route can be automatically removed from the route table and not advertised. The alternative is to simply advertise the VIP’s as part of a regular subnet advertisement like you do with any other VLAN or subnet. Again, I am simplifying this and need to point out that this is NOT something that is specific to Cisco ACE. Other vendors implement similar technologies.

KeepAlive-Appliance Protocol(KAL-AP) – There’s a few variations of the Cisco ACE, and one of those is the Global Site Selector(GSS). Its purpose is simply to provide higher level load balancing between data centers. Basically, it is a load balancer of load balancers. By using KAL-AP, the GSS can query VIP’s at multiple data centers and determine which one is the best fit to send traffic to.

There are a couple of things that the ACE 4710 appliance does that the ACE module cannot. I asked the question as to why this is the case and was told that the ACE appliance has different architecture than the module. It has certain functionality that might come to the module at some point, but for now is restricted to the appliance. These extra functions really revolve around the ACE appliance being able to cache certain HTTP objects and speeding up the process of delivering a web page to an end user. A fair amount of detail on this can be found here.

It sure seems as if I cut back on the information from days 3 and 4 when compared to 1 and 2. I did. Although there were plenty of interesting things covered in the past 2 days of class, a lot of those things would take a while to explain and draw out via diagrams. That’s also assuming that I actually understand these things well enough to explain them in depth.

That brings to me to a more philosophical point in regards to the type of niche product that Cisco ACE is. While it would be great if you knew the CLI on ACE backwards and forwards, it really isn’t necessary. What is necessary is an understanding of what a platform like ACE is capable of. I sat in a meeting today in which some developers wanted ACE to perform health checks on a server outside of a load balance pool and use the results of that query to determine whether or not servers should be removed from a load balance pool. Basically, they wanted to do something that ACE is not really designed to do. Spending 4 days in a classroom learning all about ACE gave me the information needed to have a productive meeting with these developers today. I was able to answer their questions and give better guidance than I would have a couple of weeks ago. I don’t know all the commands for ACE. I will still have to use the configuration guides to look things up now and again. The important thing is that I understand the capabilities and limitations of the ACE load balancer a lot better today than I did prior to taking the ACE class. My main goal is to know what it can and cannot do in order to design anything requiring load balancing properly. To me that is more important than memorizing commands.

ACE Boot Camp – Day 2

Day 2 of ACE boot camp did not disappoint! Another full day of lecture and labs. We covered the following topics:

Modular Policy CLI
Managing the ACE Appliance and Service Module
Security Features
Layer 4 Load Balancing
Health Monitoring

I’ll cover some general things about each topic and go into additional details on the points I thought were interesting.

Modular Policy CLI – ACE classifies which traffic it will load-balance based on policy maps, which are comprised of class maps. If this sounds a lot like how you build QoS policies on IOS based routers, it is. The big difference is that ACE is far more restrictive in what those policies contain.

Managing the ACE Appliance and Service Module – Like most Cisco devices, ACE can be managed in a number of different ways. Telnet, SSH, HTTPS, and SNMP. You can even use the XML API if you want. With SNMP, versions 1 and 2 cannot understand contexts. SNMP version 3 can. In order for SNMP version 1 and 2 to work with contexts, you have to use the community string format of “community@context” where “community” is the community name and “context” is the name of the virtual context. When the GET, SET, or whatever SNMP action you choose hits the ACE, the “@context” portion is understood and passed along to the appropriate context.

Security Features – There are a ton of different ways to restrict traffic entering and leaving the ACE. Most of the time you will be focused on traffic entering the ACE. As with applying ACL’s to interfaces on switches and routers, very rarely will you see access lists applied in the outbound direction. That feature is there in case you have some special need to use it.

An interesting capability that the access lists have in ACE is the ability to use object groups to identify which traffic to permit or deny. If you have ever worked on the PIX, ASA, or FWSM, you will be familiar with object groups. They make traffic identification much easier not to mention the simplification of the ACE configuration itself.

The much more granular security options were of great interest to me. Take something like IP fragmentation and reassembly. You can specify the max number of fragments allowed from one packet. If it exceeds the number you specify, you can just drop the traffic. Many other options exist with regards to the packet stream itself. You can enforce certain flags from being set. If violations occur, not only can you drop the traffic, but you could actually reset the flag itself and then send the traffic through the ACE. While most options are configurable, there are some rules that are always enforced. For example, the source IP of a traffic flow can never equal the destination IP.

Layer 4 Load Balancing – This is exactly what it sounds like. Load balancing based on TCP/UDP flows. I think the neatest part about this particular topic was the fact that you can actually load balance traffic across multiple firewalls and have the return traffic come back through the same firewall. This of course requires an ACE on both sides of the firewall, but withe ability for the ACE module to have up to 250 virtual contexts, it doesn’t have to be 2 separate physical ACE modules. The same module can host both contexts that live on either side of the firewall. It is fairly clever how they make this work. Essentially, when traffic comes from one firewall into the ACE, it remembers the MAC address of the sending firewall and places that connection in a state table. When traffic comes back through the ACE, it already knows which firewall to send the traffic to based on that state table. I’m not sure I would want to use an ACE module for load balancing through firewalls, but there are plenty of customers out there that are already doing it or could see the benefit in doing something like that.

Health Monitoring – If there’s one thing the ACE seems to have a fairly large amount of options on, it’s the health monitoring or probes. All the major protocols have specific probes on the ACE that are used to check the health of the back end or “real” servers. This is way beyond the load balancer simply pinging the server to make sure it is up and running. Let’s say you used the HTTP probe. Instead of just trying a simple ping to check a back end servers’ status, the HTTP probe can actually go out and make an HTTP connection to the server or serverfarm. That’s a far more intelligent way to query server status. Based on the probe results, any number of things can be done to the various serverfarms and servers ACE may be providing services for. They may be taken out of active status, have their priority reduced, etc.

There’s a LOT more to this stuff. This was only day 2 of 4! More to come.

ACE Boot Camp – Day 1

First off, let me point out that this is not a boot camp with a certification in mind. It’s a 4 day course given by Firefly Communications. Although I booked the course through Global Knowledge, I was told that they typically outsource their data center courses to Firefly. Works for me. As long as it is quality training, I don’t care if you outsource it to Elbonia. I am assuming they use the term “boot camp” because it is an end to end ACE class taught in just 4 days.

Which brings me to my first point. My company was able to use Cisco Learning Credits to pay for this class. At 30 credits, that translates to $3,000 US dollars for 4 days worth of training. Sitting in the class, I couldn’t help but notice people doing regular work while the instructor was going through his lecture. I realize most places are understaffed. Outages happen. Fires have to get put out. However, $3,000 for 4 days to me is a big deal. If you send your employees off to training that is critical/applicable for their job, LET THEM TRAIN! Leave them alone while they are there. Of course, that’s a 2 way street in that some employees need to learn to let go as well. The company will function without them for a few days. You can turn off “martyr” and “hero” mode for a couple of days. I am checking e-mail at night, but not being obsessive about it. I have very capable co-workers who can do anything and everything without my help.

Now, on to the actual class. Let me begin by commenting on the quality of instruction. I’ve been to plenty of poor classes in which someone was trying to shovel test material down your throat the whole time. I’ve also sat in several classes where the instructor was obviously out of their league and could not field questions from the crowd that weren’t covered on the vendor approved slide deck. That is simply not the case with Firefly. My instructor is very competent and when he hits the limit of his knowledge, he indicates that. So far, I think I have only seen 1 time out of the dozen or so questions he was hit with today in which that was the case. I guess that is what $3,000 a seat gets you.

It seems as if there is a fairly decent mix of people in this class. About a dozen or so in attendance. A fair amount of them are actually using the ACE 4710 appliance which I thought was rather interesting. Of course, most are using the standard ACE module. There are varying levels of experience with ACE as well. I was under the impression that I would be here mainly for the second half of the class, as I felt comfortable with the basics. Of course, just when you go and get comfortable, you realize how little you know. I learned a LOT today. Mainly, it was about things I never really bothered to dig into. You see, like most people, we probably only dig into the features we absolutely need right now. Maybe we plan on coming back and covering everything else at a later time, but I think that happens far less than we’d like it to. Some of the things we covered today that I was horribly deficient on were:

Resource Management – If you use multiple contexts, RM can prevent a single context from taking over the entire resources of the module. I don’t use this as it is currently not a concern, but good to know if things change!

HTTP Message Structure – 3 fields make up each HTTP message: Start/Request line(includes the METHOD), Header fields, and Body(which is optional)

ACE 4710 appliance – I don’t use it and never have. However, it does do a few things the module does not mainly centered around application acceleration. We have not covered that exhaustively yet, but I will take good notes when we do.

There were other things covered in which I was glad to get a decent refresher. The main one being TCP sequence numbers. They are always a bit confusing to me if I don’t study them on a fairly regular basis. Although you weren’t there with me in class today, you can read this post by Jeremy Stretch which talks about TCP sequencing. He even uses nice graphics!

We ended the day doing a pretty simple lab in which we created some contexts and messed around with resource management to see if we could oversubscribe the module in terms of CPU, memory, etc in regards to other contexts. Overall, it was a really good first day. I am eagerly anticipating what tomorrow will be like. It is also good to be taught by someone who actually helped develop the slide deck the course is taught from. He was able to add funny little details about how he created this drawing or that. It’s always nice to have someone teach who has a great sense of humor. So far, I give the Firefly ACE boot camp 2 thumbs up!

I am hoping to get a wee bit more technical in the following posts regarding ACE boot camp as the remaining days will REALLY focus on load balancing. Who knows? I might even post a graphic or two! Shocking isn’t it?