Archive

Archive for January, 2011

Be A Part Of History

January 27, 2011 Leave a comment

Henry V - Image courtesy of Wikipedia

He that outlives this day, and comes safe home,
Will stand a tip-toe when this day is nam’d,
And rouse him at the name of Crispian.
He that shall live this day, and see old age,
Will yearly on the vigil feast his neighbours,
And say ‘To-morrow is Saint Crispian.’
Then will he strip his sleeve and show his scars,
And say ‘These wounds I had on Crispian’s day.’
Old men forget; yet all shall be forgot,
But he’ll remember, with advantages,
What feats he did that day. Then shall our names,
Familiar in his mouth as household words-
Harry the King, Bedford and Exeter,
Warwick and Talbot, Salisbury and Gloucester-
Be in their flowing cups freshly rememb’red.
This story shall the good man teach his son;
And Crispin Crispian shall ne’er go by,
From this day to the ending of the world,
But we in it shall be remembered-
We few, we happy few, we band of brothers;
For he to-day that sheds his blood with me
Shall be my brother; be he ne’er so vile,
This day shall gentle his condition;
And gentlemen in England now-a-bed
Shall think themselves accurs’d they were not here,
And hold their manhoods cheap whiles any speaks
That fought with us upon Saint Crispin’s day.

Such are the words that William Shakespeare penned in “Henry the Fifth”. They come from the Saint Crispin’s Day Speech that King Henry V gave prior to the battle of Agincourt in 1415 where the English defeated the French and King Henry ended up with a French princess named Catherine as one of the spoils of war. Although the speech from Shakespeare is made up, it is still a beautiful combination of words that express the pride the English soldiers would feel in the years after the battle. Others might forget what went on, but the soldiers would never forget. They would be a part of history. Which brings me to the point of this post……

The other day Jeremy Stretch mentioned this on Twitter:

“regardless of your thoughts on IPv6 adoption, it’s a pretty interesting time to be a networker”

That’s putting it mildly and it got me thinking about the changes going on in networking these days.

1. IPv6 Transition – Certainly you have heard of IPv6 and the coming IPv4 address exhaustion. If not, you need to get out more.

2. Virtual Networking – With the explosion of vmWare and other virtualization vendors in the past several years, a fair amount of traffic is cruising around “virtual” switches inside physical servers. Guess what? You still have to manage it. You still have to secure it.

3. Wireless Explosion – Everything is wireless today. Cameras, printers, phones, tablets, laptops, and other wireless capable devices are growing in number each year. If you aren’t familiar with wireless, you better be soon.

There’s more. Storage traffic riding over the same wire as voice, video, and data. How about link encryption on your internal switch/router infrastructure? Don’t forget the rush to flatten datacenter networks to L2 courtesy of TRILL or each vendor’s implementation of it.

Some difficult and interesting days lie ahead. Difficult and interesting from the standpoint that we’ll have to implement things that we haven’t been doing for years and years. This is new ground for many of us. With the right amount of due diligence and a couple of heavily padded blocks of time from various consultants, it will all get done. Fast forward to a few years down the road. Like King Henry said in Henry V:

Then will he strip his sleeve and show his scars,
And say ‘These wounds I had on Crispian’s day.’
Old men forget; yet all shall be forgot,
But he’ll remember, with advantages,
What feats he did that day.

and

And gentlemen in England now-a-bed
Shall think themselves accurs’d they were not here,
And hold their manhoods cheap whiles any speaks
That fought with us upon Saint Crispin’s day.

We’ll all have scars, but they’ll be scars we can be proud of. This is an interesting time to be in networking, but I wouldn’t want it any other way. I foresee changes like these flushing out people who are not ready for the paradigm shifts that are coming or are already here.

New blood will come into the field. You’ll be able to guide them and mentor them and show them your scars from the IPv4, every server was physical, no wireless, TDM PBX, Frame Relay was king days. Then, they’ll produce a fake smile as you bore them with stories of how many CAT5 patch cables you have made in your past career and then they’ll mock you when you’re not around. Kind of like how we mock Thomas Watson and his inability to predict the demand for computers.

Categories: career, learning

Tech Field Day 5

January 19, 2011 2 comments

Tech Field Day

I’ve been fortunate to receive an invite to Tech Field Day 5 out in San Jose, California. The event takes place in February and will bring IT vendors and technical people together to talk about products from the various vendors and their particular vision or strategy for the part of the IT market that they do business in. That’s a nice way of saying that a bunch of people get together to geek out for a few days. While most IT professionals can listen to a variety of vendors talk about their products via the usual sales channels, events like this allow people like myself to visit the vendor on their home turf and ask all kinds different questions in a more relaxed setting.

You can read more about Tech Field Day here.

This particular Tech Field Day will be focused on the datacenter. Considering the bulk of my work focuses around the data center, I cannot stress enough how excited I am to take part in this. This will be a great chance to not only talk directly to vendors like Infoblox and Symantec, but to talk to other IT professionals who bring their own opinions and viewpoints to the table. Since I focus on the network side of the house, it will be great to spend some time with people who focus on virtualization, storage, and the systems side of things.

I plan on writing about my experiences at Tech Field Day 5 and will be active on Twitter as well during my time in San Jose. And of course, in the interest of being completely open and honest:

My travel and living expenses are being covered by the various corporate sponsors. However, I am under no obligation to write anything about the event, and if I do, I am not obligated to make it a positive article. Additionally, there may be some things I hear that are not generally released to the public yet, so I won’t speak about those things until the vendor makes them public.

Categories: career, learning, vendors Tags:

Programming Bad Performance

January 15, 2011 11 comments

Image courtesy of Wikipedia

Last week an interesting problem surfaced at work. An application engineer received reports of slow performance on a particular website and needed some help from my group to track down the source of the problem. This engineer had done some fantastic research on the problem and was able to answer almost every question we threw at him in regards to details surrounding the issue. I am going to try and run through the problem itself and the questions we asked which led us to the possible culprit. The solution to the problem was discovered a few days later and ended up surprising us as it was not even something we had considered could be the cause. Although the application engineer collected a lot of data in the form of trace logs and packet captures, my group didn’t examine any of this data. The problem was solved before we actually had to get in and look at the data ourselves. With a white board and some direct questions, we were able to point the engineer in the right direction. He did all the work.

Problem:

A URL that was Internet facing was performing very sluggish compared to others.

When did the problem start? Unknown.

Possible causes to consider:

1. Remote end of the connection
2. Internet connectivity
3. Firewall
4. Intrusion prevention sensor/Content filter/Other security hardware
5. Router/Switch/Load balancer problem on the internal network hosting the site
6. Server hosting the site
7. Web server software on server hosting the site(ie IIS,Apache)
8. Web site code (ie HTML,ASP,JScript,CSS,XML)

Troubleshooting: For the purposes of isolating the problem, we started with the remote connectivity and worked our way inward. From here on out, I am going to refer to the application engineer as Bob. That’s not his real name, but it’s a lot easier to type than “application engineer” or his actual name.

Had Bob checked into the remote side as being the source of the problem? Yes, he had. In fact he ran the same checks from other ISP’s and experienced the same result. That rules out item 1 on the list of possible causes.

Bob had a lot of additional information to add regarding this problem. First, this particular website was really a specfic URL that was problematic. Over a dozen URL’s using the same exact hostname were fine. It was just this one particular URL that was having a problem. That rules out item 2 as being the issue. Second, Bob stated that the problem was occurring on the internal network as well. That rules out items 3 and 4 from the list of possible causes. Now we’re getting somewhere. At this point, we know that we aren’t dealing with a problem isolated to the Internet. That’s actually a good thing because it’s never easy when you have to explain to people that you have no control over traffic once it leaves your network. It just comes off like you are passing the buck to non-network savvy people.

Bob added an additional piece that would vindicate the network hardware from being the culprit. He stated that the average MTU on all of the URL’s that were working great was somewhere over 1000 bytes. However, for the URL that was operating sluggish, the average MTU size was a little over 200bytes. Now the discussion goes on for a few minutes about how MTU size will affect performance and that 200byte average sizes are not good when compared to the other URL’s and their greater than 1000 MTU averages.

At this point, we know there is an MTU problem and that problem occurs on the external and internal network. Now I know that every switch this traffic is traversing on the internal network is going to allow an MTU of 1500, so I don’t think there is a piece of networking gear causing the problem. This seems like it is going to be something with the system itself. It turns out that this particular server hosting all of these URLs is one of several servers hiding behind a load balancer. I know my load balancer isn’t messing with the MTU, so I feel comfortable in ruling out item 5 as being the source of the problem.

Has Bob checked the server hosting these URLs? Bob indicates that there are 4 different servers behind the load balancer hosting these same URLs and they are all having the same problems. He tested the URL on each individual server and experienced latency. It is possible that we are dealing with a problem on all 4 servers, However, the odds of that being a sever hardware probem are very low. Considering the fact that these same servers host over a dozen more URL’s that are running with no problems, I am convinced that we can rule out item 6 as a possible culprit.

Now we are looking at the web server software or the site code itself as being the culprit. While I am by no means an expert when it comes to IIS, Apache, or other web server software, I am willing to bet that the issue is not with the web server software. My reasoning is that only 1 URL is experiencing the problem and over a dozen other URL’s are not. They are all using the same hostname, so one would expect any sort of MTU setting in the web server software, if there is one, to be the same across every URL.

At this point in the troubleshooting process, we figured it must be something in the code. Our recommendation to Bob was that he go back to the developers and have them check their code.

Bob came back several days later. He found the problem. Actually, there was no problem. The way the developers had coded this particular URL was what caused the problem. In this case, they had a bunch of really small CSS files that were used in conjunction with the URL that was problematic. The client would make the request and then it would have to grab tons of really small CSS files. Due to the small size of these files, the MTU itself was small. I suppose that small file sizes wouldn’t be too much of a problem, except in this case, there were too many files that had to be transferred. That is what was causing the latency.

In this particular case, there was nothing wrong with any infrastructure or server equipment. Everything was working as designed. If nothing else, it was a reminder that developers don’t always consider application performance over the network when designing software. They routinely get beat up for having poor security. I guess you can add poor network performance to the list as well. I think it is a generally accepted belief that programs are usually designed for low latency LAN environments, and very rarely are designed with WAN performance in mind. I shouldn’t be surprised to find a case like this in which the code wasn’t designed with network performance in mind at all.

I feel that it is also important to point out that it is fairly difficult to write code that takes all factors into consideration(ie Security, Network). Maybe the best solution is to involve the various entities during the testing of code to ensure it will perform properly. I can see how this issue would have been overlooked since it was a simple URL that was affected. Had it been an entire program that was affected, it might have been caught during testing.

Dealing With Knowledge Gaps

January 6, 2011 2 comments

Inevitably, we are all going to come across things in our jobs that we are deficient in. Maybe we know a little about a certain topic, but we need to know more. Maybe we know absolutely nothing and need a basic introduction to the topic. Regardless, there will come a time in which we need to increase our knowledge and understanding of something in this ever growing world of networking or just IT in general.

The problem as I see it, is how I go about filling in those gaps. When you just start out in the IT world, you may not have a good methodology in which to learn about IT things. If you have been in the industry for a long time, you may already have a good system that works for you. No matter which category you fall into, the fact that you will constantly have to learn is unavoidable. There are NO exceptions to this rule. If you wish to be at the top of your game in IT from a technical standpoint, you must make a habit of constantly learning new things. Failure to do so means that your knowledge will become dated and you will drift off into obscurity working as some corporate slave in a dark and dreary cubicle. This may or may not involve working for the government. 🙂

Now that we have established that static knowledge is a dead end, let’s look at how to ensure we are always at the top of our game. I offer you the 5 step plan. Others have 12 step programs. Maybe some have less. I only have 5. I am all about efficiency…..and my program doesn’t cost you a dime.

1. Examine your current level of knowledge. – How much do you already know about the subject in question? The answer to that question is going to dictate the kind of resources you use. Let’s use BGP for example. If you need to learn about the basics of it, there are a few good books that can handle that. There are also plenty of websites with white papers and blog posts that give a generic overview of BGP. There are some classes out there that will accomplish the same thing. However, there are quite a few books and white papers that will completely blow your mind if you don’t already have a decent understanding of BGP. The service provider side of BGP comes to mind. Enterprises and service providers use BGP in VERY different ways.

2. Find out where the information is. – For starters, you need to identify what kind of learner you are. Some of us are visual learners. Some of us are audible learners. Some of us learn by doing. Perhaps you are a mix of several different methods. Only you know what works best for you. If you need a lot of pictures and the topic is relatively mainstream, maybe a visual CBT(computer based training) course is what you need. If that is the case, I highly recommend you check out CBT Nuggets. If what you are looking for is somewhat more obscure, then I would recommend asking other people who do what you do. There are a variety of resources in which you can ask these questions like LinkedIn, forums, or Twitter. I prefer Twitter because it is a lot quicker. The only possible problem would be having enough people see the request. If you are new to Twitter, or very rarely use it, you may not have many followers who would see your message. Feel free to engage others in a substantive manner and over time your followers will grow. If all you do is tell everyone what you ate for lunch or what the weather is like in your part of the world, you probably aren’t going to get anywhere. If you absolutely refuse to use something like Twitter, then consider posting on Cisco’s forums if your issue is of a Cisco nature or networking-forum.com. There are other forums out there as well as mailing lists(NANOG comes to mind). All of the major vendors have support forums as well. Keep in mind that you may have to sift through tons of information before you finally find the information you are looking for. There is not always going to be a technical paper or book that explains exactly what you are looking for. Sometimes you have to piece it together from multiple sources. Actually, I would recommend that you use multiple sources unless it is some vendor specific thing that you can only get in one place. I have found out that you cannot trust a single source for 100% accuracy. Not that all sources are wrong, but imperfect human beings write books, white papers, and blog posts. Other imperfect human beings double check these same sources. When the content is of a technical nature, things get missed. This is especially true for the deeper technical things.

3. Execute. – You have all of the appropriate resources identified. Now you just need to get that information into your head. There are no shortcuts. While I wish I could learn kung-fu like Neo did in The Matrix, it isn’t going to happen. You have to put in the time required to absorb all of that information. Sometimes it can be done in a matter of minutes. Sometimes it takes weeks.

4. Ignore any distractions. – In the course of your learning, you are bound to come across something else that is interesting or neat. Resist the temptation to get sidetracked and stay focused on the main thing you are trying to learn. If you want to go back at another time and research the other items that pop up, then make a note of them. By focusing on the main thing you are trying to learn, you have a better chance of retaining information then if you start going in 100 different directions with every new thing that appears.

5. Allow the information to digest. – Sometimes it helps to simply think about things. Just go over it in your head. I tend to do this in conjunction with step 3. If I need to absorb a large amount of information, I like to take it in pieces and digest it little by little. By stopping to sort things out in your head, you can really come to terms with what makes sense and what doesn’t. I am very thankful my current employer allows me the freedom to do this. While it may look like I am spacing out on any given day in my cubicle, lots of times I am just thinking about something I just read or watched. It’s my way of performing a “write memory” on my brain. One of the other things I will do is drive to and from work in complete silence. That really helps because all I have to focus on is not crashing the car, which is relatively simple.

**Note – When asking others about a certain technology or product, do yourself a favor and research it first. Try and figure some things out on your own. This isn’t so much a problem with people who have been in the industry for a number of years as it is with those who have only been in IT for a few years or less. It’s not that people don’t want to answer the question. There will always be someone who will just blurt out an answer. The issue with asking without having done any research on your own is that you miss out on a great opportunity to develop your own research methods. There’s a reason that lmgtfy.com was created and is often quoted on Twitter. It has been my experience that those who last in IT are the ones that only need a nudge in the right direction. They don’t want their hand held. They just want a sanity check every now and then. The people who never want to put in the time or effort to figure something out and habitually want you to solve their problems are the ones that won’t make it in the long run. Well, they might have a job, but they won’t be anywhere near what they could be if they put forth some effort.

I am not going to make the bold claim that the 5 steps I laid out will work for everyone. They work for me when I follow them, and I don’t always follow them. I find the instances in which I have tried to cram something new into my head without following these steps ends badly. I forget something and have to start all over again. When I take the time to really dig into something and not rush it, it tends to stay with me at least from a conceptual point of view.