ACE Boot Camp – Days 3 and 4

Days 3 and 4 did not disappoint! I don’t know if I stated this in the earlier posts, but the days basically consisted of lecture in the morning and labs after lunch. I REALLY, REALLY enjoyed the lecture portion. Again, I have to state that the instructor was fairly knowledgeable in regards to ACE, so he was able to actually teach instead of regurgitate a slide deck like other classes I have been in. That makes all the difference in the world. As for the labs, I guess they do some good if you have not had much experience with the ACE CLI. We did not do any labs using the built in GUI or ANM. The problem I have with labs is that they are a very canned and controlled environment. You end up just going through the motions without actually soaking up what it is that you are doing. Ideally, the labs would need to be tailored to your environment to have the greatest effect. This of course, is not realistic. Having said that, I am sure there are some people who get something out of it. My opinion was shared by others in the class in regards to the effectiveness of the labs, so I am not the only one who feels this way. However, the effectiveness of the lecture portion completely overshadowed any shortcomings of the lab portion.

In the interest of brevity, I am going to touch on the things I thought were the most interesting, but I don’t want this post to be so long it requires a coffee break to finish.

Route Health Injection – On a simplistic level, RHI allows the ACE to inject a host route into the network. You would use this to advertise the VIP(virtual IP) that clients use to connect to a server farm. If the server farm is not available due any number of issues, the host route can be automatically removed from the route table and not advertised. The alternative is to simply advertise the VIP’s as part of a regular subnet advertisement like you do with any other VLAN or subnet. Again, I am simplifying this and need to point out that this is NOT something that is specific to Cisco ACE. Other vendors implement similar technologies.

KeepAlive-Appliance Protocol(KAL-AP) – There’s a few variations of the Cisco ACE, and one of those is the Global Site Selector(GSS). Its purpose is simply to provide higher level load balancing between data centers. Basically, it is a load balancer of load balancers. By using KAL-AP, the GSS can query VIP’s at multiple data centers and determine which one is the best fit to send traffic to.

There are a couple of things that the ACE 4710 appliance does that the ACE module cannot. I asked the question as to why this is the case and was told that the ACE appliance has different architecture than the module. It has certain functionality that might come to the module at some point, but for now is restricted to the appliance. These extra functions really revolve around the ACE appliance being able to cache certain HTTP objects and speeding up the process of delivering a web page to an end user. A fair amount of detail on this can be found here.

It sure seems as if I cut back on the information from days 3 and 4 when compared to 1 and 2. I did. Although there were plenty of interesting things covered in the past 2 days of class, a lot of those things would take a while to explain and draw out via diagrams. That’s also assuming that I actually understand these things well enough to explain them in depth.

That brings to me to a more philosophical point in regards to the type of niche product that Cisco ACE is. While it would be great if you knew the CLI on ACE backwards and forwards, it really isn’t necessary. What is necessary is an understanding of what a platform like ACE is capable of. I sat in a meeting today in which some developers wanted ACE to perform health checks on a server outside of a load balance pool and use the results of that query to determine whether or not servers should be removed from a load balance pool. Basically, they wanted to do something that ACE is not really designed to do. Spending 4 days in a classroom learning all about ACE gave me the information needed to have a productive meeting with these developers today. I was able to answer their questions and give better guidance than I would have a couple of weeks ago. I don’t know all the commands for ACE. I will still have to use the configuration guides to look things up now and again. The important thing is that I understand the capabilities and limitations of the ACE load balancer a lot better today than I did prior to taking the ACE class. My main goal is to know what it can and cannot do in order to design anything requiring load balancing properly. To me that is more important than memorizing commands.

ACE Boot Camp – Day 2

Day 2 of ACE boot camp did not disappoint! Another full day of lecture and labs. We covered the following topics:

Modular Policy CLI
Managing the ACE Appliance and Service Module
Security Features
Layer 4 Load Balancing
Health Monitoring

I’ll cover some general things about each topic and go into additional details on the points I thought were interesting.

Modular Policy CLI – ACE classifies which traffic it will load-balance based on policy maps, which are comprised of class maps. If this sounds a lot like how you build QoS policies on IOS based routers, it is. The big difference is that ACE is far more restrictive in what those policies contain.

Managing the ACE Appliance and Service Module – Like most Cisco devices, ACE can be managed in a number of different ways. Telnet, SSH, HTTPS, and SNMP. You can even use the XML API if you want. With SNMP, versions 1 and 2 cannot understand contexts. SNMP version 3 can. In order for SNMP version 1 and 2 to work with contexts, you have to use the community string format of “community@context” where “community” is the community name and “context” is the name of the virtual context. When the GET, SET, or whatever SNMP action you choose hits the ACE, the “@context” portion is understood and passed along to the appropriate context.

Security Features – There are a ton of different ways to restrict traffic entering and leaving the ACE. Most of the time you will be focused on traffic entering the ACE. As with applying ACL’s to interfaces on switches and routers, very rarely will you see access lists applied in the outbound direction. That feature is there in case you have some special need to use it.

An interesting capability that the access lists have in ACE is the ability to use object groups to identify which traffic to permit or deny. If you have ever worked on the PIX, ASA, or FWSM, you will be familiar with object groups. They make traffic identification much easier not to mention the simplification of the ACE configuration itself.

The much more granular security options were of great interest to me. Take something like IP fragmentation and reassembly. You can specify the max number of fragments allowed from one packet. If it exceeds the number you specify, you can just drop the traffic. Many other options exist with regards to the packet stream itself. You can enforce certain flags from being set. If violations occur, not only can you drop the traffic, but you could actually reset the flag itself and then send the traffic through the ACE. While most options are configurable, there are some rules that are always enforced. For example, the source IP of a traffic flow can never equal the destination IP.

Layer 4 Load Balancing – This is exactly what it sounds like. Load balancing based on TCP/UDP flows. I think the neatest part about this particular topic was the fact that you can actually load balance traffic across multiple firewalls and have the return traffic come back through the same firewall. This of course requires an ACE on both sides of the firewall, but withe ability for the ACE module to have up to 250 virtual contexts, it doesn’t have to be 2 separate physical ACE modules. The same module can host both contexts that live on either side of the firewall. It is fairly clever how they make this work. Essentially, when traffic comes from one firewall into the ACE, it remembers the MAC address of the sending firewall and places that connection in a state table. When traffic comes back through the ACE, it already knows which firewall to send the traffic to based on that state table. I’m not sure I would want to use an ACE module for load balancing through firewalls, but there are plenty of customers out there that are already doing it or could see the benefit in doing something like that.

Health Monitoring – If there’s one thing the ACE seems to have a fairly large amount of options on, it’s the health monitoring or probes. All the major protocols have specific probes on the ACE that are used to check the health of the back end or “real” servers. This is way beyond the load balancer simply pinging the server to make sure it is up and running. Let’s say you used the HTTP probe. Instead of just trying a simple ping to check a back end servers’ status, the HTTP probe can actually go out and make an HTTP connection to the server or serverfarm. That’s a far more intelligent way to query server status. Based on the probe results, any number of things can be done to the various serverfarms and servers ACE may be providing services for. They may be taken out of active status, have their priority reduced, etc.

There’s a LOT more to this stuff. This was only day 2 of 4! More to come.

ACE Boot Camp – Day 1

First off, let me point out that this is not a boot camp with a certification in mind. It’s a 4 day course given by Firefly Communications. Although I booked the course through Global Knowledge, I was told that they typically outsource their data center courses to Firefly. Works for me. As long as it is quality training, I don’t care if you outsource it to Elbonia. I am assuming they use the term “boot camp” because it is an end to end ACE class taught in just 4 days.

Which brings me to my first point. My company was able to use Cisco Learning Credits to pay for this class. At 30 credits, that translates to $3,000 US dollars for 4 days worth of training. Sitting in the class, I couldn’t help but notice people doing regular work while the instructor was going through his lecture. I realize most places are understaffed. Outages happen. Fires have to get put out. However, $3,000 for 4 days to me is a big deal. If you send your employees off to training that is critical/applicable for their job, LET THEM TRAIN! Leave them alone while they are there. Of course, that’s a 2 way street in that some employees need to learn to let go as well. The company will function without them for a few days. You can turn off “martyr” and “hero” mode for a couple of days. I am checking e-mail at night, but not being obsessive about it. I have very capable co-workers who can do anything and everything without my help.

Now, on to the actual class. Let me begin by commenting on the quality of instruction. I’ve been to plenty of poor classes in which someone was trying to shovel test material down your throat the whole time. I’ve also sat in several classes where the instructor was obviously out of their league and could not field questions from the crowd that weren’t covered on the vendor approved slide deck. That is simply not the case with Firefly. My instructor is very competent and when he hits the limit of his knowledge, he indicates that. So far, I think I have only seen 1 time out of the dozen or so questions he was hit with today in which that was the case. I guess that is what $3,000 a seat gets you.

It seems as if there is a fairly decent mix of people in this class. About a dozen or so in attendance. A fair amount of them are actually using the ACE 4710 appliance which I thought was rather interesting. Of course, most are using the standard ACE module. There are varying levels of experience with ACE as well. I was under the impression that I would be here mainly for the second half of the class, as I felt comfortable with the basics. Of course, just when you go and get comfortable, you realize how little you know. I learned a LOT today. Mainly, it was about things I never really bothered to dig into. You see, like most people, we probably only dig into the features we absolutely need right now. Maybe we plan on coming back and covering everything else at a later time, but I think that happens far less than we’d like it to. Some of the things we covered today that I was horribly deficient on were:

Resource Management – If you use multiple contexts, RM can prevent a single context from taking over the entire resources of the module. I don’t use this as it is currently not a concern, but good to know if things change!

HTTP Message Structure – 3 fields make up each HTTP message: Start/Request line(includes the METHOD), Header fields, and Body(which is optional)

ACE 4710 appliance – I don’t use it and never have. However, it does do a few things the module does not mainly centered around application acceleration. We have not covered that exhaustively yet, but I will take good notes when we do.

There were other things covered in which I was glad to get a decent refresher. The main one being TCP sequence numbers. They are always a bit confusing to me if I don’t study them on a fairly regular basis. Although you weren’t there with me in class today, you can read this post by Jeremy Stretch which talks about TCP sequencing. He even uses nice graphics!

We ended the day doing a pretty simple lab in which we created some contexts and messed around with resource management to see if we could oversubscribe the module in terms of CPU, memory, etc in regards to other contexts. Overall, it was a really good first day. I am eagerly anticipating what tomorrow will be like. It is also good to be taught by someone who actually helped develop the slide deck the course is taught from. He was able to add funny little details about how he created this drawing or that. It’s always nice to have someone teach who has a great sense of humor. So far, I give the Firefly ACE boot camp 2 thumbs up!

I am hoping to get a wee bit more technical in the following posts regarding ACE boot camp as the remaining days will REALLY focus on load balancing. Who knows? I might even post a graphic or two! Shocking isn’t it?