You just got a call about an application slowness issue. You’ve been told that it’s not an application issue and that it must be somewhere on the network.
When you need to capture some data, for whatever reason, one question that inevitably comes up is where you should capture. What’s the best location for a packet capture? That’s a good question to ask yourself.
If you don’t, you should be. So let’s talk about that now.
Locations, Locations!
If you have servers in one location, and clients in another, you ideally want to capture at both ends. Even better is if you can capture packets at multiple points in case there are firewalls and WAN acceleration devices that are terminating TCP connections.
But oftentimes, you are unable to capture on both sides, much less capturing at multiple points. When that happens, you want to capture as close to the client as possible. The main reason for this is because you hope to be able to see in your packet capture what the user who’s complaining about an issue is experiencing.
So let’s say a doctor is having problems accessing an electronic health record (EHR) application that runs in a colocated data center. This data center is connected to the health systems network via dual 1Gbps virtual private network (VPN) connections. The doctor complains and complains. You get on the case and start looking at data going in and out of the data center. You come back and say that everything looks good.
But the doctor’s complaints continue because using the EHR is still slow.
What to do!
Server Side Include
Capturing near the server is a thought that often comes to mind. You’re thinking, well, there are other servers in the data center and a bunch of network devices is there. But the problem with capturing near the server only is that the server is likely doing its job. The primary job of a server is to send data out of its network connection to its local network switch as fast as possible. These days, you’re talking about 10Gbps port speeds. With these types of Ethernet connections, there are plenty of servers capable of doing this.
So the server side is less likely to be the issue when you only have one side you can capture on.
Plus, capturing data on the server side doesn’t exactly tell you what the user is experiencing. So even if it’s not a server issue, there are so many things between the server and the client that can skew the client’s experience, like load balancers, and of course, routers.
Route Collector
You can capture on a router as an option. One problem with that is resources. Routers, unlike servers, are not built to serve as much data as quickly as possible necessarily. Their primary job is to decide the best path from the server to the client. They do this using various routing protocols like OSPF for inter-AS communication or BGP for AS-to-AS communication. The router needs to pick which interface a packet of data needs to go, and this needs to be done as fast.
Capturing data is processing-intensive. Routers protect themselves from this by giving a packet capture a lower priority than other more important functions like, uh, routing. This is why using port mirroring should be avoided.
Also, at the router, you’re away from the server and the client. So the data you’re capturing could have been manipulated by a series of other network devices, like firewalls, WAN optimization, or load-balancing devices.
Because of these things, it’s possible that you may not even see the issue there. So capturing on a router isn’t likely to help much.
The Client’s Base
When you can only do a packet capture on one side, doing so on the client side is best because the client’s side doesn’t have all the processing power that the data center has with all of its servers. And it also doesn’t always have the most powerful network devices. The client side tends to have smaller access layer switches. It’s possible that everything works fine up until the packets get down to the “last mile” on the client side.
What if a smaller network device can’t handle the data? And if a smaller network connection isn’t big enough at that particular time of day when doctors see the problem?
What happens is the packets slow down.
There could be a number of reasons. And capturing data on the client side when you can only capture on one side will help you see this.
Non-Ideal Acceptance
Ideally, you want to capture at all locations when there’s application slowness. This will allow you the best chance to rule out an application or network issue. But more often than not, this will not happen. You either won’t be able to get access to everything, or security protocols will not allow it. After, packets don’t lie. Even encrypted packets can tell a story.
Many times, you can only capture data in only one location. Go with the client location to give yourself the best chance of quickly resolving application slowness.