Note: This post was originally written for the Netreo blog. You can check out the original here.
If you want to know how to analyze network traffic, you came to the right place.
Analyzing your network’s traffic can be daunting. It involves collecting, storing, and monitoring all the data traversing your on-premises, hybrid, or multi-cloud infrastructure. You’ll need to visualize and search this data for network planning and design. You also need notifications when something’s gone wrong to effectively troubleshoot. So it can be a lot to deal with.
To help you feel better about the whole thing, let’s break down the steps you need to take.
Step 1: Identify Your Data Sources
The first step is to find out what’s out there on your network. You can’t analyze and monitor something if you don’t know it exists. There are two parts to this step.
Determine Data Source Types
You’ll need to identify and categorize the types of sources you can collect data from. There are applications, desktops, servers, routers, switches, firewalls, and more. Each of these can provide various metrics you can collect for analysis.
Decide Methods of Identification
Next, you’ll need to determine the best methods you can use to identify your data sources. You can use a manual or automated approach. The manual approach involves sifting through topology maps and other documentation, but they quickly go stale. So consider the automated method with application and network discovery. Common auto-discovery methods include using SNMP, Windows Management Instrumentation (WMI), flow-based protocols, and transaction tracing. Doing this now will later help you find application and network dependencies and maximize infrastructure visibility.
Step 2: Determine the Best Way to Collect from Data Sources
The next step is to find out the best way to collect the data you need from your data sources. There are broadly two ways to collect network traffic data: with and without agents.
Agent-Based Collection
Collecting data using an agent involves deploying software on your data sources. Agents can collect information about running software processes, system resource performance, and inbound/outbound network communications. While agent-based collection can provide very granular data, it can also create processing and storage issues.
Agentless Collection
Collecting data without agents involves using processes, protocols, or APIs already supported by your data sources. Agentless collection includes methods such as SNMP on network devices and WMI on Windows servers. Syslog enabled on firewalls helps identify security events, and flow-based protocols help identify traffic flows. Agentless collection doesn’t always produce data as granular as agent collection, but it works well enough to give you the user and system data you need to properly analyze network traffic.
Step 3: Determine Any Collection Restrictions
Once you know your data sources and the best way to extract network traffic data from them, it’s tempting to just get going. But your organization likely has rules and restrictions on what and how infrastructure is managed. Not determining any of these requirements beforehand will adversely affect your ability to analyze network traffic.
So make sure to find out if there are any ports you need to open up for collection, for example. Also be sure to find out if departmental approval is required before data collection can begin. This can help you break down silos by collecting data from other parts of the network.
And think about the industry your organization is in. Highly regulated industries like healthcare or finance may not allow you to collect certain types of data or may require you to store data for a longer period. Having more historical data can be helpful for network traffic analysis, but this takes up storage. So be aware of any rules restricting or governing data collection.
Step 4: Start a Small and Diverse Data Collection
The next step is to enable your data sources for collection. The key here is to start small with a diverse set of data sources, especially if you run a large network. This will help identify issues with any systems before you expand your reach across the network. The last thing you want is to collect data from all your Windows servers, for example, and then find out that certain groups of servers keep crashing. So start small with a diverse group and expand from there.
Step 5: Determine the Data Collection Destination
You need to determine the destination for all the data you’re collecting. Network traffic can be stored using special-purpose hardware or virtual appliances. Installing monitoring software on your physical or virtual devices is also an option.
Consider the size and complexity of your network. If large portions include virtual devices, for example, virtual appliances may be more appropriate. If your organization still mostly uses on-premises physical infrastructure, a hardware device may be the better option. Avoid using a virtual appliance to monitor a busy virtual network inside that network.
The destination appliance for network traffic storage determines how you can analyze it. An appliance with no ability to view the data via a web UI, for example, makes analysis harder. If you have a software component, your life will be easier because it may help you analyze data as well as collect it.
Step 6: Enable Continuous Monitoring
Analyzing network traffic usually isn’t a one-time event. There are times when you need to troubleshoot a specific problem, such as an unanticipated security breach or sudden link failure. You might also need to help analyze network traffic from an area of the network that, despite all your efforts above, isn’t reachable or restricts monitoring. In these cases, you may need to collect and analyze traffic one time or for a specific period.
But to properly analyze network traffic, you need to continuously monitor and collect data from your infrastructure. Continuous monitoring is paramount for real-time and historical traffic collection. So be sure to enable continuous monitoring with whatever solution you chose as the destination for network traffic in the previous step.
Step 7: View and Search Collected Data
Analyzing network traffic involves sifting through gigabytes or more of data. And you have to view, search, and make sense of all of it. Maybe you’re a terminal wizard and can grep your way through it to find what you’re looking for, and you think text files stored on your server or that appliance might be fine. But traffic analysis involves being able to categorize network data into buckets such as application, byte size, protocol, IP subnet, etc. It’s not easy doing that via the command line.
So you need to ensure that you have a monitoring solution in place to see all the collected data. Being able to visualize network traffic via dashboards and reports can greatly reduce the amount of time it takes to troubleshoot an application problem. This will help you identify who the top talkers are on your network and what they’re experiencing. It can help you find the most-used applications and the issues they’re having. By taking this step, you can also find low-bandwidth network connections that you can get rid of to save money.
Step 8: Set Up Alerts
The last step is to make sure you’re notified when there’s a problem. You can’t sit in front of your screen all day viewing dashboards and reports. So you need to configure your monitoring solution to alert you if something goes wrong. Alerts are often sent via email, but you can also alert yourself and your team with integrations you get from monitoring tools like Netreo. Whichever monitoring tool you use, it must send the right alerts so you can avoid alert fatigue.
Don’t forget to also set custom thresholds. The right monitoring tool should be able to help you detect anomalies, but you know your network best. If you know certain ports aren’t allowed through your firewalls, you should create alert thresholds for that. Even if the tool is newly deployed, it can still help you know when something’s not right, and you can start digging deeper.
Follow These Rules
If you follow the steps above, there should be enough congratulations for your whole team. Because at the end of the day, analyzing network traffic is a team game. So implement these steps and help get your IT team members to do the same. This way, your entire team can benefit from knowing how to analyze network traffic.