The role of encrypted traffic analysis for threat detection [Q&A]

Everyone is striving to make their systems more secure and in many cases that means adopting encryption in order to protect data.

But the use of encrypted traffic over networks presents a headache for security teams as malicious content can be harder to detect. We spoke to Thomas Pore, director of security products at Live Action, to find out more about the problem and how it can be addressed.

BN: How is encrypted traffic impacting network threat detection today?

TP: The increased adoption of encrypted network protocols is causing the deterioration of network visibility for security teams, and legacy tools are increasingly less effective. In Q4 of 2021 alone, 78 percent of malware delivered via encrypted connections were evasive, according to a recent report, highlighting the growing threat of advanced malware attacks. Additionally, the rising acceptance of HTTPS, rapid deployment of encrypted protocols such as DNS over HTTPS, and TLS 1.3 are greatly decreasing visibility into server identity and content inspection, making threat detection more difficult, and in many cases nearly impossible, for network defenders. Once inside an organization's network, threat actors are leveraging encrypted sessions to move laterally -- east to west. Traditional detection tools only inspect north-south traffic. This gives attackers the advantage they need to complete advanced actions, like a ransomware attack.

BN: What is encrypted traffic analysis and why is it important to threat detection and response?

TP: Encrypted traffic analysis is a type of side-channel analysis that allows network defenders to do their jobs while maintaining the privacy and network integrity provided by a fully encrypted system. Encrypted Traffic Analysis, coupled with machine learning capabilities, evaluates complex data patterns over time and differentiates normal and abnormal activities, all without requiring access to the content of the data. It allows security teams to leverage varying types of C2 activity (such as beaconing, TLS fingerprinting and sequence of packet lengths) to quickly uncover malicious behavior and network anomalies, which are vital for effective threat detection and response. Effectively, ETA enables network transaction visibility, which provides valuable insights about the encrypted traffic to aid network defenders.

BN: What is encryption blindness and how can it impact organizational security?

TP: Encryption blindness is caused by a lack of visibility into encrypted traffic leading to missed (hidden) threats in the network. Because most modern IT network traffic is now concealed in encryption, hackers can leverage this gap in security to hide their actions inside encrypted traffic. In other words, a large amount of traffic in organizations today goes uninspected simply because it's encrypted, opening the door to attacks. As threats get more sophisticated and the attack surface grows, the effectiveness of many traditional strategies is decreasing, such as IDS, IPS, and break-and-inspect decryption. This is challenging the effectiveness of organizational security more than ever.

BN: What is the difference between Deep Packet Inspection (DPI) and Deep Packet Dynamics (DPD) for ETA?

TP: Deep Packet Dynamics (DPD) is a new approach to evaluating network packets that eliminate the need for payload inspection. By analyzing more than 150 packet traits and behaviors across multi-vendor, multi-domain, and multi-cloud network environments, it can more reliably evaluate both encrypted and unencrypted traffic.

When DPD is coupled with machine learning and ETA, it enables unique capabilities for regaining visibility into encrypted traffic and delivers some of the most advanced network detection and response capabilities available today. This includes a variety of benefits such as detecting threats and anomalies others miss; detecting threats in real-time; eliminating encryption blindness; decreasing the time a SOC needs to investigate and respond to threats; validating end-to-end encryption compliance; offering visibility from core to edge to cloud; and enabling the security team to create a coordinated and cohesive response through other security tools like SIEM, SOAR, etc.

In contrast, Deep Packet Inspection (DPI) is an older legacy approach that primarily works on unencrypted or clear text protocols such as HTTP. But encryption undermines DPI and allows malicious payloads to hide in encrypted traffic. In short, DPD offers network defenders a much clearer vision of encrypted network traffic than DPI does.

BN: What role does ETA play in broader network detection and response solutions?

TP: Encrypted traffic analysis is a way to restore network visibility for defenders while maintaining privacy for users by combining DPD and advanced behavior analysis combined with machine learning. Malicious threat actors and malware system operators communicate with infected target systems using a set of techniques called Command and Control (C2). Threat actors employ C2 techniques to mimic expected, benign traffic using common ports and standard encryption protocols to avoid detection. Despite these precautions, ETA with machine learning effectively identifies malicious C2 activity on the network so you can stop an attack. Even with zero visibility into the content of the connection, ETA can tell a great deal about the behavior of encrypted traffic and helps network defenders prioritize their network detection and response activities.

BN: What's next – or on the horizon -- when it comes to ETA?

TP: Encrypted traffic analysis will further fortify the long-term security strategies of organizations, through the continued characterization of encrypted flows and behavioral pattern recognition. This extends across endpoints, assets, and end-to-end encryption, mapping benign and expected traffic against malicious anomalies. Phishing and remote access protocols (RDP/VPN) continue to be the leading infection vectors of ransomware and state-sponsored APT actors. ETA's high-fidelity detection of anomalous characterization will be the difference in stopping the attack into the future.

Photo credit: Rawpixel.com / Shutterstock