Install and Configure AnyConnect NVM 4.7.xor Later and Related Splunk EnterpriseComponents for CESA Contents

IntroductionPrerequisitesRequirementsComponents UsedDeployment Overview Background InformationCisco Anyconnect Secure Mobility Client - More than VPNInternet Protocol Flow Information Export (IPFIX)IPFIX CollectorSplunk EnterpriseTopologyConfigureAnyconnect NVM Client ProfileConfigure NVM Client Profile via ASDMConfigure NVM Client Profile via Anyconnect Profile EditorConfigure Web-Deployment on Cisco ASAConfigure Web-Deployment on Cisco ISETrusted Network DetectionDeployStep 1. Configure Anyconnect NVM on Cisco ASA/ISE.Step 2. Set up IPFIX Collector Component.Step 3. Set up Splunk with Cisco NVM App and Add-On for Splunk.VerifyValidate Anyconnect NVM InstallationValidate Collector status as RunningValidate SplunkTroubleshootPacket FlowBasic Steps to Troubleshoot Trusted Network Detection (TND)Flow TemplatesRecommended ReleaseRelated DefectsRelated Information

Introduction

This document describes how to install and configure the Cisco AnyConnect Network VisibilityModule (NVM) on an end-user system using AnyConnect 4.7.x or higher as well as how to installand configure the associated Splunk Enterprise components and NVM Collector.

For more information about the solution please refer to www.cisco.com/go/cesa.

The components that make up the solution are:

Cisco AnyConnect Secure Mobility Client with Network Visibility Module (NVM) enabled●

Cisco AnyConnect Network Visibility Module (NVM) App for Splunk●

Cisco NVM Technology Add-On for Splunk●

NVM Collector (bundled in a zip file with the Splunk Application)●

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

  

AnyConnect 4.3.x or higher with NVM●

AnyConnect APEX license●

ASDM 7.5.1 or higher●

Familiarity with Splunk Enterprise and how to install Splunk Apps and Add-ons●

Components Used

The information in this document is based on these software and hardware versions:

  

Cisco AnyConnect Security Mobility Client 4.3.x or later●

Cisco AnyConnect Profile Editor●

Cisco Adaptive Security Appliance (ASA), version 9.5.2●

Cisco Adaptive Security Device Manager (ASDM), version 7.5.1●

Splunk Enterprise 6.3 or later●

Ubuntu 14.04.3 LTS as a collector device●

The information in this document was created from the devices in a specific lab environment. All ofthe devices used in this document started with a cleared (default) configuration. If your network islive, make sure that you understand the potential impact of any command.

Deployment Overview 

This is a high level overview of deployment in its simplest form. This is an all-in-one configurationrunning on 64-bit Linux. 

This configuration is how most demonstrations are setup and is also useful in a small production

deployment.

This is a more comprehensive set of options that are available for deployment. Typically, aproduction setup is distributed and has several Splunk Enterprise nodes.

Background Information

The Cisco AnyConnect Network Visibility Module provides a continuous feed of high valueendpoint telemetry. NVM empowers organizations to see endpoint and user behavior on theirnetwork, collects flows from endpoints both on and off-premise along with valuable contexts likeusers, applications, devices, locations, and destinations. Splunk Enterprise consumes thetelemetry data and provides the analytics capabilities and reports.

This technote is a configuration example for AnyConnect NVM with Splunk Enterprise as part ofthe new CESA solution.

Cisco Anyconnect Secure Mobility Client - More than VPN

Cisco Anyconnect is a unified agent that delivers multiple security services to protect theenterprise. AnyConnect is most commonly used as an enterprise VPN client, but it also supportsadditional modules that cater to different aspects of enterprise security. The additional modulesenable security features like posture assessment, web security, malware protection, networkvisibility and more.

This technote is about Network Visibility Module (NVM), which integrates with Cisco Anyconnect toprovide administrators the ability to monitor endpoint application usage.

For more information regarding Cisco Anyconnect, refer to:

Cisco AnyConnect Secure Mobility Client Administrator Guide, Release 4.7

Internet Protocol Flow Information Export (IPFIX)

IPFIX is an IETF protocol to define a standard for exporting IP flow information for variouspurposes like accounting/auditing/security. IPFIX is based on Cisco NetFlow protocol v9, thoughnot directly compatible.  Cisco nvzFlow is a protocol specification based on the IPFIX protocol. Bydesign, IPFIX is an extensible protocol allowing one to define new parameters to conveyinformation. Cisco nvzFlow protocol extends the IPFIX standard and defines new InformationElements as well as defines a standard set of IPFIX templates that are conveyed as part of thetelemetry used by AnyConnect NVM.

For more information on IPFIX, refer to rfc5101,rfc7011,rfc7012,rfc7013,rfc7014,rfc7015.

IPFIX Collector

A collector is a server that receives and stores IPFIX data. It can then feed this data to Splunk.

Cisco provides a collector specifically designed for the nvzFlow protocol and bundled with theSplunk App.

Splunk Enterprise

Splunk Enterprise is a powerful tool that collects and analyses diagnostic data to give meaningfulinformation about the IT infrastructure. It provides a one-stop location for administrators to collectdata that is crucial in understanding the health of the network.

Splunk is a partner of Cisco's and the CESA solution was created in collaboration with them.

Topology

IP address conventions in this technote : 

Collector IP address: 192.0.2.123

Splunk IP address:    192.0.2.113

Configure

This section covers the configuration of Cisco NVM components.

Anyconnect NVM Client Profile

Anyconnect NVM configuration is saved in an XML file that contains information about thecollector IP address and port number, along with other information. The collector IP address and aport number need to be correctly configured on the NVM client profile.

For correct operation of the NVM module, the XML file is required to be placed in this directory:

For Windows 7 and later: %ALLUSERSPROFILE%\Cisco\Cisco AnyConnect SecureMobility Client\NVM

●

For Mac OSX: /opt/cisco/anyconnect/nvm●

If the profile is present on Cisco ASA/Identity Services Engine (ISE), then it is auto-deployed alongwith Anyconnect NVM deployment.

XML profile example:

<?xml version="1.0" encoding="UTF-8"?>

-<NVMProfile xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

xsi:noNamespaceSchemaLocation="NVMProfile.xsd">

-<CollectorConfiguration>

<CollectorIP>192.0.2.123</CollectorIP>

<Port>2055</Port>

</CollectorConfiguration>

<Anonymize>false</Anonymize>

<CollectionMode>all</CollectionMode>

</NVMProfile>

NVM profile can be created using two different tools:

Cisco ASDM●

Anyconnect Profile Editor●

Configure NVM Client Profile via ASDM

This method is preferable if Anyconnect NVM is being deployed via Cisco ASA.

   

1. Navigate to Configuration > Remove Access VPN > Network (Client) Access >Anyconnect Client Profile.

2. Click Add, as shown in the image.

   

3. Give the profile a name. In Profile Usage, select Network Visibility Service Profile.

4. Assign it to the group-policy being used by Anyconnect users and click on OK, as shown in theimage.

   

 5. The new policy is created, click on Edit, as shown in the image.

6. Enter the information regarding the Collector IP address and port number and click on OK.

7. Now click on Apply, as shown in the image.

    

Configure NVM Client Profile via Anyconnect Profile Editor

This is a stand-alone tool available on Cisco.com. This method is preferable if Anyconnect NVM isbeing deployed via Cisco ISE. The NVM profile created using this tool can be uploaded to CiscoISE, or copied directly to endpoints.

For detailed information on Anyconnect Profile Editor, refer to:

The AnyConnect Profile Editor

Configure Web-Deployment on Cisco ASA

This technote assumes that Anyconnect is already configured on the ASA, and only NVM moduleconfiguration needs to be added. For detailed information on ASA Anyconnect configuration, referto:

ASDM Book 3: Cisco ASA Series VPN ASDM Configuration Guide, 7.5

In order to enable Anyconnect NVM module on Cisco ASA, perform these steps:

1. Navigate to Configuration > Remote Access VPN > Network (Client) Access > GroupPolicies.

2. Select relevant group-policy and click on Edit, as shown in the image.

3. Within the group-policy pop-up, navigate to Advanced > Anyconnect Client.

4. Expand Optional Client Modules to Download and select Anyconnect Network Visibility.

5. Click OK and apply changes.

Configure Web-Deployment on Cisco ISE

In order to configure the Cisco ISE for Anyconnect Web-Deployment, perform these steps:

In Cisco ISE GUI, navigate to  Policy > Policy Elements > Results.1.Expand Client Provisioning to show Resources, and select Resources.2.

Add Anyconnect Image:

Step 1. Select Add > Agent Resources, and upload the Anyconnect package file.

Step 2. Confirm the package's hash in the pop-up.

The file-hash can be verified against Cisco.com download page or using third-party tool.

This step can be repeated to add multiple Anyconnect images. (for Mac OSX and Linux OS)

Add Anyconnect NVM profile:

Step 1. Select Add > Agent Resources, and upload the NVM client profile.

Add Anyconnect configuration file:

   

Step 1. Click on Add and choose AnyConnect Configuration

Select the package uploaded in the previous step.

Step 2. Enable NVM in the AnyConnect Module Selection along with the policy required.

In this section, we enable AnyConnect Client modules, profiles, customization/language packages,and the Opswat packages.

For detailed information regarding web-deployment configuration on Cisco ISE, refer to:

Web-Deploying AnyConnect

Trusted Network Detection

AnyConnect NVM sends flow information only when it is on a Trusted Network. It uses the TNDfeature of AnyConnect client to learn if the endpoint is in a trusted network or not. 

Trusted Network Detection is configured in the AnyConnect Client Profile (XML) used for VPNregardless of whether the VPN component is being used in the environment or not.  TND isenabled by configuring the Automatic VPN Policy section in the profile.  At a minimum, a singleTrusted DNS Domain or Trusted DNS Server must be populated.  The actions taken byAnyConnect when the client has determined that it is on a Trusted Network can be set toDoNothing mode using the pull-down for the Trusted and Untrusted Network Policy.

For additional details on TND configuration, refer to:Configure Trusted Network Detection

Deploy

Deploy Anyconnect NVM solution involves these steps:

1. Configure Anyconnect NVM on Cisco ASA/ISE.

2. Set up IPFIX Collector component.

3. Set up Splunk with Cisco NVM App and Add-On.

Step 1. Configure Anyconnect NVM on Cisco ASA/ISE.

This step has been covered in detail in the Configure section.

Once NVM is configured on Cisco ISE/ASA, it can be auto-deployed to client endpoints.

Step 2. Set up IPFIX Collector Component.

The Collector Component is responsible for collecting and translating all IPFIX data from theendpoints and forwarding it to the Splunk Add-On. The NVM collector runs on 64-bit Linux.CentOS, Ubuntu and Docker configuration scripts are included. The CentOS install scripts andconfiguration files can also be used in Fedora and Redhat distributions as well.

In a typical distributed Splunk Enterprise deployment, the collector should be run on either astandalone 64-bit Linux system or aSplunk Forwarder node running on 64-bit Linux.

Note: The solution can also be run on a single 64-bit Linux system that includes the NVMcollector and Splunk Enterprise components for use in a small deployment or fordemonstration purposes.

In order to install the collector, you need to copy the application in theacnvmcollector.zipfile,located in the $APP_DIR$/appserver/addon/ directory to the system you plan to install it on.Extract the files on the system where you plan to install the collector on and execute the install.sh

script with superuser privileges. It is recommended to read the $PLATFORM$_README file inthe .zip bundle before it executes the install.sh script. The $PLATFORM$_README file providesinformation on the relevant configuration settings that need to be verified and modified (ifnecessary) before the install.sh script is executed. At a minimum, you need to configure theaddress of the Splunk instance that you forward data to. Failing to properly configure the systemcan cause the collector to operate incorrectly.

Note: Ensure that network and host firewalls are properly configured to allow the UDP trafficfor the source and destination addresses and ports

A single NVM collector instance can handle a minimum of 5000 flows per second on a properlysized system. The collector needs to be configured and running before the Splunk App can beused.

By default, the collector receives flows from AnyConnect NVM endpoints on UDP port 2055.

Additionally, the collector produces three data feeds for Splunk,Per Flow Data, Endpoint IdentityData, andEndpoint Interface Data, on UDP ports 20519, 20520 and 20521 respectively.

The receive and data feed ports can be changed by altering the acnvm.conf file and restarting thecollector instance. Make sure that any host/network firewalls between endpoints and the collectoror between the collector and Splunk system(s) are open for the configured UDP ports andaddresses. Also, ensure that your AnyConnect NVM configuration matches your collectorconfiguration. 

Once all components are installed and running, refer to the Help files section from within theSplunk application for detailed information about the pre-configured reports, data model andinformation elements that are created by the solution.

You may want to restart one of your AnyConnect endpoints and validate that data is being sent tothe solution.

The information needs to be configured in the configuration file (acnvm.conf):

1. The IP address and listening port of Splunk instance.

2. Listening port for the collector (incoming IPFIX data).

Per Flow Data Port,  Endpoint Identity Data Port, Endpoint Interface Data and Collector Port arepre-configured to default settings in the configuration file. Ensure that these values are changed ifnon-default ports are being used.

This information is added in the configuration file (acnvm.conf):

GNU nano 2.2.6 File: acnvm.conf

{

"syslog_server_ip" : "192.0.2.113",

"syslog_flowdata_server_port" : 20519,

"syslog_sysdata_server_port" : 20520,

"syslog_intdata_server_port" : 20521,

"netflow_collector_port" : 2055,

"log_level" : 7

}

For more information, refer to:

https://splunkbase.splunk.com/app/2992/#/details

Step 3. Set up Splunk with Cisco NVM App and Add-On for Splunk.

Cisco AnyConnect NVM App for Splunk is available on Splunkbase. This app helps with pre-defined reports and dashboards to use IPFIX (nvzFlow) data from end points in usable reports andcorrelates user and endpoint behavior.

Link for Cisco NVM App for Splunk on Splunkbase:

https://splunkbase.splunk.com/app/2992/

Link for Cisco NVM Add-On for Splunk on  Splunkbase:

https://splunkbase.splunk.com/app/4221/

Install:

Step 1. Navigate to Splunk > Apps and install the tar.gz file downloaded from the Splunkbase orsearch within the Apps section.

Step 2. Next, you need to install the Add-On following the same process. Confirm that both areinstalled by viewing Splunk Apps page:

The default configuration receives three data feeds for Splunk, Per Flow Data, Endpoint IdentityData and Endpoint Interface Data, on UDP ports 20519, 20520 and 20521 respectively. (see Step2)

The Add-On then maps these to Splunk sourcetypes cisco:nvm:flowdata, cisco:nvm:sysdataand cisco:nvm:ifdata.

In order to change default ports, navigate to Splunk > Settings > Data Input > UDP, as shown inthe image.

Verify

Validate Anyconnect NVM Installation

After successful installation, the Network Visibility Module should be listed in Installed Modules,within in the Information section of Anyconnect Secure Mobility Client.

Also, verify if the nvm service is running on the endpoint and profile is in the required directory.

Validate Collector status as Running

Ensure that the collector status is running. This ensures that the collector is receiving IPFIX/cflowfrom the endpoints at all times.

root@ubuntu-splunkcollector:~$ /etc/init.d/acnvmcollectord status

* acnvmcollector is running

root@ubuntu-splunkcollector:~$

Validate Splunk

Ensure that Splunk and its relevant services are running. For documentation on troubleshootingSplunk, please refer to their website.

Troubleshoot

Packet Flow

1. IPFIX packets are generated on client endpoints by Anyconnect NVM module.

2. The client endpoints forward IPFIX packets to the Collector IP address.

3. The collector collects the information and forwards it to Splunk.

4. Collector sends traffic to Splunk on two different streams: Per Flow Data and Endpoint IdentityData.

All traffic is UDP based on there is no acknowledgment of traffic.

Default port for traffic:

IPFIX data         2055

Per Flow Data   20519

Endpoint Data   20520

Interface Data   20521

NVM module caches IPFIX data and sends it to a collector when it is in Trusted Network. This caneither be when the laptop is connected to the corporate network (on-prem) or when it is connectedvia VPN.

Basic Steps to Troubleshoot 

Ensure network connectivity between client endpoint and collector.●

Ensure network connectivity between collector and splunk.●

Ensure that NVM is correctly installed on client endpoint.●

Apply captures on endpoint to see if IPFIX traffic is being generated.●

Apply captures on collector to see if it is recieving IPFIX traffic, and if it is forwarding traffic toSplunk.

●

Apply captures on Splunk to see if it is recieving traffic.●

IPFIX traffic as seen in Wireshark:

Trusted Network Detection (TND)

NVM relies on TND for detecting when the endpoint is within a trusted network. If the TNDconfiguration is incorrect, this will cause issues with NVM.

TND works based on information received via DHCP: domain-name and DNS server. If the DNSserver and/or domain-name match the configured values, then the network is deemed to betrusted.

If NVM is not forwarding traffic to the collector, then it could be an issue with TND.

Flow Templates

IPFIX flow templates are sent to the collector at the start of the IPFIX communication. These

templates help the collector to make sense of the IPFIX data.

The collector also preloads templates to ensure that even if the client has not sent them that thedata can be parsed. If a newer version of the client is released with protocol changes, the newtemplates sent by the client will be used.

A template is sent out under the following conditions:

There is a change in the NVM client profile.1.There is a network change event.2.The nvmagent service is restarted.3.The endpoint is rebooted/restarted.4.

In rare circumstances, a template may not be found.  This can be easily remedied by restartingone of the endpoints.

The issue can be identified by observing no template found in a packet capture on the endpoint,or no templates for flowset in the collector logs.

Packet capture

Collector logs:

Jan 20 12:48:54 csaxena-ubuntu-splunkcollector NVMCollector: no templates

for flowset 258 for 10.150.176.167 yet

Jan 20 12:48:55 csaxena-ubuntu-splunkcollector NVMCollector:

HandleReceivedIPFIX: exporter=10.150.176.167 bytes_recvd=234 totlength=234

Jan 20 12:48:55 csaxena-ubuntu-splunkcollector NVMCollector:

=================> flowsetid=258 flowsetlen=218

Jan 20 12:48:55 csaxena-ubuntu-splunkcollector NVMCollector: no templates

for flowset 258 for 10.150.176.167 yet

Recommended Release

Cisco always recommends the latest software version of AnyConnect at the time of use orupdating. While choosing the AnyConnect version, please use the latest 4.7.x client or later. Thisgives the latest enhancements with respect to NVM.

Related Defects

CSCva21660 - Anyconnect NVM Handles/Leak for acnvmagent.exe*32 process (fixed in 4.3MR1).

Related Information

Cisco Endpoint Security Analytics on Splunk (Quick StartGuide): https://www.cisco.com/c/dam/en/us/products/se/2019/8/Collateral/endpoint-sec-analy-quickstart-guide.pdf

●

Cisco AnyConnect Network Visibility (NVM) App for Splunk:https://splunkbase.splunk.com/app/2992/

●

Splunk Documentation on Splunk Collector Setup and installing collector scripts:https://splunkbase.splunk.com/app/2992/#/documentation

●

Cisco AnyConnect Secure Mobility Client- Administration Guide●

Release notes of AnyConnect 4.x●