securonix enterprise 5.0 troubleshooting guide · 2019. 10. 10. · migrationcomponents...

SECURONIX UEBA 5.0

TROUBLESHOOTINGGUIDE

Securonix Proprietary Statement

Thismaterial constitutes proprietary and trade secret information of Securonix, andshall not be disclosed to any thirdparty, nor usedby the recipient except under the terms andconditions prescribedby Securonix.

The trademarks, servicemarks, and logos of Securonix andothers usedherein are the property of Securonix or theirrespective owners.

Securonix Copyright Statement

Thismaterial is also protectedby FederalCopyright Lawand is not to be copiedor reproduced in any form,using anymedium,without the prior written authorization of Securonix.

However, Securonix allows the printing of the Adobe Acrobat PDF files for the purposes of client training and reference.

Information in this document is subject to change without notice. The software described in this document is furnishedunder a license agreement or nondisclosure agreement. The softwaremay be usedor copiedonly in accordance withthe terms of those agreements.Nothing herein shouldbe construedas constituting an additionalwarranty. Securonixshall not be liable for technical or editorial errors or omissions containedherein.Nopart of this publicationmay bereproduced, stored in a retrieval system,or transmitted in any formor anymeans electronicormechanical, includingphotocopying and recording for any purpose other than the purchaser's internal use without the written permission ofSecuronix.

Copyright 2017©Securonix, Inc.All rights reserved.

Contact Information

Securonix, Inc.

14665Midway Rd.Ste. 100,Addison, TX 75001

www.securonix.com

855.732.6649

Revision History

Date Product Version Description

07/14/2017 5.0 Initial draft

Copyright©2017SecuronixPage | 2

Securonix UEBA5.0 TroubleshootingGuide

CONTENTSTroubleshooting 4

Monitoring the Application 4

Troubleshooting the Application 5

Database Overview 5

MySQL Query Management 9

Alerts and Common Issues 11

Securonix Application Issues 11

Activity Import Issues 24

UI Issues 30

Other Errors in Log 32


Securonix UEBA 5.0 Troubleshooting Guide

TroubleshootingThis guide describes how to troubleshoot the Securonix Enterprise 5.0 application. For more troubleshootingtips and tricks, see the Securonix Support Forums.

For issues not found in this guide or in the Forums, contact Securonix Support [email protected].

Monitoring the ApplicationThe following components are critical to monitoring:

System Resources

l Java 8

l java -version

l Tomcat processmonitoring-Tomcat 8

l service tomcat status

l ps –ef |grep tomcat

l MySQLMemory/CPU-Mysql 5.6.34

l sudo service mysql status

l Disk Usage

l df -sh

l df -u

l UI URLmonitoring

l Service/Port monitoring

l Log Monitoring

l /Securonix/Tomcat/logs/Securonix.log

l /Securonix/Tomcat/logs/catalina.out

l Job Monitoring

l /Securonix/Securonix_home/import/success

l /Securonix/Securonix_home/import/failed

Some common scenarios in which alerts may trigger:

Issue Solution

Long-running jobs result in high tomcat heapusage on themaster node.

Ensure that the behavior jobhas been configuredto run incrementally,which will accrue behaviorbaselines on adaily basis.

HighMySQLMemory/CPUusage on themaster couldbecausedby long-running queries ormultiple copies of the samequeries firedover andover.

1. Stop Tomcat.

2. Kill the queries.

3. Restart Tomcat.


Securonix UEBA5.0 TroubleshootingGuideTroubleshooting

https://securonixsupport.freshdesk.com/helpdesk

Issue Solution

High disk/Javausagemay indicate aproblem in the ingestionof logs by Securonix.

High diskusage import/success, import/failed indicates thathousekeeping jobsmay not be regularly clearing theprocessed logs.

OutOfMemory exceptions in catalina.out indicate that the heaplimits set on the tomcat JVMhave been reached.

This can be resolvedby analyzing the usage trendsin Sitescope.

A restartmay be necessary to return Tomcat tonormaloperation after encountering this exception.

Troubleshooting the ApplicationTo troubleshoot any errors, first look at the various steps in which data flows:

1. Scripts

2. Analyzers/Preprocessors (If Any)

3. Securonix Ingestion Job

Logging/file output at multiple levels assists in troubleshooting:

l /Securonix/Securonix_home/conf/applicationcontext.xml

l /Securonix/Tomcat/conf/Catalina/localhost/Profiler.xml

l Securonix/Tomcat/logs/catalina.out

l Securonix/Tomcat/logs/securonix.log

l Securonix/securonix_home/import/success

l Securonix/securonix_home/import/failed

l MySQL – information_schema.processlist

For example, if we find a certain policy not producing the expected results:

1. Examine line filters and input configurations to check if the events are not being filtered out.

a. Check files in /Securonix/Securonix_home/import/failed to see if any relevant lines have failed theimport

2. If a Analyzer is associated with that datasource, check /Securonix/Tomcat/logs/securonix.log and/Securonix/Tomcat/logs/catalina.out for information on the preprocessing.

Database Overview

Note: Important tables, foreign keys, and their description are provided in a separate document.



Troubleshooting

Migration ComponentsThe following components are involved during a migration process in Securonix. Migration typically involvesmoving over new data source, user, and other Securonix entity-based import configurations from oneenvironment to another ( ex: UAT to Prod).

l ConfigResourceParsers

l Table primarily used for migration of Activity Import configurations, policies, and behaviorconfigurations between environments.

l Contains import, policy and behavior settings as XML.

l Connection Types aka Data sources (FromConfigure > Connection Types)

l These contain information on the configuration for retrieving the data - e.g. File location, Splunkconnection credentials

l Other configurations

l User import settings: Attribute mapping, encryption setting and other rules set up in user imports

l Configxml: Additional XML configuration present in configxml table

l Lookup Tables (Using MySQL dump),Workflows

l Lookup tables and workflows can bemigrated from one environment to another using MySQLdump/restore.

l The dump must be restored on bothmaster & child nodes.

l Patches &Connectors

l These are performed by an exact copy of the relevant directories and files. Instructions will beprovided with each new package that must be applied.

Migration of CRPsThe following stepsmust be followed to update the exported configurations in the configresourceparserstable to the most recent version. As of now, selectively exporting certain policies/behavior is not possiblethrough the UI. Select all behavior/policies while performing the export, and run the SQL provided below toremove unwanted items.



1. Save the following configurations via UI (Select all options):

a. Line Filters fromAdd Data > Activity > [Datasource] > Line Filters > Save Template.

b. Policies fromAnalytics > Policy Violations >[Sort by Datasource] > Save Template.

c. Policies fromAnalytics > Activity Outliers > Save Template.



Troubleshooting

d. Behavior Configuration fromAnalytics > Behavior Profiles > Save Template.

2. Create a new datasource fromAdd Data > Activity > New Datasource.

a. Select resource type that has beenmigrated.

3. SQL to remove unwanted items:

a. Delete Behavior Profile.

b. Update configresourceparsers set bprofilexml=updatexml(bprofilexml,"//bprofileconfig[id/bpname='PROFILE_NAME']", "") where resource_type='RESOURCE_TYPE';

c. Delete Policy



d. Update configresourceparsers set policyxml=updatexml(policyxml,"//policyMaster[name='POLICY_NAME']","") where resource_type='RESOURCE_TYPE';

e. Show Behavior Profiles

f. Select extractvalue(bprofilexml, "BprofileconfigList/bprofileconfig/id/bpname") fromconfigresourceparsers where resourcetype='RESOURCE_TYPE';

g. Show Policies

h. Select extractvalue(policyxml, "PolicyMasterList/policyMaster/name") from configresourceparserswhere resourcetype='RESOURCE_TYPE';

MySQL Query ManagementRunning MySQL queries are displayed using the following:

show [full] processlist;

They can also be viewed via the table information_schema.processlist:

mysql> desc information_schema.processlist;

E x a m p l e :

+---------+---------------------+------+-----+---------+-------+| Field | Type | Null | Key | Default | Extra |+---------+---------------------+------+-----+---------+-------+| ID | bigint(21) unsigned | NO | | 0 | || USER | varchar(16) | NO | | | || HOST | varchar(64) | NO | | | || DB | varchar(64) | YES | | NULL | || COMMAND | varchar(16) | NO | | | || TIME | int(7) | NO | | 0 | || STATE | varchar(64) | YES | | NULL | || INFO | longtext | YES | | NULL | |+---------+---------------------+------+-----+---------+-------+8 rows in set (0.00 sec)

Using a select on this table, restrict the output to a specific user, DB, or Query.

In case of long-running queries holding up the system, run the following command to get the processids and kill them:

mysql -N -u operations -p -e "select id from information_schema.processlistwhere user='operations' and info like 'select%' and and command <>'Sleep'" |cut -f1 | sed -e 's/^/kill /' | sed -e 's/$/;/' > /tmp/kill.sql

mysql –u operations –p < /tmp/kill.sql

Note: In the example above, the query used to fetch the process IDs is limited to select queriesrunning through the webuser user account, as well as those which are active (not sleeping).



Troubleshooting

Note: Ensure that the binlog dump query (run by the user replica, with Info field “binlog dump”) isnot killed – this will kill the replication.Tomcat has its own layer of query management. Killing running queries fromMySQLmay causeerrors in the application , requiring a tomcat restart.Killing insert/update/delete queries is not recommended. These trigger time-consuming rollbacks,as well as resulting in loss of data.



Alerts and Common IssuesThe following describes common alerts and issues within the Securonix Enterprise application and how toresolve these issues.

Securonix Application Issues

ERROR connection. DatasourceConnectionProvider: Could not find datasource:java: /comp/env/jdbc/securonixDSSeverity 1

Details

Securonix application after starting tomcat does not come up. Findmore details regarding the errors in the followinglogs:

l /Securonix/Tomcat/logs/securonix.log


Typically, this issue occurswhen the Profiler.xml file is accidentally deleted fromthe /Tomcat/conf/Catalina/localhostfolder. This file acts as the connection between the application and the database.

Solution

1. Validate that all the important Securonix configuration files exist in the paths below:

a. Profiler.xml (/Securonix/Tomcat/conf/Catalina/localhost/)

b. application-context.xml (/Securonix/Securonix_home/conf)

2. Replace the file in the folder from the backup files. Example: $ scp secitprtpprd04:/[backuplocation]/Profiler.xml]/Securonix/Tomcat/conf/Catalina/localhost/

3. Verify that Tomcat services are not already running: $ ps -ef|grep tomcat

4. When the copy has completed, restart the application: $ service tomcat restart

Validation



Alerts and Common Issues

Details

1. After restarting, navigate to (/Securonix/Tomcat/conf/Catalina/localhost/) and verify Profiler.xmlexists.

$ ls /Securonix/Tomcat/conf/Catalina/localhost/

2. Tail the logs:

$ tail -123f /Securonix/Tomcat/logs/securonix.log$ tail -123f /Securonix/Tomcat/logs/catalina.out

Ensure no errors exist in logs.

3. Browse the application on the UI:

https://hostname:8443/Profiler

This application shouldnow loadwithout any issues.

404 Error after loading the Securonix URL in the browserSeverity 2

Details

When loading the Securonix URL,Http404 error is shown.

Findmore details regarding the errors in the following logs:



Solution


Securonix UEBA5.0 TroubleshootingGuideAlerts andCommon Issues

Details

Clear browser cache and reload the page. If this does not resolve the issue proceedwith steps below:

Validate using the following steps:

1. Check if the Securonix tomcat service is up and running using the following command: $ ps -ef |grep tomcat (Linux Based Systems) ( This should show a pid for tomcat )

2. Check for MySQL service: $ ps -ef | grep mysql ( This should show a pid)

3. Validate that the Profiler.xml file located under /Securonix/Tomcat/conf/Catalina/localhost exists.This file is essential as it provides the connection between the application and the database.

4. Validate the parameters setwithin the JDBCURL.Specifically, check the name of the database in the jdbcURL.Ensure it's pointing to the correctDB.$ vi/Securonix/Tomcat/conf/Catalina/localhost/Profiler.xml

JdbcURL: jdbc:mysql://[Node_IP]:3306/[database_name]?autoReconnect=true&rewriteBatchedStatements=true&useUnicode=true&characterEncoding=UTF-8&maxReconnects=10

5. If both these services are upand running and the parameters in the Profiler.xml are correct andpage is still notcoming up,CheckSecuronix andcatalina logs:


Please refer this document to fix any other errors thatmight appear.

Validation

1. Browse the application on the UIhttps://hostname:8443/Profiler

This application shouldnow loadwithout any issues

2. Ensure no errors exist in logs:


Application throwing Java Heap Space Error /OutOfMemoryExceptionSeverity 2

Details

Errors in the Securonix log include JavaheapSpace andout ofmemory errormessages.







Details

Solution

1. Check the Javamemory parameters. These are defined in CATALINA_OPTS(specified in startup.sh(/Securonix/Tomcat/bin/startup.sh) file of tomcat).

$ vi /Securonix/Tomcat/bin/startup.sh

2. Checkparameters -Xms8g -Xmx50g. If these values are small, you can increase these values depending on theavailable RAMsize. Ex:256GBRamcan have heapsizemin andmax of 70 to 90G.However, increasing thesevalues is contextual andmay not always be necessary.

A large file size at the time of activity import getting loaded intomemorymay cause Javaheapspace For imports insuch cases, check that the split file functionality is enabled (tag is found in application-context.xml file found in/Securonix/securonix_home/conf folder).

$ vi /Securonix/securonix_home/conf/application-context.xmlFind additional information about OutOfMemoryExceptions in thefollowing resource: Understand the OutOfMemoryError Exception

3. If you need to change any parameters above, stop Tomcat on all nodes beginning with the childnodes then the master:$ service tomcat stop

4. Perform the necessary changes as described above.

5. Start tomcat on all nodes beginning withMaster node then child nodes: $ service tomcatstart

Validation





Tomcat/Securonix does not start properlySeverity 1

Details

Securonix application after starting tomcat does not come up.






https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/memleaks002.html#CIHHJDJE

Details

Solution

1. Type ps -ef |grep tomcat to determine howmany tomcat instances are up.

2. If multiple instances are up perform the following: kill -9 {PID} of the tomcat instance(s) from step 1.

3. Restart Tomcat: $ service tomcat restart

OR

1. Check if the Node database is lockeddue to:

a. Select and insert statements being run at the same time on the same table

Login tomysql:

$ mysql –u root –p database_nameShow processlist;

b. Offline/live database dumpoccurring

In either case, youwill have towait for the process to complete before Tomcat can be restarted.

Validation





Communications link failure with auto reconnect JDBC URLsSeverity 1

Details

Communications link failure with auto reconnect JDBCURLs prevents job import.







Details

Solution



Details

Toprevent connection timeouts, ensure all database connection URLs, use the following four parameters:

l autoReconnect=true

l rewriteBatchedStatements=true

l useUnicode=true

l characterEncoding=UTF-8

Thismust be changed in several places:

a. Profiler.xml

$ vi /Securonix/Tomcat/conf/Catalina/localhost/Profiler.xml

url="jdbc:mysql://<host>:<port>/<database>?autoReconnect=true&rewriteBatchedStatement=true&useUnicode=true&characterEncoding=UTF-8"

testWhileIdle="true"

testOnBorrow="true"

testOnReturn=“true"

validationQuery="SELECT1"

validationInterval="30000"

b. Hibernate Templates (on Master and Child)

a. Master Nodes:

$ vi /Securonix/securonix_home/conf/hibernate/hibernate.cfg.<childname>.xml

<propertyname="hibernate.connection.url">jdbc:mysql://<childhost>:<port>/<database>?autoReconnect=true&rewriteBatchedStatements=true&useUnicode=true&characterEncoding=UTF-8</property>

b. Child Nodes:

$ vi /Securonix/securonix_home/conf/hibernate/hibernate.cfg.master.xml

<propertyname="hibernate.connection.url">jdbc:mysql://<masterhost>:<port>/<database>?autoReconnect=true&rewriteBatchedStatements=true&useUnicode=true&characterEncoding=UTF-8</property>

c. Database Connections in Securonix UI:

In each of the Database Connections between theMaster and the Child, and the archive connections,ensure that the four parameters are in the DBURL:

1. ClickConfigure->Connection Types.

JDBCURL:jdbc:mysql://<masterhost>:<port>/<database>?autoReconnect=true&rewriteBatchedStatements=true&useUnicode=true&characterEncoding=UTF-8




Details

2. If you need to change any parameters above first stop tomcat on all nodes beginning withchild nodes thenmaster: $ service tomcat stop

3. Perform the necessary changes as described above.

4. Start Tomcat on all nodes beginning withMaster node then child nodes: $ servicetomcat start

Validation





Error after upgrading nodes - "Out of resources/Too many files open" errorSeverity 2



Details

A limitationmay be set on the number of concurrent open files that yourMySQLmaybe allowed to keepopen.




Error:

7:15:03,453 ERROR [JDBCExceptionReporter] Out of resources when openingfile './securonix_node1/activityfreqnwtime15#P#week3.MYD' (Errcode: 24 -Too many open files) 17:15:03,456 ERROR [GrailsRuntimeConfigurator][RuntimeConfiguration] Unable to load beans from resources.groovyorg.codehaus.groovy.runtime.InvokerInvocationException:java.lang.ExceptionInInitializerErrorat grails.spring.BeanBuilder.invokeBeanDefiningClosure(BeanBuilder.java:723)at grails.spring.BeanBuilder.beans(BeanBuilder.java:573)at grails.spring.BeanBuilder.invokeBeanDefiningClosure(BeanBuilder.java:723)at grails.spring.BeanBuilder.beans(BeanBuilder.java:573)at grails.spring.BeanBuilder.invokeMethod(BeanBuilder.java:519)at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)at java.util.concurrent.FutureTask.run(FutureTask.java:262)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)at java.lang.Thread.run(Thread.java:745)Caused by: java.lang.ExceptionInInitializerErrorat com.securonix.application.hibernate.util.DbUtil.executeHQLQuery(DbUtil.java:1187)

Solution

1. Step 1: Stop Tomcat on all nodes beginning with child nodes thenmaster: $ service tomcatstop

2. Step 2: Edit the limits.conf file shown below:$ sudo nano /etc/security/limits.conf

securonix soft nofile 24000

securonix hardno file 32000

3. Step 3: Stop mysql on all nodes: $ service mysql stop

4. Step 4:Under the [mysqld] section ofmy.cnf, add the following parameter:$ vi /usr/my.cnf

open_files_limit = 100000

5. Step 5: Start MySQL on all nodes: $ service mysql start

6. Step 6: Start Tomcat on all nodes beginning withMaster node then child nodes: $ servicetomcat start




Details

Validation



Error in QRTZ SchedulerSeverity 2

Details

Error shows upon application startup. securonix.log shows the following errors:

ERROR [QuartzSchedulerListener] Error in Quartz scheduler: Couldn'tacquire next trigger: You have an error in your SQL syntax; check themanual that corresponds to your MySQL server version for the right syntaxto use near 'OPTION SQL_SELECT_LIMIT=5' at line 1org.quartz.JobPersistenceException: Couldn't acquire next trigger: Youhave an error in your SQL syntax; check the manual that corresponds toyour MySQL server version for the right syntax to use near 'OPTION SQL_SELECT_LIMIT=5' at line 1 [See nested exception:com.mysql.jdbc.exceptions.jdbc4.MySQLSyntaxErrorException: You have anerror in your SQL syntax; check the manual that corresponds to your MySQLserver version for the right syntax to use near 'OPTION SQL_SELECT_LIMIT=5' at line 1]at org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTrigger(JobStoreSupport.java:2794)at org.quartz.impl.jdbcjobstore.JobStoreSupport$36.execute(JobStoreSupport.java:2737)at org.quartz.impl.jdbcjobstore.JobStoreSupport.executeInNonManagedTXLock(JobStoreSupport.java:3768)at org.quartz.impl.jdbcjobstore.JobStoreSupport.acquireNextTrigger(JobStoreSupport.java:2733)at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:264)




Solution



Details

1. Updatemysql-jdbcdriver(.jar file).

This file is located in Tomcat/lib folder andProfiler/WEB-INF/lib folder.

Take abackupof the jar file:

$ cp /Securonix/Tomcat/lib/(mysql-jdbc driver).jar /Securonix/archive/(mysql-jdbc driver).jar.bkp$ cp /Securonix/Tomcat/webapps/Profiler/WEB-INF/lib/(mysql-jdbcdriver).jar /Securonix/archive/(mysql-jdbc driver).jar.bkp

Copy the new jar files to the locations below:

$ cp /path/to/jarfile /Securonix/Tomcat/lib/(mysql-jdbc driver).jar$ cp /path/to/jarfile /Securonix/Tomcat/webapps/Profiler/WEB-INF/lib/(mysql-jdbc driver).jar

2. Stop Tomcat on all child nodes beginning with child nodes thenmaster: $ service tomcatstop

3. Restart Tomcat onmaster: $ service tomcat restart

4. Start Tomcat on child nodes:$ service tomcat start

The version of the jar file in both the folders shouldbe the same.

Validation



Job MisfiredSeverity 2

Details

This error shows up in logswhen a jobhasmissed its schedule and runs at a later time. The jobusually continues tocompletion.




Solution




Details

1. Allow the job to run and monitor the logs for any subsequent errors

2. If there are any errors refer this document to take the necessary course of action or contact withSecuronix Support [email protected] for a solution.

3. If the job takes longer time to complete, manual intervention is required to delete the job andreschedule the job.

Validation



Null Pointer ExceptionSeverity 1

Details

This error shows up in logs for different reasons.




Solution

1. Investigation: Tail the logs and identify the root cause of the issue.

2. Take corrective action if it’s due to data anomaly or configuration issue. For any other issues contactSecuronix Support [email protected].

Validation



SecuronixDB not configuredSeverity 2



Details

This error shows up in logs only during the initial Securonix application configuration.




Solution

1. On the Securonix UI, clickConfigure -> Connection Types.

If it exists, verify that Profiler.xmlexists at /Securonix/Tomcat/conf/Catalina/localhost.

If the Profiler.xml file ismissing replace it fromthe backup location on StandByMaster and restart Tomcat:$ service tomcat restart

If the SecuronixDBconnection type does not exist, create it basedon the stepby stepdirections providedon theUI.

2. Under Connection Types, check if the SecuronixDBconnection type exists.

3. Test connection to ensure it is successful.

Validation



2. Check that the SecuronixDB connection type is visible on UI underConfigure->Connection Types.3. Test connection to ensure it is successful.

Invalid HQLSeverity 2

Details

This error shows up in logswhen anHQL-basedpolicy runs.




Solution




Details

1. Identify the policy which is causing the invalid HQL errors in logs.

2. Check the HQL for the policy and correct it based on HQL syntax.

3. Convert the HQL to SQL and test the validity of the query.

4. Schedule or run the policy.

Validation



Activity Import Issues

Errors while connecting to Splunk: Unable to connect to SplunkSeverity 2

Details

Securonix to splunkconnection at the time of configuring an activity import throws error as below




Error:

javax.net.ssl.SSLHandshakeException: No appropriate protocol (protocol isdisabled or cipher suites are inappropriate)at sun.security.ssl.Handshaker.activate(Handshaker.java:470)at sun.security.ssl.SSLSocketImpl.kickstartHandshake(SSLSocketImpl.java:1438)at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1308)at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1359)at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1343)at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:563)at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)at sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1092)at sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)

Solution



Details

1. Go to Java installation folder

2. Open {JRE_HOME}/lib/security/java.security -file in text editor:3. Backup java.security file with dataprior to editing the file.

$ vi {JRE_HOME}/lib/security/java.security

4. Go to the last line

5. Comment out the following line "jdk.tls.disabledAlgorithms=SSLv3"

6. Restart Tomcat: $ service tomcat restart

Validation


$ tail -123f /Securonix/Tomcat/logs/securonix.log

$ tail -123f /Securonix/Tomcat/logs/catalina.out

2. Step2:On Securonix UI App, clickAdd Data -> Activity.

Click the name of the data source that has aSplunksource.

Scroll to the bottomof the pop-up screen andclickRefresh Splunk Searches.

3. Ensure Splunk searches display without errors in logs.

Activity Import Failed (Permission denied errors)Severity 1




Details

Activity import fails as input log files have different ownership as compared to theTomcat owner/application owner.This results in the application being unable to pickup log files fromthe in folder. The following errors are often seenwhen this issue occurs:

securonix.log.2015-09-09: java.io.FileNotFoundException: /securonix_home/import/success/tacacs-output1.txt (Permission denied)securonix.log.2015-09-09: java.io.FileNotFoundException: /securonix_home/import/success/tacacs-output-20150909003001.log (Permission denied)securonix.log.2015-09-09: java.io.FileNotFoundException: //securonix_home/import/success/tacacs-output-20150909023001.log (Permission denied)securonix.log.2015-09-09: java.io.FileNotFoundException: /securonix_home/import/success/tacacs-output-20150909043001.log (Permission denied)




Solution

The tomcat/application ownershipwas changed fromthe default ‘root’ ownership to adifferent owner. If the input fileshave root ownership, they are not pickedby the application during import.

1. Change the ownershipof the input files to the owner of the Tomcat to resolve the issue:

$ chown -R securonix.securonix [Securonix filepath]

Note: Consult Securonix Support [email protected] changing permissions for the files related to theSecuronix application on the platform.

Validation



2. Access the path or location while logged in to console as the Tomcat user. You shouldnot experience any issuesor errors

Date/TimeMapping Missing error during activity importSeverity 1



Details

Error logs showDateTimemissing errors during the activity import.




Solution




Details

For the Activity Import to be successful, the format of the date time value coming in to the activity feedmust be correct.If the format is specified incorrectly in Step2ofActivity Import (Line Filter), the datawon't be imported into Securonix.

Ensure the format is configuredcorrectly as in the following image:

For example, if the input data feeddisplays the Date as 4/21/2015 andTime as 12:45:00, the format shouldbe Date:MM/dd/yyyy andTime:HH:mm:ss.

Refer to the following link formore options:Class SimpleDateFormat.

Validation

1. Test the jobagain to ensure no errors exist in logs:




http://docs.oracle.com/javase/6/docs/api/java/text/SimpleDateFormat.html

Error in the Activity Import: Required Data is missing during activity importSeverity 1

Details

Error logs show requireddatamissing during the activity import:

10:18:39,243 ERROR [QuartzJob] [JobId:238] Error in job execution -ActivityImport_pp test_2015_7_30_10_18_37com.securonix.application.exception.matcher.MatcherReaderException:com.securonix.application.exception.matcher.MatcherReaderException: Nullvalue encountered, required data missing




Solution

For the Activity Import to be successful, the following fields cannot be null. This error can occur in the followingscenarios:

1. Activity log data field that is mapped to the transaction string is null.

2. Activity log data field that is mapped to the account name (AccountID) is null.

These two fields are requiredandcannot be NULL.

Validation



Job Misfired: Required data is missing during activity importSeverity 1




Details

Error logs show requireddatamissing during the activity import:

10:18:39,243 ERROR [QuartzJob] [JobId:238] Error in job execution -ActivityImport_pp test_2015_7_30_10_18_37com.securonix.application.exception.matcher.MatcherReaderException:com.securonix.application.exception.matcher.MatcherReaderException: Nullvalue encountered, required data missing




Solution

For the Activity Import to be successful, the following fields cannot be null. This error can occur in the followingscenarios:

1. Activity log data field that is mapped to the transaction string is null.

2. Activity log data field that is mapped to the account name (AccountID) is null.

These two fields are requiredandcannot be NULL.

Validation



UI Issues

New Policy Category Widget Not Showing UpSeverity 1

Details

Policy Category widget does not showupon the security dashboardon the left-handpane even though there arerelevant policy violations against it

Solution

If special characters are included in the policy name, the widgetwon't appear on the left navigation pane, even thoughit shows upon the Threats screen.

1. When creating a new policy category, ensure the name does not include any special characters (eg '-').



Details

Validation

1. When you havemodified the policy to exclude special characters, ensure it appears on the security dashboard.

Emails Not Being Sent Out from the ServerSeverity 1

Details

Emails are not sent out fromthe Securonix server

Solution

1. Checkandconfigure for the following parameters underConfigure->Settings->SMTP Server Settings:

a. SMTP Port

b. Email credentials

c. Hostname

2. Click Save and Send Test Email to confirm the parameters are set correctly.

Validation

When you receive the test email, emails are being sent out correctly.

Preview not shown when connected to DBSeverity 1

Details

When getting data (Activity/Access/User) fromadatabase, the previewdoes not showany dataand insteaddisplaysthe error 'error getting preview fromthe database'.

Solution

Newline charactersmay be included in the query being used in Securonix.

1. Rewrite the query replacing all newline characters with space.

Validation

Clickpreviewafter modifying as above and the resulting preview should loadon screen.

Securonix Application unable to connect to an SQL server instanceSeverity 1




Details

When trying to connect to aSQL Server, an error ' Login Failed. The login is froman untrusteddomain andcannot beusedwithWindows authentication'.

Cause: The account used to connect to the SQL server is going throughWindows (Active Directory) authenticationtrying to connect to the SQL Server.

Solution

1. Use the following format in your connection URL:jdbc:jtds:sqlserver://IP:1433/database_name;useNTLMv2=true;domain=windows_domain_name

Validation

1. Enter the connection details as above.

2. Click Test Connection to ensure Connection Successful message appears.3. Ensure no errors exist in logs:

$ tail -123f /Securonix/Tomcat/logs/securonix.log$ tail -123f /Securonix/Tomcat/logs/catalina.ou

Other Errors in Log

User XYZ not found in directorySeverity 1



Details

Error logs show ‘User XYZnot found in directory’

12:36:44,728 ERROR [CustomLDAPAuthAuthenticator] Error - User admin notfound in directory.12:36:44,802 DEBUG [LoginController] in LoginController => authfail =>customMessage => User authentication failed.12:36:44,811 DEBUG [AuditUIUtil] in auditLog() => username => admin12:36:44,811 DEBUG [AuditUIUtil] in auditLog() => action => authfail ,controller => LOGINCONTROLLER , userAction => LOGIN , auditMessage => Userauthentication failed. , status => ERROR12:36:44,901 DEBUG [LoginController] authfail BadCredentialsException -Authentication Failed. Check the username/password you entered and tryagain.12:36:44,907 DEBUG [AuditUIUtil] in OPERATION === title = , desc = FailedLogin12:36:44,907 DEBUG [AuditUIUtil] in executeAuditLog()...=> currentUserName=> admin Auth name => admin , HostAddress => 10.144.180.112controllerConstant => LOGIN_CONTROLLER, controller => LOGINCONTROLLER,action => LOGIN remote add => 10.144.180.11212:36:44,907 DEBUG [AuditUIUtil] checkAudit() parameters ==> title => ,action => authfail , controller => LOGINCONTROLLER , status => ERROR12:36:44,919 DEBUG [AuditUIUtil] customMessage ==> User authenticationfailed..

Cause: The account trying to login does not exist.




Solution

If this usermust be grantedaccess, contact the Securonix Admin to allowaccess.

For all userswhose login is unexpected, check /Securonix/Tomcat/logs/securonix.log formore details related to thisaccess as shown above.

Validation

No validation steps required.




securonix enterprise 5.0 troubleshooting guide · 2019. 10. 10. · migrationcomponents...

Documents