troubleshoot problems with secure remote …...services (srs) contents and overview before you begin...

For links to all Isilon customer troubleshooting guides, visit the Customer Troubleshooting - Isilon Info Hub.

We appreciate your help in improving this document. Submit your feedback at http://bit.ly/isilon-docfeedback.______________________________________________

1 - Dell EMC Isilon Customer Troubleshooting Guide: Troubleshoot Problems with Secure Remote Services (SRS)

Abstract This guide helps you troubleshoot problems with gathering logs through SRS (formerly ESRS), or problems with alerts not connecting home through SRS.

April 20, 2020

DELL EMC ISILON CUSTOMER TROUBLESHOOTING GUIDE

TROUBLESHOOT PROBLEMS WITH SECURE REMOTE SERVICES (SRS)

OneFS 7.2 - 8.2

http://bit.ly/isilon-docfeedback

http://www.dell.com/support/article/SLN318797

http://bit.ly/isi-docfeedback




Contents and overview

Before you beginPage 3

Appendix A If you need further assistance

Start troubleshootingPage 4

Problems gathering or uploading logs through SRSPage 5

Remote notification (Connect Home) is not workingPage 9

SRS with OneFS 8.1.x or laterPage 14

Note Follow all of these steps, in order, until you reach a resolution.

1. Follow these steps.

2. Perform troubleshooting steps in order.

3. Appendixes

Appendix BHow to use this flowchart

Appendix CExample output







Configure screen logging through SSH

We recommend that you configure screen logging to log all session input and output during your troubleshooting session. This log file can be shared with Isilon Technical Support, if you require assistance at any point during troubleshooting.

1. Open an SSH connection to the cluster and log in by using the root account.

Note: If the cluster is in compliance mode, use the compadmin account to log in. All compadmin commands must be preceded by the sudo prefix.

2. Change the directory to / i f s/ dat a/ I si l on_Suppor t by running the following command:

cd / i f s/ dat a/ I si l on_Suppor t

3. Run the following command to capture all input and output from the session:

scr een - L

This will create a file named scr eenl og. 0 that will be appended to during your session.

4. Perform troubleshooting.

Before you begin

CAUTION!If the node, subnet, or pool that you are working on goes down during the course of troubleshooting and you do not have any other way to connect to the cluster, you could experience data unavailability.

Therefore, make sure that you have more than one way to connect to the cluster before you start this troubleshooting process. The best method is to have a serial console connection available. This way, if you are unable to connect through the network, you will still be able to connect to the cluster physically.

For specific requirements and instructions for making a physical connection to the cluster, see article 304071 on the Online Support site.

Before you begin troubleshooting, confirm that you can connect through either another subnet or pool, or that you have physical access to the cluster.




https://support.emc.com/kb/304071




Start troubleshooting

IntroductionStart troubleshooting here. For an overview of the conventions used in this flowchart, see Appendix B: How to use this flowchart.

Follow the instructions in the table for the problem that you are experiencing.

Start

If you have not done so already, log in to the cluster and configure screen logging through SSH, as described on Page 3.

Problem Action

You are having trouble gathering logs or uploading logs through SRS.

Go to Page 5

Remote notification from the Isilon cluster to Support (Connect Home) is not occurring. A service request (SR) was not created for an

event that occurred on the cluster.

Go to Page 9

You are not receiving an email or SNMP notification of events that occur on the cluster.

Any other issue. Contact Isilon Technical Support

Make sure that you configured your event notification settings correctly. See the

"Managing event notification settings" or the "Events and Notifications" section of the

OneFS Web Administration Guide.

If that does not fix the problem, contact Isilon Technical Support and report an issue with

CELOG.

___________________________

___________________

Your cluster is running OneFS 8.1.x or later. Go to Page 14

Note SRS Gateway 2.24 or later must be installed in your environment. To upgrade SRS to the latest version, go to: SRS product page.




http://www.emc.com/contact/contact-us.htm



https://www.dell.com/support/home/us/en/04/product-support/product/emc-secure-remote-services/drivers




Problems gathering or uploading logs through SRS

You could have arrived here from:

- Page 4 - Start troubleshooting

Is SRS enabled on the cluster?

Manually upload the logs by following the instructions in the

following article:

How to upload files to Isilon Technical Support, article 304567

No

Go to Page 6

Yes

End troubleshooting

Verify that SRS upload is enabled by running the following command:

i si di agnost i cs gat her set t i ngs vi ew

Note In the OneFS web administration interface, navigate to Cluster Management > Diagnostics. On the Gather Settings tab, make sure Enable ESRS upload is checked.

From the command-line interface, enable SRS upload by running the following command:

i si di agnost i cs gat her set t i ngs modi f y - - esr s=t r ue

Does the Upl oad field have

the value of Yes?

Yes

No

Go to Page 6





Problems gathering or uploading logs through SRS (2)


- Page 5 - Problems gathering or uploading logs through SRS

Verify the SRS settings by running the following command for your version of SRS:

OneFS 8.1 and lateri si esr s vi ew

OneFS 7.2 - 8.0i si r emot esuppor t connect emc vi ew

See example output in the box on this page.

Open an SSH connection to the cluster and log on by using the root account.

Example outputExample 8.1 and later:cl ust er - 1# i si esr s vi ew Enabl ed: No Pr i mar y ESRS Gat eway: esr s. t est . i si l on. com Secondar y ESRS Gat eway: Al er t on Di sconnect : Yes Gat eway Access Pool s: subnet 0: pool 0 Gat eway Connect i vi t y Check Per i od: 60Li cense Usage I nt el l i gence Repor t i ng Per i od: 86400 Downl oad Enabl ed: No ESRS Fi l e Downl oad Ti meout Per i od: 50 ESRS Fi l e Downl oad Er r or Ret r i es: 3 ESRS Fi l e Downl oad Chunk Si ze: 1000000 ESRS Downl oad Fi l esyst em Li mi t : 80 Gat eway Connect i vi t y St at us: Di sconnect ed

Example 7.2 - 8.0 output:cl ust er - 1# i si r emot esuppor t connect emc vi ew Enabl ed: No Pr i mar y Esr s Gat eway: esr s. t est . i si l on. com Secondar y Esr s Gat eway: - Use SMTP Fai l over : NoEmai l Cust omer On Fai l ur e: No Gat eway Access Pool s: subnet 0: pool 0

Run the following command to ping the SRS gateway to see if it responds:

pi ng <ESRS- gat eway>

See the box at the bottom of this page for example output.

Did the gateway

respond?

End troubleshooting.If you need further assistance,

upload log files and contact Isilon Technical Support, as instructed in

Appendix A.

There is a networking issue. Contact your network

administrator to resolve the issue.

No

Yes

Go to Page 7

- Page 6 - Problems gathering or uploading logs through SRS (2)

Use the output from the following command to verify whether the cluster can communicate with the primary or secondary SRS

gateway. Run the following command(s) for your version of OneFS, where <ESRS- gat eway> is the Pr i mar y Esr s Gat eway:

OneFS 8.1 and lateri si _f or _ar r ay - s nc - vz <ESRS- gat eway> 9443

OneFS 7. 2 - 8. 0i si _f or _ar r ay - s nc - vz <ESRS- gat eway> 443i si _f or _ar r ay - s nc - vz <ESRS- gat eway> 8118i si _f or _ar r ay - s nc - vz <ESRS- gat eway> 9443

Note For OneFS 7.2 - 8.0, you will need to run the command once for each port: 443, 8118, and 9443.

Go to Page 8

Do you receive the error message in

the note box at the bottom of this page?

Yes

No

This indicates that your firewall is blocking SSL to port 9443. Engage your

Networking administrator to unblock this port.If you need further assistance, open a Service Request (SR)

for your SRS product.

Example outputnc: connect t o 10. 20. 30. 400 por t 9443 ( t cp) f ai l ed: Connect i on r ef used

- Page 7 - Problems gathering or uploading logs through SRS (3)

Run the following command, where:- <#> is the number of the node the command is being run on- is the IP address of the node that is being tested, from the

network pool that is associated with the gateway access pool- <GW- I P> is the SRS gateway IP address

OneFS 8.1 and lateri si _f or _ar r ay - n <#> nc - s - z <GW- I P> 9443

OneFS 7.2 - 8.0i si _f or _ar r ay - n <#> nc - s - z <GW- I P> 449i si _f or _ar r ay - n <#> nc - s - z <GW- I P> 8118i si _f or _ar r ay - n <#> nc - s - z <GW- I P> 9443

Do you receive the error message in

the note box at the bottom of this page?

Yes

No

This indicates that your firewall is blocking SSL to port 9443. Engage your

Networking administrator to unblock this port.If you need further assistance, open a Service Request (SR)

for your SRS product.

Go to Page 9

Example outputnc: connect t o 10. 20. 30. 400 por t 9443 ( t cp) f ai l ed: Connect i on r ef used

Note For OneFS 7.2 - 8.0, you will need to run the command once for each port: 443, 8118, and 9443.




Remote notification (Connect Home) is not working


- Page 4 - Start troubleshooting- Page 8 - Problems gathering or uploading

logs through SRS (4)

Is the test event listed in the New Events

table?No

Yes

Create a test event from the OneFS web administration interface, as follows:

1. Click Dashboard > Events > Notification Settings. 2. Click Send test event.3. Wait a few minutes and then go to the Summary tab. 4. Check whether the event is listed in the New Events table.

Yes

Has remote notification (Connect Home) worked

correctly in the past?

Note the page number that you are currently on.

Upload log files and contact Isilon Technical Support to

investigate issues with CELOG, as instructed in Appendix A.

There is an issue with CELOG.

No

Go to Page 10

___________________________________________________________________________







Remote notification (Connect Home) is not working (2)


- Page 9 - Remote notification (Connect Home) is not working

Example output

Name: Connect EMC Type: connect emcReci pi ent s:Cat egor i es:LEVELS EVENT TYPE CE SYS_DI SK_EVENTS ( 100000000) CE NODE_STATUS_EVENTS ( 200000000) CE REBOOT_EVENTS ( 300000000) CE SW_EVENTS ( 400000000) CE QUOTA_EVENTS ( 500000000) CE SNAP_EVENTS ( 600000000) CE WI NNET_EVENTS ( 700000000) CE FI LESYS_EVENTS ( 800000000) CE HW_EVENTS ( 900000000)Speci f i c Event t ypes:LEVELS EVENT TYPE I SW_CLUSTER_MONTHLY_STATUS ( 400090001)

Run the following command:

i si event s not i f i cat i ons l i st - - name=Connect EMC

Does the output exactly match the output in the box

on this page?

No

Open an SSH connection to the cluster and log on using the root account.

Go to Page 11

Yes

Go to Page 13




Example output Name: Connect EMC Type: connect emcReci pi ent s:Cat egor i es:LEVELS EVENT TYPE CE SYS_DI SK_EVENTS ( 100000000) CE NODE_STATUS_EVENTS ( 200000000) CE REBOOT_EVENTS ( 300000000) CE SW_EVENTS ( 400000000) CE QUOTA_EVENTS ( 500000000) CE SNAP_EVENTS ( 600000000) CE WI NNET_EVENTS ( 700000000) CE FI LESYS_EVENTS ( 800000000) CE HW_EVENTS ( 900000000)Speci f i c Event t ypes:LEVELS EVENT TYPE I SW_CLUSTER_MONTHLY_STATUS ( 400090001)



- Page 10 - Remote notification (Connect Home) is not working (2)

Run the following command:

i si event s not i f i cat i ons l i st - - name=Connect EMC

Disable and reenable SRS to recreate the settings. Run the following two commands on any node in the cluster, for your version of OneFS:

OneFS 8.0 and lateri si esr s modi f y - - enabl ed=f al se

i si esr s modi f y - - enabl ed=t r ue - - user name=<user name> - - passwor d=<passwor d>

OneFS 7.2i si r emot esuppor t connect emc modi f y - - enabl ed=f al se

i si r emot esuppor t connect emc modi f y - - enabl ed=t r ue

Yes Go to Page 12


Upload log files and contact Isilon Technical Support, as instructed in Appendix A.

Does the output exactly match the output in the box

on this page?

No









Yes

Create a test event from the OneFS web administration interface, as follows:

1. Click Dashboard > Events > Notification Settings. 2. Click Send test event.3. Wait a few minutes and then go to the Summary tab. 4. Check whether the event is listed in the New Events table.

Is the test event listed in the New Events

table?

No

Go to Page 13











Does the test event appear in the

list of closed SRs?

Yes

No

The test event should initiate an SR which will then be automatically closed.

Find your list of closed SRs at https://support.emc.com/servicecenter/srManagement/

and check whether an SR was created for the test event that you sent.


Upload log files and contact Isilon Technical Support, as instructed in

Appendix A.

CELOG is performing as expected and SRs are being generated for events that

occur on the cluster.

End troubleshooting.

If the problem occurs again, contact Isilon Technical Support.

________________________________

____________________________________________________

____________________




https://support.emc.com/servicecenter/srManagement/





SRS with OneFS 8.1.x or later


- Page 4 - Start troubleshooting- Page 16 - SRS with OneFS 8.1.x or later (3)

Is this problem related to a new

installation?Go to Page 15Yes

Check / var / l og/ i si _esr s_d. l og for errors that contain:

Er r or r esponse code f r om gat eway:

No

Are there similar errors present?

Go to Page 17

No

Yes

See:- Isilon OneFS 8.1: Error enabling ESRS gateway: Error

response code from gateway: 401, Unathorized - Event ID: 400030001, article 511087

- Isilon: Device failed. Error response code from gateway: 500, Internal Server Error. Response dictionary was: (User Correctable) (User Correctable), article 522338

Were you able to resolve

the issue?

End troubleshooting



No

Yes

___________________________________________________________

___________

___________







SRS with OneFS 8.1.x or later (2)


- Page 14 - SRS with OneFS 8.1.x or later

Run the following command to verify that SRS is enabled:

i si esr s vi ew

See Appendix C for example output.

Is SRS enabled?

Enable SRS by running the following command, where <user name> and <passwor d> are the same credentials you use to log into the Dell EMC Support site:

i si esr s modi f y - - enabl ed=t r ue - - user name=<user name> - - passwor d=<passwor d>

Go to Page 16

No

Yes

Go to Page 16

_________






- Page 15 - SRS with OneFS 8.1.x or later (2)

Using the output from i si esr s vi ew, verify that the values in the following fields are configured correctly:

- Pr i mar y ESRS Gat eway- Gat eway Access Pool s

Are those fields configured

correctly?

Configure these settings correctly. For instructions, refer to the "Enable and configure ESRS support" section of the OneFS CLI Administration guide

for your version of OneFS.

No

Using the same output, verify that the status of Gat eway Connect i vi t y

St at us field is Connect ed.

Yes

Is the gateway connectivity connected?

The username and password that you are using should be the same credentials you use to access the Dell EMC Support site.

Contact your Account Team to ensure your ESRS configuration is properly configured.

Return to Page 14

Yes

No










- Page 14 - SRS with OneFS 8.1.x or later

In the OneFS web administration interface, navigate to Cluster Management > General Settings > Remote Support

and look for this error:

"The OneFS Administration Interface does not support the version of of ESRS that is currently running on your cluster"

"Your OneFS license is unsigned"

Is this error present?

SeeIsilon OneFS 8.1 : Your OneFS license is unsigned, article 511053Yes

Are you having issues while trying to upload log files to Dell

EMC Support?

No

SeeOneFS 8.1 reporting 401 errors from secondary ESRS gateway, when attempting to upload, article 514581

Yes



No









Contact Isilon Technical Support

If you need to contact Isilon Technical Support during troubleshooting, reference the page or step that you need help with. This information and the log file will help Isilon Technical Support staff resolve your case more quickly.

Appendix A: If you need further assistance

Upload node log files and the screen log file to Isilon Technical Support

1. When troubleshooting is complete, in the command-line interface, type exi t to end your screen session.

2. Gather and upload the node log set and include the SSH screen log file by using the command appropriate for your method of uploading files. If you are not sure which method to use, use FTP.

ESRS:i si _gat her _i nf o - - esr s - - l ocal - onl y - f / i f s/ dat a/ I si l on_Suppor t / scr eenl og. 0

FTP: i si _gat her _i nf o - - f t p - - l ocal - onl y - f / i f s/ dat a/ I si l on_Suppor t / scr eenl og. 0

HTTP: i si _gat her _i nf o - - ht t p - - l ocal - onl y - f / i f s/ dat a/ I si l on_Suppor t / scr eenl og. 0

SupportIQ: Copy and paste the following command. Note: When you copy and paste the command into the command-line interface, it will appear on multiple lines (exactly as it appears on the page), but when you press Enter, the command will run as it should.

i si _gat her _i nf o - - l ocal - onl y - f / i f s/ dat a/ I si l on_Suppor t / scr eenl og. 0 - - noupl oad \

- - syml i nk / var / cr ash/ Suppor t I Q/ upl oad/ f t p

3. If you receive a message that the upload was unsuccessful, refer to article 304567 for directions on how to upload files over FTP.

___________









Decision diamondYes No

Process stepProcess step with command:

command xyz

Go to Page #

Page # Note

Provides context and additional information. Sometimes a note is linked to a process step with a colored dot.

CAUTION!Caution boxes warn that a particular step needs to be performed with great care, to prevent serious consequences.

End pointDocument ShapeCalls out supporting documentation for a process step. When possible, these shapes contain links to the reference document.Sometimes linked to a process step with a colored dot.

Optional process step

IntroductionDescribes what the section helps you to accomplish.


- Page 4 - Start Troubleshooting

Appendix B: How to use this flowchart

Directional arrows indicate the path through the process flow.








- Page 15 - SRS with OneFS 8.1.x or later (2)

Appendix C: Example output

Example outputcl ust er - 2# i si esr s vi ew Enabl ed: Yes Pr i mar y ESRS Gat eway: 192. 168. 1. 1 Secondar y ESRS Gat eway: 192. 168. 1. 2 Al er t on Di sconnect : Yes Gat eway Access Pool s: subnet 0: pool 0 Gat eway Connect i vi t y Check Per i od: 3600Li cense Usage I nt el l i gence Repor t i ng Per i od: 86400 Downl oad Enabl ed: No ESRS Fi l e Downl oad Ti meout Per i od: 50 ESRS Fi l e Downl oad Er r or Ret r i es: 3 ESRS Fi l e Downl oad Chunk Si ze: 1000000 ESRS Downl oad Fi l esyst em Li mi t : 80 Gat eway Connect i vi t y St at us: Connect ed







Copyright © 2020 Dell Inc. or its subsidiaries. All rights reserved.

Dell believes the information in this publication is accurate as of its publication date. The information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED "AS-IS." DELL MAKES NO REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. USE, COPYING, AND DISTRIBUTION OF ANY DELL SOFTWARE DESCRIBED IN THIS PUBLICATION REQUIRES AN APPLICABLE SOFTWARE LICENSE.

Dell, EMC, and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be the property of their respective owners.

EMC CorporationHopkinton, Massachusetts 01748-91031-508-435-1000 in North America 1-866-464-7381www.EMC.com




https://www.emc.com

troubleshoot problems with secure remote …...services (srs) contents and overview before you begin...

Documents