grid services monitoring emi ipv6 testbed dusan klinec supervisor: andrew elwell it-gt-sl

53
Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

Upload: lindsey-harper

Post on 20-Jan-2016

224 views

Category:

Documents


2 download

TRANSCRIPT

Page 1: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

Grid services monitoringEMI IPv6 testbed

Dusan KlinecSupervisor: Andrew Elwell

IT-GT-SL

Page 2: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

Outline Brief introduction to IPv6 enabled server code

Nagios dual stack service monitoring

Nagios probes extension to support dual stack check

EMI IPv6 Testbed

Page 3: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

IPv6 enabled server

Page 4: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

05/09/2012 4

Dual stackServer binds both 0.0.0.0, :: addresses

Page 5: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

socket(PF_INET6, SOCK_STREAM, IPPROTO_TCP) = 4setsockopt(4, SOL_SOCKET, SO_SNDBUF, [65536], 4) = 0setsockopt(4, SOL_SOCKET, SO_RCVBUF, [65536], 4) = 0setsockopt(4, SOL_TCP, TCP_NODELAY, [1], 4) = 0fcntl(4, F_GETFL) = 0x2 (flags O_RDWR)fcntl(4, F_SETFL, O_RDWR|O_NONBLOCK) = 0setsockopt(4, SOL_IPV6, IPV6_V6ONLY, [1], 4) = 0bind(4, {sa_family=AF_INET6, sin6_port=htons(12366), inet_pton(AF_INET6, "::", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = 0Listen(4, 100) = 0

socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 5setsockopt(5, SOL_SOCKET, SO_SNDBUF, [65536], 4) = 0setsockopt(5, SOL_SOCKET, SO_RCVBUF, [65536], 4) = 0setsockopt(5, SOL_TCP, TCP_NODELAY, [1], 4) = 0fcntl(5, F_GETFL) = 0x2 (flags O_RDWR)fcntl(5, F_SETFL, O_RDWR|O_NONBLOCK) = 0bind(5, {sa_family=AF_INET, sin_port=htons(12366), sin_addr=inet_addr("0.0.0.0")}, 16) = 0listen(5, 100) = 0

select(6, [4 5], [4 5], [4 5], {30, 0}) = 0 (Timeout)select(6, [4 5], [4 5], [4 5], {30, 0}) = 1 (in [5], left {12, 929341})fcntl(5, F_GETFL) = 0x802 (flags O_RDWR|O_NONBLOCK)fcntl(5, F_SETFL, O_RDWR|O_NONBLOCK) = 0accept(5, {sa_family=AF_INET, sin_port=htons(34736), sin_addr=inet_addr("127.0.0.1")}, [16]) = 6setsockopt(6, SOL_SOCKET, SO_SNDBUF, [65536], 4) = 0setsockopt(6, SOL_SOCKET, SO_RCVBUF, [65536], 4) = 0setsockopt(6, SOL_TCP, TCP_NODELAY, [1], 4) = 0recvfrom(6, "POST / HTTP/1.1\r\nHost: 127.0.0.1"..., 65536, 0, NULL, NULL) = 606

Page 6: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

05/09/2012 6

Dual stackServer binds :: address, using IPv4 mapped addresses (RFC

4038)

...but: IPv4 mapped addresses – not supported by OpenBSD, Windows {2000, XP, 2003}

Page 7: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

05/09/2012 7

Dual stackServer binds only 0.0.0.0

=> Service is unavailable on IPv6 socket

Problem if: Host has AAAA DNS record Client prefers IPv6 to IPv4 In case of fail client doesn't try IPv4

OK if:Clients are aware of this (MySQL) => no try for IPv6 connection

Page 8: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

05/09/2012 8

Getaddrinfo() IP protocol version agnostic Used for DNS queries Used for acquiring addresses for bind() Returns linked list

The sorting function used within getaddrinfo() is defined in RFC 3484 The order can be tweaked for a particular system by editing /etc/gai.conf /etc/gai.conf

Page 9: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

Real world example 1[root@gtv6-emi14 ~]# cat /etc/redhat-release Scientific Linux release 6.3 (Carbon)

[root@gtv6-emi14 ~]# uname -aLinux gtv6-emi14 2.6.32-131.2.1.el6.x86_64 #1 SMP Thu Jun 2 09:49:26 \CDT 2011 x86_64 x86_64 x86_64 GNU/Linux

[root@gtv6-emi14 ~]# ./getaddrinfo.app #00: IPv4 address: 0.0.0.0 (-)#01: IPv6 address: :: (-)

[root@gtv6-emi14 ~]# netstat -tunap | grep srmTcp 0 0 0.0.0.0:8446 0.0.0.0:* LISTEN 1092/srmv2.2

Page 10: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

ConsequenceClient is unable to connect via IPv6

SYN: 14:02:57.041337 IP6 2001:1458:301:a873::100:140.53729 > 2001:1458:301:a87c::100:1cd.8446: S 117254423:117254423(0) win 5760 <mss 1440,sackOK,timestamp 2159405331 0,nop,wscale 7>

RESET: 14:02:57.041923 IP6 2001:1458:301:a87c::100:1cd.8446 > 2001:1458:301:a873::100:140.53729: R 0:0(0) ack 117254424 win 0

Page 11: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

Consequence detail

Page 12: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

Quick hack[root@gtv6-emi14 ~]# cat /etc/gai.conf label ::/0 0 label 0.0.0.0/0 1 precedence ::/0 40 precedence 0.0.0.0/0 10

[root@gtv6-emi14 ~]# ./getaddrinfo.app #00: IPv6 address: :: (-)#01: IPv4 address: 0.0.0.0 (-)

[root@gtv6-emi14 ~]# netstat -tunap | grep srmTcp 0 0 :::8446 :::* LISTEN 4215/srmv2.2

Page 13: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

Nagios dual stack monitoring

Page 14: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

05/09/2012 14

Nagios checks dual stack

Before using NCG patch

After using NCG patch

Page 15: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

05/09/2012 15

Nagios configuration

Page 16: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

• Metric configuration change to declare dual stack check support:

$WLCG_SERVICE->{'org.sam.SRM-All'}->{native} = "Nagios";

$WLCG_SERVICE->{'org.sam.SRM-All'}->{config} = {%{$SERVICE_TEMPL->{60}}};

$WLCG_SERVICE->{'org.sam.SRM-All'}->{probe} = 'org.sam/SRM-probe';

$WLCG_SERVICE->{'org.sam.SRM-All'}->{metricset} = "org.sam.SRM";

$WLCG_SERVICE->{'org.sam.SRM-All'}->{dependency}->{"hr.srce.SRM2-CertLifetime"} = 1;

$WLCG_SERVICE->{'org.sam.SRM-All'}->{dependency}->{"hr.srce.GridProxy-Valid"} = 0;

# line declaring that this service supports --4 and --6 switches

$WLCG_SERVICE->{'org.sam.SRM-All'}->{flags}->{'DEFAULTDUALSTACK'} = 1;

05/09/2012 16

NCG Hash.pm

AAAA DNS record is requiredFor host to use dual stack checks

Page 17: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

Probe extension to support dual stack tests

Page 18: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

ProblemWe want probes to support --4, --6 switches

If no switch from {--4,--6} is provided:-> default behavior (let system resolver decide)

If --4 is provided:-> probe MUST use SOME IPv4 address from DNS response for testing service

If --6 is provided:-> probe MUST use SOME IPv6 address from DNS response for testing service

Page 19: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

ProbesSome probes uses clients with support for particular IP stack test => no need to hack in order to test particularstack.

hr.srce.check_nmap_tcp uses nmap (network scanner)[root@gtv6-emi02 ~]# nmap gtv6-emi14.cern.ch -p 8446 -v

Starting Nmap 4.11 ( http://www.insecure.org/nmap/ ) at 2012-09-04 14:53 Initiating ARP Ping Scan against 128.142.136.156 [1 port] at 14:53Initiating SYN Stealth Scan against gtv6-emi14.cern.ch (128.142.136.156) Interesting ports on gtv6-emi14.cern.ch (128.142.136.156):PORT STATE SERVICE8446/tcp open unknownMAC Address: 00:15:5D:FF:53:79 (Microsoft)

Page 20: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

ProbesSome probes uses clients with support for particular IP stack test => no need to hack in order to test particularstack.

hr.srce.check_nmap_tcp uses nmap (network scanner)[root@gtv6-emi02 ~]# nmap gtv6-emi14.cern.ch -p 8446 -v -6

Starting Nmap 4.11 ( http://www.insecure.org/nmap/ ) at 2012-09-04 14:55 Machine 2001:1458:301:a87c::100:1cd is actually LISTENING on probe port 80Initiating Connect() Scan against gtv6-emi14.cern.ch \

(2001:1458:301:a87c::100:1cd) [1 port] at 14:55Discovered open port 8446/tcp on 2001:1458:301:a87c::100:1cdInteresting ports on gtv6-emi14.cern.ch (2001:1458:301:a87c::100:1cd):PORT STATE SERVICE8446/tcp open unknown

Page 21: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

05/09/2012 21

WrappingWrapping does not work with many probes (BDII lookup):

2012-09-04T12:44:35ZQuerying BDII ldap://emiipv6bdiit.cern.ch:21702012-09-04T12:44:35ZNo information for [base: o=grid; filter:

(|(&(GlueChunkKey=GlueSEUniqueID=128.142.136.156)(|(GlueSAAccessControlBaseRule=dteam)(GlueSAAccessControlBaseRule=VO:dteam)))(&(GlueChunkKey=GlueSEUniqueID=128.142.136.156)(|(GlueVOInfoAccessControlBaseRule=dteam)(GlueVOInfoAccessControlBaseRule=VO:dteam)))(&(GlueServiceUniqueID=*://128.142.136.156*)(GlueServiceVersion=2.*)(GlueServiceType=srm*))); attribute(s): ['GlueServiceEndpoint', 'GlueSAPath', 'GlueVOInfoPath']] in [ldap://emiipv6bdiit.cern.ch:2170 [128.142.140.128]].

CRITICAL: METRIC FAILED [org.sam.SRM-GetSURLs]: CRITICAL: No information for [attribute(s):

['GlueServiceEndpoint', 'GlueSAPath', 'GlueVOInfoPath']] in [ldap://emiipv6bdiit.cern.ch:2170 [128.142.140.128]].

Page 22: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

Resolver

Page 23: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

Resolver

Page 24: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

Resolver

Page 25: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

Resolver

Page 26: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

Resolver

Page 27: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

Resolver

Page 28: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

Resolver

Page 29: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

Resolver

Page 30: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

05/09/2012 30

Using resolver

2012-09-04T11:50:55ZQuerying BDII ldap://gtv6-emi03.cern.ch:21702012-09-04T11:50:55ZGlueServiceEndpoint: httpg://gtv6-emi14.cern.ch:8446/srm/managerv2Resolving gtv6-emi14.cern.ch to 128.142.136.156 SRM endpoint(s) to test:srm://128.142.136.156:8446/srm/managerv2?SFN=

/dpm/cern.ch/home/dteam

2012-09-04T12:06:17ZQuerying BDII ldap://gtv6-emi03.cern.ch:21702012-09-04T12:06:17ZGlueServiceEndpoint: httpg://gtv6-emi14.cern.ch:8446/srm/managerv2GlueVOInfoPath: /dpm/cern.ch/home/dteamResolving gtv6-emi14.cern.ch to [2001:1458:301:a87c::100:1cd] SRM endpoint(s) to test:srm://[2001:1458:301:a87c::100:1cd]:8446/srm/managerv2?SFN=

/dpm/cern.ch/home/dteam

With --4 switch

With –-6 switch

Page 31: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

Gridmon python probes

resolver.setRecord('cern.ch' , '127.0.0.1' )resolver.resolve('cern.ch' ) # will return 127.0.0.1resolver.unsetRecord('cern.ch' )resolver.resolve('cern.ch' ) # will return cern.ch

def setResolver (self , resolver):"""Set another resolver to probe"""def resolveHost(self, host):"""Resolve host with internal resolver, if none - use identity"""

endpoint2=endpoint.replace(self.hostName, \self.resolveHost(self.hostName))

Framework for Nagios probes Central metric invocation => suitable for extension

Resolver object:

Methods in base class:

Usage in metric (base class takes care about resolving):

Page 32: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

Gridmon perl probesLibrary for Nagios probes

Lack of central metric invocation, only helper classes

Usage:use GridMon::DualStackUtils qw( &getResolver );use GridMon::DualStackResolver;use Socket qw( AF_INET AF_INET6 );

my $resolver = getResolver( $plugin->opts->hostname, Socket::AF_INET);$ENV{ DPNS_HOST} = $resolver->resolve( $plugin->opts->hostname);

Page 33: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

Want to know more?

See documentation:

https://tomtools.cern.ch/jira/browse/GTSL-32https://tomtools.cern.ch/jira/browse/GTSL-33

Page 34: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI IPv6 testbed

Page 35: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

TestbedSite name: cert-tb6-cernVO: emiipv6

Page 36: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

TestbedService Host name IPv6 TestedBDII Top emiipv6bdiit.cern.

ch

BDII Site gtv6-emi03.cern.ch

VOMS gtv6-emi06.cern.ch

MyProxy gtv6-emi08.cern.ch

DPM head+disk gtv6-emi04.cern.ch

DPM head+disk gtv6-emi14.cern.ch

LFC gtv6-emi07.cern.ch

FTS+FTA gtv6-emi11.cern.ch

CREAM, Torque emi-ipv6-ce.cern.ch

WN gtv6-emi05.cern.ch

ARGUS gtv6-emi13.cern.ch

APEL emiipv6apel.cern.ch

Missing WMS/RMGA

L&B gtv6-emi12.cern.ch

Missing WMS

SAM Nagios gtv6-emi02.cern.ch

Page 37: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

Java IPv6 compliance“Using IPv6 in Java is easy; it is transparent and automatic. Unlike in many other languages, no porting is necessary. In fact, there is no need to even recompile the source files.”

=> java based services should work with high probability

[ http://docs.oracle.com/javase/1.5.0/docs/guide/net/ipv6_guide/index.html ]

Page 38: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

Java based servicesService Host name IPv6 TestedBDII Top emiipv6bdiit.cern.

ch

BDII Site gtv6-emi03.cern.ch

VOMS gtv6-emi06.cern.ch

MyProxy gtv6-emi08.cern.ch

DPM head+disk gtv6-emi04.cern.ch

DPM head+disk gtv6-emi14.cern.ch

LFC gtv6-emi07.cern.ch

FTS gtv6-emi11.cern.ch

CREAM emi-ipv6-ce.cern.ch

WN gtv6-emi05.cern.ch

ARGUS gtv6-emi13.cern.ch

APEL emiipv6apel.cern.ch

L&B gtv6-emi12.cern.ch

SAM Nagios gtv6-emi02.cern.ch

Page 39: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

Does it really work?

Page 40: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

Service testing All daemons are running, according to Systemadministrator guide or service reference card.

No critical errors or important warnings were foundin log files

SAM Nagios for all services-> no critical problem reported

Tested by user interface (UI) client applications (if applicable -> service has all needed dependencies to run)

no error found, everything works test of high-level services (FTS, CE) documentedin testing protocol ({packet trace, log files, system call trace} available)

Page 41: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

FTS test coverageService Host name IPv6 TestedBDII Top emiipv6bdiit.cern.

ch

BDII Site gtv6-emi03.cern.ch

VOMS gtv6-emi06.cern.ch

MyProxy gtv6-emi08.cern.ch

DPM head+disk gtv6-emi04.cern.ch

DPM head+disk gtv6-emi14.cern.ch

LFC gtv6-emi07.cern.ch

FTS+FTA gtv6-emi11.cern.ch

CREAM, Torque emi-ipv6-ce.cern.ch

WN gtv6-emi05.cern.ch

ARGUS gtv6-emi13.cern.ch

APEL emiipv6apel.cern.ch

L&B gtv6-emi12.cern.ch

SAM Nagios gtv6-emi02.cern.ch

Page 42: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

CE Test coverageService Host name IPv6 TestedBDII Top emiipv6bdiit.cern.

ch

BDII Site gtv6-emi03.cern.ch

VOMS gtv6-emi06.cern.ch

MyProxy gtv6-emi08.cern.ch

DPM head+disk gtv6-emi04.cern.ch

DPM head+disk gtv6-emi14.cern.ch

LFC gtv6-emi07.cern.ch

FTS+FTA gtv6-emi11.cern.ch

CREAM, Torque emi-ipv6-ce.cern.ch

WN gtv6-emi05.cern.ch

ARGUS gtv6-emi13.cern.ch

APEL emiipv6apel.cern.ch

L&B gtv6-emi12.cern.ch

SAM Nagios gtv6-emi02.cern.ch

Page 43: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

Testbed summary

All deployed services are running Majority of services was tested by client programs Services are being tested by Nagios probes Grid testbed is usable - submit job, transmission job,

dpm, voms, etc... (no problem was found)

But it is still not perfect...

Page 44: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

IPv4 only servicesService Host name IPv4-OnlyBDII Top emiipv6bdiit

BDII Site gtv6-emi03

VOMS gtv6-emi06 VOMS entry 15000

MyProxy gtv6-emi08 MyProxy srvr 7512

DPM head+disk gtv6-emi04

DPM head+disk gtv6-emi14

LFC gtv6-emi07

FTS+FTA gtv6-emi11

CREAM, Torque emi-ipv6-ce Bnotifier, MAUI, PBS_server

WN gtv6-emi05 PBS_mom (u1023, 15002, 15003)

ARGUS gtv6-emi13

APEL emiipv6apel

L&B gtv6-emi12

SAM Nagios gtv6-emi02

Page 45: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

Strange binding svcs

ARGUS:tcp 0 0 ::ffff:128.142.18.55:8150 :::* LISTEN 55256/java tcp 0 0 ::ffff:128.142.18.55:8152 :::* LISTEN 61197/java tcp 0 0 ::ffff:128.142.18.55:8154 :::* LISTEN 61102/java

Cause: YAIM configuration uses host FQDN to specify socket to bind -> by DNS resolved to public IP address

[root@gtv6-emi13 ~]# telnet 127.0.0.1 8150Trying 127.0.0.1...telnet: connect to address 127.0.0.1: Connection refused

[root@gtv6-emi13 ~]# telnet 128.142.18.55 8150Trying 128.142.18.55...Connected to 128.142.18.55.Escape character is '^]'.

Page 46: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

Tools for testing

Page 47: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

Manipulated DNS serverHostname: emiipv6dns.cern.chRecursive DNS server + ...Uses Response Policy Zone (RPZ) mechanism in orderto answer on DNS queries from foreign zone with definedanswer – ability to tamper DNS responses.

[root@emiipv6dns dklinec]# cat /var/named/rpz $TTL 60@ IN SOA localhost. root.localhost. ( 100 ; serial 10m ; refresh 10m ; retry 10m ; expiry 10m) ; minimum IN NS localhost.

; just testing recordnon-existing-domain.com CNAME www.cern.ch.

emi-ipv6-ce.cern.ch IN A 137.138.163.53; hide IPv6 record;emi-ipv6-ce.cern.ch IN AAAA 2001:1458:201:b30a:215:5dff:feff:449b

Page 48: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

Getaddrinfo() res. order

[root@gtv6-emi14 ~]# ./getaddrinfo.app #00: IPv6 address: :: (-)#01: IPv4 address: 0.0.0.0 (-)

Returns result of getaddrinfo() suitable for bindingUsing PF_UNSPEC, AI_PASSIVE

Usage:

Page 49: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

Port binding checkHelps to reveal IPv4-Only services, not properly configuredservices, firewall configuration problems#################################################################################Netstat analysis host: gtv6-emi13.cern.ch################################################################################All listening services:tcp 2170 (8076/slapd) |W: 0.0.0.0 1 :: 0tcp 8150 (55256/java) |W: 0.0.0.0 0 :: 1tcp 8152 (61197/java) |W: 0.0.0.0 0 :: 1tcp 8154 (61102/java) |W: 0.0.0.0 0 :: 1

IPv6 Only services:tcp 8150 (55256/java) |W: 0.0.0.0 0 :: 1tcp 8152 (61197/java) |W: 0.0.0.0 0 :: 1tcp 8154 (61102/java) |W: 0.0.0.0 0 :: 1

IPv4 Only services!!! :tcp 2170 (8076/slapd) |W: 0.0.0.0 1 :: 0

Results:L4 Protocol: tcp! Problem with tcp 8150 (55256/java) on :: IPv4: 0 IPv6: 1; Closed port on 2001:1458:301:a868::100:2a! Problem with tcp 8152 (61197/java) on :: IPv4: 0 IPv6: 1; Closed port on 2001:1458:301:a868::100:2a! Problem with tcp 8154 (61102/java) on :: IPv4: 0 IPv6: 1; Closed port on 2001:1458:301:a868::100:2a

Page 50: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

$ ./wrapper.py --cmd './longtest.sh' --strace --tcpdump --cmdid longtest \--destdir /tmp/longtest/ --prefix alfa --suffix t0 --wait 10## Starting TCPDump: /usr/bin/sudo /usr/sbin/tcpdump -w "/tmp/longtest//alfa-tcpdump-longtestt0.pcap"## Starting work job: ./longtest.sh## Starting blocking operation: ['/usr/bin/strace', '-f', '-s', '512', '-v', '-o','/tmp/longtest//alfa-strace-longtest-t0', '--', './longtest.sh']## Thread should be stopped now: ['/usr/bin/strace', '-f', '-s', '512', '-v', '-o','/tmp/longtest//alfa-strace-longtest-t0', '--', './longtest.sh']## Work finished!## Stdout+stderr (/tmp/longtest//alfa-stdout-longtest-t0):================================================================================PING 128.142.18.54 (128.142.18.54) 56(84) bytes of data.64 bytes from 128.142.18.54: icmp_req=1 ttl=58 time=0.965 ms64 bytes from 128.142.18.54: icmp_req=2 ttl=58 time=1.24 ms64 bytes from 128.142.18.54: icmp_req=3 ttl=58 time=1.08 ms--- 128.142.18.54 ping statistics ---3 packets transmitted, 3 received, 0% packet loss, time 2003msrtt min/avg/max/mdev = 0.965/1.097/1.243/0.117 msOKEndingNow!================================================================================## Going to sleep for 10 seconds.^C## Exception reported, ending waiting,## Stopping tcpdumps## Stopping dumper <cmdRunner(Thread-2, started 140725401147136)>## Stopping tailers## Thread should be stopped now: ['/usr/bin/sudo', '/usr/sbin/tcpdump', '-w', '/tmp/longtest//alfatcpdump-longtest-t0.pcap']

Artifact collection (wrapper)

Page 51: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

Artifact collection (wrapper)

Files produced:$ ll /tmp/longtest//total 372-rw-r--r-- 1 root root 404 Aug 31 12:40 alfa-stdout-longtest-t0-rw-r--r-- 1 root root 29234 Aug 31 12:40 alfa-strace-longtest-t0.29352-rw-r--r-- 1 root root 6054 Aug 31 12:40 alfa-strace-longtest-t0.29353-rw-r--r-- 1 root root 17483 Aug 31 12:40 alfa-strace-longtest-t0.29354-rw-r--r-- 1 root root 6054 Aug 31 12:40 alfa-strace-longtest-t0.29355-rw-r--r-- 1 root root 164 Aug 31 12:40 alfa-tcpdump-longtest-t0.out-rw-r--r-- 1 root root 236649 Aug 31 12:40 alfa-tcpdump-longtest-t0.pcap

Page 52: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI I

NFS

O-R

I-261

611

Artifact collection (server side)

[root@gtv6-emi11 glite-transfer-submit]# lltotal 8128-rw-r--r--. 1 root root 0 Aug 30 17:17 logfile-fts-catalina.out-t0-rw-r--r--. 1 root root 141654 Aug 30 17:20 logfile-fts-glite-transfer-channel-agent-urlcopy-CERN3-CERN3.log-t0-rw-r--r--. 1 root root 138377 Aug 30 17:20 logfile-fts-glite-transfer-vo-agent-DTEAM.log-t0-rw-r--r--. 1 root root 145599 Aug 30 17:20 logfile-fts-glite-transfer-vo-agent-EMIIPV6.log-t0-rw-r--r--. 1 root root 2982 Aug 30 17:19 logfile-fts-org.glite.data-t0-rw-r--r--. 1 root root 0 Aug 30 17:17 logfile-system-messages-t0-rw-r--r--. 1 root root 0 Aug 30 17:17 logfile-system-secure-t0-rw-r--r--. 1 root root 1784 Aug 30 17:17 strace-fts- glite_transfer_channel_agent_urlcopy_CERN3_CERN3.63194-t0.13001-rw-r--r--. 1 root root 16392 Aug 30 17:17 strace-fts-glite_transfer_channel_agent_urlcopy_CERN3_CERN3.63194-t0.13002-rw-r--r--. 1 root root 45 Aug 30 17:17 strace-fts-glite_transfer_vo_agent_DTEAM.59354-t0.59354-rw-r--r--. 1 root root 788651 Aug 30 17:20 strace-fts-glite_transfer_vo_agent_DTEAM.59354-t0.59355-rw-r--r--. 1 root root 45 Aug 30 17:17 strace-fts-glite_transfer_vo_agent_EMIIPV6.59373-t0.59373-rw-r--r--. 1 root root 902175 Aug 30 17:20 strace-fts-glite_transfer_vo_agent_EMIIPV6.59373-t0.59374-rw-r--r--. 1 root root 45 Aug 30 17:17 strace-fts- _usr_lib_jvm_jre_1_6_0_openjdk_x86_64_bin_java.57775-t0.57775-rw-r--r--. 1 root root 51 Aug 30 17:17 strace-fts-_usr_lib_jvm_jre_1_6_0_openjdk_x86_64_bin_java.57775-t0.57780-rw-r--r--. 1 root root 129817 Aug 30 17:19 strace-fts-_usr_lib_jvm_jre_1_6_0_openjdk_x86_64_bin_java.57775-t0.58739-rw-r--r--. 1 root root 162 Aug 30 17:20 tcpdump-fts-t0-rw-r--r--. 1 root root 371476 Aug 30 17:20 tcpdump-fts-t0.pcap-rw-r--r--. 1 root root 164 Aug 30 17:20 tcpdump-system-t0-rw-r--r--. 1 root root 2772184 Aug 30 17:20 tcpdump-system-t0.pcap

Page 53: Grid services monitoring EMI IPv6 testbed Dusan Klinec Supervisor: Andrew Elwell IT-GT-SL

EMI is partially funded by the European Commission under Grant Agreement RI-261611

Questions?

05/09/2012 53