Experience in implementing SSL between Oracle DB and Oracle Clients

Agenda

● Project overview

● SSL session (1 way/2 ways)

● Certificate Authorities & Certificates

● 3 Different Oracle Listeners configurations (insecure/secure/ssl)

● SSL & Oracle Clients setup (tns configuration parameters)

● Issues faced & troubleshooting tools

● Conclusion

● Q & A

RAC Cluster

Project overview

SSL ListenerSEC Listener

SCAN Listeners

Developers & End users / tools directly

connecting to DB

Main application technology stack

OBIEE, WLS, Forms, Reports

IntegrationsOther DBs, Essbase,

Ora Net Clients

SSL ListenerSEC Listener

SSL ListenerSEC Listener

How does SSL/TLS work?

How does SSL/TLS work? from http://security.stackexchange.com/ => here

How does SSL work? What is an SSL handshake? from http://www.symantec.com/ => here

Transport Layer Security from https://en.wikipedia.org/ => here

SSL session overview (1 way)

ServerInitial request (to https://domainname.com)Hello .... SSLv2, SSLv3, TLSv1, .... ciphers, hash function

● Server Certificate● Server Private Key

Server Certificate (signed by Certificate Authority) Public Key + Domain Name (subject) + Org name ....● List of Trusted

CA certificates

Establish trust ...

Agree on the symmetric encryption key ...

premaster secret

master secret & generate session keys

Use session key and agreed encryption cipher to send data ...

SSL session overview (2 way)

Server

● Server Certificate● Server Private Key● List of Trusted

CA certificates

Establish trust ...

Agree on the symmetric encryption key ...

master secret & generate session keys

Use session key and agreed encryption cipher to send data ...

premaster secret

● Client Certificate● Client Private Key

● List of TrustedCA certificates

● There are 3 options to obtain an SSL certificate

a. Generate your own self-signed certificate (from https://www.linux.com here)

b. Request certificate from your organisation CA

c. Request certificate from public CA

■ VeriSign

■ GeoTrust

■ ... https://en.wikipedia.org/wiki/Certificate_authority

● In A & B you or your organisation controls hashing and encryption protocols

● However no clients will recognize certificate by default

● In option C you will need to make sure your client versions supports protocols

dictated by external CA

a. Recent example => Many CAs announced that they will not support SHA-1 forcing SHA256/SHA-2

implementation

Certificate Authorities & Certificates

● Intermediate certificate authorities

○ https://en.wikipedia.org/wiki/Intermediate_certificate_authorities

Certificate Authorities & Certificates

●

Certificate Authorities & Certificates

●

Certificate Authorities & Certificates

●

Certificate Authorities & Certificates

Oracle

Listener Config

3 Different Oracle Listeners configurations

● Default Listener (insecure)

○ Username/password are encrypted

● Secure Listener

○ Use encryption the same way as SSL Listener

○ No authentication phase (SSL handshake)

● SSL Listener

○ Use encryption

○ Could be configured with 1 way or 2 ways authentication

Default Listener (insecure)

listener.ora

LISTENER = (DESCRIPTION = (ADDRESS = (PROTOCOL=TCP)(HOST = db_host)(PORT = 1521)) (ADDRESS = (PROTOCOL=IPC)(KEY = REGLSNR_LISTENER)) )

Secure Listener

listener.ora

LISTENER = (DESCRIPTION = (ADDRESS = (PROTOCOL=TCP)(HOST = db_host)(PORT = 1522)) (ADDRESS = (PROTOCOL=IPC)(KEY = REGLSNR_LISTENER)) )

sqlnet.ora

sqlnet.crypto_checksum_server = REQUIREDsqlnet.encryption_server = REQUIREDsqlnet.crypto_checksum_types_server = (SHA1)sqlnet.encryption_types_server = (AES256)

listener.ora

LISTENER = (DESCRIPTION = (ADDRESS = (PROTOCOL=TCPS)(HOST = db_host)(PORT = 1523)) (ADDRESS = (PROTOCOL=IPC)(KEY = REGLSNR_LISTENER)) )

WALLET_LOCATION = (SOURCE = (METHOD = FILE) (METHOD_DATA = (DIRECTORY = /u01/app/oracle/SSL/certs) ) )

SSL_CLIENT_AUTHENTICATION = FALSESSL_CIPHER_SUITES = (SSL_RSA_WITH_AES_256_CBC_SHA)

SSL Listener

oracle@host > openssl s_client -connect db_host:1523CONNECTED(00000003)depth=0 CN = PROD...verify return:1Certificate chain 0 s:/CN=PROD i:/CN=PROD---Server certificate-----BEGIN CERTIFICATE-----MIIBkDCB+gIBADANBgkqhkiG9w0BAQQFADARMQ8wDQYDVQQDEwZFQlNMTlAwHhcNMTQwNzAyMDQzNTI4WhcNNDExMTE2MDUzNTI4WjARMQ8wDQYDVQQDEwZFQlNMTlAwgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAJGbN1hxmtFKx1SpUU8JPvRyjMzW...-----END CERTIFICATE-----subject=/CN=PRODissuer=/CN=PROD---Acceptable client certificate CA names/C=US/O=VeriSign, Inc./OU=Class 1 Public Primary Certification Authority/C=US/O=GTE Corporation/OU=GTE CyberTrust Solutions, Inc./CN=GTE CyberTrust Global Root/C=US/O=RSA Data Security, Inc./OU=Secure Server Certification Authority/C=US/O=VeriSign, Inc./OU=Class 2 Public Primary Certification Authority/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA/C=US/O=Equifax/OU=Equifax Secure Certificate Authority---SSL handshake has read 1269 bytes and written 210 bytes---New, TLSv1/SSLv3, Cipher is AES256-SHA...

SSL & Oracle Clients setup

SSL & Oracle Clients setup...---SSL handshake has read 1269 bytes and written 210 bytes---New, TLSv1/SSLv3, Cipher is AES256-SHAServer public key is 1024 bitSecure Renegotiation IS supportedCompression: NONEExpansion: NONESSL-Session: Protocol : TLSv1 Cipher : AES256-SHA Session-ID: Session-ID-ctx: Master-Key: 68E402B5CD161618522350EE... Key-Arg : None Krb5 Principal: None PSK identity: None PSK identity hint: None Start Time: 1447632207 Timeout : 300 (sec) Verify return code: 18 (self signed certificate)---oracle@host >

Client Config

== TCP ==========================================================================tnsnames.oraPROD = (DESCRIPTION =

(ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCP)(HOST = prod-db-scan.corp.myorg.lv)(PORT = 1523)) ) (CONNECT_DATA = (SERVICE_NAME = PROD_ETL01) ) )

== TCPS =========================================================================tnsnames.oraPROD_SSL = (DESCRIPTION = (SECURITY=(SSL_SERVER_CERT_DN="CN=prod-db.corp.myorg.lv,O=MyOrg Inc,L=Riga,C=Latvia")) (ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCPS)(HOST = prod-db-scan.corp.myorg.lv)(PORT = 7048)) ) (CONNECT_DATA = (SERVICE_NAME = PROD_ETL01) ) )

SSL & Oracle Clients setup (OCI)

== TCPS ==========================================================================sqlnet.oraSSL_SERVER_DN_MATCH = yesSSL_CLIENT_AUTHENTICATION = FALSESSL_CIPHER_SUITES = (SSL_RSA_WITH_AES_256_CBC_SHA)

WALLET_LOCATION = (SOURCE = (METHOD = FILE) (METHOD_DATA = (DIRECTORY = /u01/app/oracle/SSL/cert) ) )

SSL & Oracle Clients setup (OCI)

● java … -Doracle.net.ssl_server_dn_match=true …

● jdbc:oracle:thin:@(DESCRIPTION =(SECURITY=(SSL_SERVER_CERT_DN="CN=prod-db.corp.myorg.lv,O=MyOrg Inc,L=Riga,C=Latvia"))(ADDRESS_LIST = (ADDRESS = (PROTOCOL = TCPS)(HOST = prod-db-scan.corp.myorg.lv)(PORT = 7048)))(CONNECT_DATA = (SERVICE_NAME = PROD_ETL01)))

● Java has it own certificate store

SSL & Oracle Clients setup (JDBC)

Certificates Config

SSL & Oracle Clients setup (OCI)Using the orapki Utility to Manage PKI Elements

https://docs.oracle.com/database/121/DBSEG/asoappf.htm#DBSEG610

$ ls -l /u01/app/oracle/SSL/cert/*total 16-rw-r--r--@ 1 yvel 5000 1365 Aug 21 2014 cwallet.sso-rw-r--r--@ 1 yvel 5000 1288 Aug 21 2014 ewallet.p12

oracle@host:/home/oracle> orapki wallet display -wallet /u01/app/oracle/SSL/cert ; dateOracle PKI Tool : Version 11.2.0.4.0 - ProductionCopyright (c) 2004, 2013, Oracle and/or its affiliates. All rights reserved.

Requested Certificates:User Certificates:Subject: CN=PRODTrusted Certificates:Subject: OU=Class 1 Public Primary Certification Authority,O=VeriSign\, Inc.,C=USSubject: OU=Equifax Secure Certificate Authority,O=Equifax,C=USSubject: OU=Class 3 Public Primary Certification Authority,O=VeriSign\, Inc.,C=USSubject: CN=GeoTrust Global CA,O=GeoTrust Inc.,C=USSubject: OU=Secure Server Certification Authority,O=RSA Data Security\, Inc.,C=USSubject: OU=Class 2 Public Primary Certification Authority,O=VeriSign\, Inc.,C=US

Wed Sep 9 16:27:36 PDT 2015oracle@kpfp2:/home/oracle>

orapki wallet create -wallet $WALLET -auto_login -pwd $v_pwdorapki wallet remove -wallet $WALLET -trusted_cert_all -pwd $v_pwdorapki wallet display -wallet $WALLET -pwd $v_pwdorapki wallet jks_to_pkcs12 -wallet $WALLET -keystore /full/path/to/certificate.jks -jkspwd <pwd>orapki wallet add -wallet $WALLET -trusted_cert -cert GIAG2.crt -pwd $v_pwd

SSL & Oracle Clients setup (OCI)

SSL & Oracle Clients setup (OCI)$ openssl pkcs12 -info -in /u01/app/oracle/SSL/cert/ewallet.p12Enter Import Password:MAC Iteration 1024MAC verified OKPKCS7 Encrypted data: pbeWithSHA1And3-KeyTripleDES-CBC, Iteration 1024Certificate bagBag Attributes localKeyID: E6 B6 52 DD 00 00 00 04 00 00 00 00 00 00 00 03 00 00 00 00 00 00 00 04subject=/C=US/O=MyOrg Inc/CN=MyOrg Internet Authority G2issuer=/C=US/O=GeoTrust Inc./CN=GeoTrust Global CA-----BEGIN CERTIFICATE-----MIIEBDCCAuygAwIBAgIDAjppMA0GCSqGSIb3DQEBBQUAMEIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1HZW9UcnVzdCBJbmMuMRswGQYDVQQDExJHZW9UcnVzdCBHbG9iYWwgQ0EwHhcNMTMwNDA1MTUxNTU1WhcNMTUwNDA0MTUxNTU1WjBJMQswCQYDVQQGEwJVUzETMBEGA1UEChMKR29vZ2xlIEluYzElMCMGA1UEAxMcR29vZ2xlIEludGVybmV0IEF1dGhvcml0eSBHMjCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJwqBHdc2FCROgajguDYUEi8iT/xGXAaiEZ+4I/F8YnOIe5a/mENtzJEiaB0C1NPVaTOgmKV7utZX8bhBYASxF6UP7xbSDj0U/ck5vuR6RXEz/RTDfRK/J9U3n2+oGtvh8DQUB8oMANA2ghzUWx//zo8pzcGjr1LEQTrfSTe5vn8MXH7lNVg8y5Kr0LSy+rE...

PKCS 12 => https://en.wikipedia.org/wiki/PKCS_12It defines an archive file format for storing many cryptography objects as a single file. It is commonly used to bundle a private key with its X.509 certificate.

● JDBC Clients

○ Have it own default certificate store with preloaded trusted CA list

$ v_java_cert=/Library/Java/JavaVirtualMachines/jdk1.8.0_60.jdk/Contents/Home/jre/lib/security/cacerts$ keytool -list -keystore $v_java_cert -storepass changeit

Keystore type: JKSKeystore provider: SUN

Your keystore contains 93 entries

digicertassuredidrootca, Apr 16, 2008, trustedCertEntry,Certificate fingerprint (SHA1): 05:63:B8:63:0D:62:D7:5A:BB:C8:AB:1E:4B:DF:B5:A8:99:B2:4D:43comodorsaca, May 12, 2015, trustedCertEntry,Certificate fingerprint (SHA1): AF:E5:D2:44:A8:D1:19:42:30:FF:47:9F:E2:F8:97:BB:CD:7A:8C:B4thawtepremiumserverca, May 26, 2015, trustedCertEntry,Certificate fingerprint (SHA1): E0:AB:05:94:20:72:54:93:05:60:62:02:36:70:F7:CD:2E:FC:66:66...

$ keytool -exportcert -alias digicertassuredidrootca -keystore $v_java_cert -storepass changeit -file test.crt -rfc

$ keytool -printcert -file test.crtOwner: CN=DigiCert Assured ID Root CA, OU=www.digicert.com, O=DigiCert Inc, C=USIssuer: CN=DigiCert Assured ID Root CA, OU=www.digicert.com, O=DigiCert Inc, C=USSerial number: ce7e0e517d846fe8fe560fc1bf03039Valid from: Thu Nov 09 16:00:00 PST 2006 until: Sun Nov 09 16:00:00 PST 2031Certificate fingerprints:

MD5: 87:CE:0B:7B:2A:0E:49:00:E1:58:71:9B:37:A8:93:72

SSL & Oracle Clients setup (JBDC)

SSL & Oracle Clients setup (JDBC)

● A JRE client must be updated with the JCE to enable

TLS_RSA_WITH_AES_256_CBC_SHA cipher.

○ Download appropriate JCE archive with 2 jar files

○ Copy the jar files to $JAVA_HOME/jre/lib/security/ directory

● The JCE could be obtained from the URL below depending on a JRE version

○ 1.6 JCE○ 1.7 JCE ○ 1.8 JCE

Issues faced

RAC Cluster

Variety of clients and versions to cover ...

SSL ListenerSEC Listener

SCAN Listeners

Developers & End users using tools

directly connecting to DB

Main application technology stack

OBIEE, WLS, Forms, Reports

IntegrationsOther DBs, Essbase,

Ora Net Clients

SSL ListenerSEC Listener

SSL ListenerSEC Listener

1. 2.3.

Some components used old Oracle Clients

● Essbase

○ Blend in 11.1.0.7 Oracle Client

○ Challenging to update

● OBIEE

○ Presentation layer use 11.1.0.7 Oracle Client

SSL Listener & TCP Buffer

Problem definition

Initial:● DB Instances doesn't register DB Services with a remote SCAN

listeners, after a new certificate has been added to server wallet

Current:● All SSL connections from one node hangs connecting to any SSL

enabled listeners that runs on a remote node if a listener is able to send data out quickly enough

SSL Listener & TCP Buffer

SSL Listener works if ….

a) strace-ing the listener process

b) turning debug logging up to a high level in the listener

c) Running the listener on a non-bonded interface

d) Having the listener increase its send buffer size from 16k (default) to 32k or 64k on the socket

d1) We can also do this at the OS level via `echo "4096 32768 4194304" > /proc/sys/net/ipv4/tcp_wmem'

SSL Listener & TCP Buffer

Normal / Expected processing

sPORT:sIP - cIP:cPORT

TCP socketTCP socket buffer

Processowner of the socket

O_NONBLOCK

W3 W2 W1 DATA

Data writesO_NONBLOCK

sPORT:sIP - cIP:cPORTProcessowner of the socket DATA

Kernel waits a bit for more data and sends it out

Buffer is full. Kernel sends EAGAIN back

5k 1k 3k

3k

W3b

2k

W3bEAGAIN

sPORT:sIP - cIP:cPORTProcessowner of the socket

The process code handles error and keep sending the rest of the data until successful

3k

W3b

3k

Kernel refuses writes until buffer is free

Oracle Listener - fails to process EAGAIN

TCP socketTCP socket bufferO_NONBLOCK

Data writesO_NONBLOCK

sPORT:sIP - cIP:cPORTSSL Listener DATA

Buffer is full. Kernel sends EAGAIN back3k

W3b

2k

W3bEAGAIN

sPORT:sIP - cIP:cPORTSSL Listener

Listener never re-sends the W3b part Kernel ready to process writes after some time

Oracle Listener ssl handshake function fails to process EAGAIN errors

sPORT:sIP - cIP:cPORTSSL Listener DATA

Buffer is full. Kernel sends EAGAIN back

2k

W3b

3k

W3b

The W3b part is dropped

Where is client's response?

W3b

The W3b part is dropped

SSL Listener & TCP Buffer

Conclusions

RAC Cluster

Variety of clients and versions to cover ...

SSL ListenerSEC Listener

SCAN Listeners

Developers & End users using tools

directly connecting to DB

Main application technology stack

OBIEE, WLS, Forms, Reports

IntegrationsOther DBs, Essbase,

Ora Net Clients

SSL ListenerSEC Listener

SSL ListenerSEC Listener

Q & A