dns high-availability tools - open-source load balancing solutions

© Men & Mice http://menandmice.com

DNS High-Availability ToolsOpen-Source Load Balancing

Solutions

11Wednesday 7 December 16

© Men & Mice http://menandmice,com

Resolver HA

• The DNS protocol has built-in high availability for authoritative DNS servers, but client machines can see a degraded DNS service if a DNS resolver (caching DNS server) is failing

• In this webinar, we will look into

• how the DNS clients in popular operating systems (Windows, Linux, macOS/iOS) choose the DNS resolver among a list of available servers

• and how a DNS resolver service can be made failure-tolerant with open-source solutions such as “dnsdist” from PowerDNS and “relayd” from OpenBSD.


http://www.menandmice.com



Authoritative DNS





“”

is.

menandmice.is.

local caching DNS Server

a

b

c

Name Server

RTT

a 3

b 5

c 2

Roundtrip Time





ftp://ftp.menandmice.is.

“”

is.

menandmice.is.


a

b

c

Name Server

RTT

a 3

b 5

c 2

Roundtrip Time




http://www.example.com




“”

is.

menandmice.is.


What is the address of

ftp.menandmice.is.

a

b

c

Name Server

RTT

a 3

b 5

c 2

Roundtrip Time








“”

is.

menandmice.is.


a

b

c

Name Server

RTT

a 3

b 5

c 2

Roundtrip Time








“”

is.

menandmice.is.


a

b

c

Name Server

RTT

a 3

b 5

c 2


ftp.menandmice.is.

Roundtrip Time







2


“”

is.

menandmice.is.


a

b

c

Name Server

RTT

a 3

b 5

c

Roundtrip Time







2


“”

is.

menandmice.is.


a

b

c

Name Server

RTT

a 3

b 5

c

Here is a list of “is.” Name

Servers

Roundtrip Time







338


“”

is.

menandmice.is.


a

b

c

Name Server

RTT

a 3

b 5

c

Here is a list of “is.” Name

Servers

Roundtrip Time







“”

fr.

yahoo.fr.


a

b

c

Name Server

RTT

a 3

b 5

c 338

Roundtrip Time





http://www.yahoo.fr.

“”

fr.

yahoo.fr.


a

b

c

Name Server

RTT

a 3

b 5

c 338

Roundtrip Time








“”

fr.

yahoo.fr.



www.yahoo.fr.

a

b

c

Name Server

RTT

a 3

b 5

c 338

Roundtrip Time






http://www.yahoo.fr

http://www.yahoo.fr



“”

fr.

yahoo.fr.


a

b

c

Name Server

RTT

a 3

b 5

c 338

Roundtrip Time








“”

fr.

yahoo.fr.


a

b

c

Name Server

RTT

a 3

b 5

c 338


www.yahoo.fr.

Roundtrip Time






http://www.yahoo.fr

http://www.yahoo.fr


331

“”

fr.

yahoo.fr.


a

b

c

Name Server

RTT

a 3

b 5

c


Roundtrip Time







331

“”

fr.

yahoo.fr.


a

b

c

Name Server

RTT

a 3

b 5

c

Here is a list of “fr.” Name

Servers


Roundtrip Time







331

85

“”

fr.

yahoo.fr.


a

b

c

Name Server

RTT

a

b 5

c

Here is a list of “fr.” Name

Servers


Roundtrip Time







“”

edu.

berkeley.edu.


a

b

c

Name Server

RTT

a 85

b 5

c 331

Roundtrip Time





dig @ns.berkeley.edu

“”

edu.

berkeley.edu.


a

b

c

Name Server

RTT

a 85

b 5

c 331

Roundtrip Time





dig @ns.berkeley.edu

“”

edu.

berkeley.edu.



ns.berkeley.edu.

a

b

c

Name Server

RTT

a 85

b 5

c 331

Roundtrip Time




http://www.yahoo.fr

http://www.yahoo.fr


dig @ns.berkeley.edu.

“”

edu.

berkeley.edu.


a

b

c

Name Server

RTT

a 85

b 5

c 331

Roundtrip Time






“”

edu.

berkeley.edu.


a

b

c

Name Server

RTT

a 85

b 5

c 331


ns.berkeley.edu.

Roundtrip Time





5

83

324

“”

edu.

berkeley.edu.


a

b

c

Name Server

RTT

a

b

c


Roundtrip Time





5

83

324

“”

edu.

berkeley.edu.


a

b

c

Name Server

RTT

a

b

c

Here is a list of “edu.” Name

Servers


Roundtrip Time





315

83

324

“”

edu.

berkeley.edu.


a

b

c

Name Server

RTT

a

b

c

Here is a list of “edu.” Name

Servers


Roundtrip Time





UNIX / Linux Stub Resolver





UNIX / Linux Stub Resolver

•UNIX/Linux stub resolvers use a configuration file called resolv.conf

•This file is usually found in the /etc directory





Name Server List

• Syntax:

• nameserver <IP address>

• Example:

• nameserver 192.168.0.1

• Notes:

• Most UNIX/Linux servers allow up to 3 nameserver entries

• If multiple are listed, they are queried in the order given





Unix DNS-Client Resolver timeout

Attempt1 DNS-

Resolver2 DNS-Resolver

3 DNS-Resolver

1 5s 2x 5s 3x 5s

2 10s 2x 5s 3x 3s

Total 15s 20s 24s





Unix DNS-Client Resolver timeout

• the Unix-DNS Resolver timeout can be changed in the file /etc/resolv.conf

option timeout:1 attempts:4nameserver 100.64.1.100nameserver 100.64.2.120

• attempts: how many queries send to each DNS resolver (max 5)

• timeout: initial timeout for a query to a name server in resolv.conf (max 30s). For the second and successive rounds of queries, the resolver still doubles the initial timeout and divides by the number of name servers in resolv.conf





Unix DNS-Client Resolver “Round-Robin”

• the order in which the DNS-Resolvers are queried can be tweaked in /etc/resolv.conf

option rotatenameserver 100.64.1.100nameserver 100.64.2.120

•rotate: use all DNS-Resolvers in each resolver-session. Only take effect if the client program sends multiple queries after opening the DNS-Client resolver. Not many programs do this.





Send Client-Resolver options via DHCP (1/2)

•there are not standard DHCP options to transport the attempt, timeout and rotate resolver options

•in the ISC-DHCP Server, add a new option definition (file /etc/dhcp/dhcpd.conf)option resolv-options code 232 = text;option resolv-options "timeout:2 attempts:4 rotate";





Send Client-Resolver options via DHCP (2/2)

•on each ISC-DHCP Client, add a new option definition(file /etc/dhcp/dhclient.conf)option resolv-options code 232 = text;request resolv-options;

•and also add a new DHCP-Script hook (File /etc/dhcp/dhclient-enter-hooks.d/resolvoptions)if [ "$new_resolv_options" ]; then echo "options $new_resolv_options" >> /etc/resolv.conffi





Windows Stub Resolver





Obtain DNS servers via DHCP





Obtain DNS servers via DHCP

Configure listed DNS servers manually





List of additional DNS-Resolver to query





Windows DNS-Client Resolver Timeouts, 1 DNS-Server

Time DNS Query

0s initial query, wait 1s

1s 2nd query, wait 1s

2s 3rd query, wait 2s

4s 4th query, wait 4s

8s 5th query, wait 4s

12s Client-Resolver gives up

https://support.microsoft.com/de-de/kb/2834226







Windows DNS-Client Resolver Timeouts, 2 DNS-Server

Time DNS Query

0sinitial query to 1st DNS server in the

list, wait 1s

1sinitial query to the 2nd DNS server in the

list, wait 1s

2s2nd query to the 2nd DNS server in the

list, wait 2s

4squery to all DNS server in the list,

wait 4s


wait 4s









Windows DNS-Client Resolver Timeouts, 3+ DNS-Server

Time DNS Query

0sinitial query to 1st DNS server in the

list, wait 1s

1sinitial query to the 2nd DNS server in the

list, wait 1s

2sinitial query to the 3rd DNS server in the

list, wait 2s


wait 4s


wait 4s









Adjusting the Windows DNS-CLient timeouts

•The DNS-Client timeouts can be customized using the registry value

HKLM\System\CurrentControlSet\Services\dnscache\Parameters\DNSQueryTimeouts

•This value does not exist by default and then the pre-defined default values are used

• https://blogs.technet.microsoft.com/stdqry/2011/12/02/dns-clients-and-timeouts-part-1/

• https://blogs.technet.microsoft.com/stdqry/2011/12/14/dns-clients-and-timeouts-part-2/




https://blogs.technet.microsoft.com/stdqry/2011/12/02/dns-clients-and-timeouts-part-1/







Demo Setup





DNS-Resolver without HA

30

Internet




31

Internet

172.22.1.210 172.22.1.217




31

Internet

/etc/resolv.confnameserver 172.22.1.210nameserver 172.22.1.217

172.22.1.210 172.22.1.217



Unix resolver demo





OpenBSD relayd





relayd

•relayd is a daemon to relay and dynamically redirect incoming connections to a target host

•available on OpenBSD (and older versions on FreeBSD)

•relayd can dynamically reconfigure the OpenBSD firewall “pf” to redirect traffic

•relayd can also work as an application layer proxy





DNS-Resolver with relayd

35

Internet

172.22.1.210172.22.1.206

172.22.1.217172.22.1.206

CARP-Protocol



DNS-Resolver with relayd

35

Internet

/etc/resolv.confnameserver 172.22.1.206nameserver 172.22.1.210nameserver 172.22.1.217

172.22.1.210172.22.1.206

172.22.1.217172.22.1.206

CARP-Protocol



relayd redirect configuration

36

# Layer 3 forwarding

table <dnsserver> { 172.22.1.210, 172.22.1.217 }

redirect dnsbalance { listen on 172.22.1.206 tcp port 53 listen on 172.22.1.206 udp port 53 forward to <dnsserver> check tcp}

file /etc/relayd.conf



OpenBSD relayd

37

OpenBSD Kernel

Userspace

DNS-Server(BIND 9) relayd

PF-Firewall

Layer 3 redirect



OpenBSD relayd

38

OpenBSD Kernel

Userspace


PF-Firewall

probes

Layer 3 redirect



OpenBSD relayd

39

OpenBSD Kernel

Userspace


PF-Firewall

probes

OK

configuresPF rules

Layer 3 redirect



OpenBSD relayd

40

OpenBSD Kernel

Userspace


PF-Firewall

probes

OK

configuresPF rules

DNS-Query

Layer 3 redirect



OpenBSD relayd

41

OpenBSD Kernel

Userspace


PF-Firewall

probes

OK

configuresPF rules

DNS-Query

DNS-Query

Layer 3 redirect



OpenBSD relayd

42

OpenBSD Kernel

Userspace

DNS-Server(BIND 9)DOWN

relayd

PF-Firewall

probes

Layer 3 redirect



OpenBSD relayd

43

OpenBSD Kernel

Userspace

relayd

PF-Firewall

probes

Not-OK

configuresPF rules


Layer 3 redirect



OpenBSD relayd

44

OpenBSD Kernel

Userspace

relayd

PF-Firewall

probes

Not-OK

configuresPF rules

DNS-Query


Layer 3 redirect



OpenBSD relayd

45

OpenBSD Kernel

Userspace

relayd

PF-Firewall

probes

Not-OK

configuresPF rules

DNS-Query

DNS-Query


Layer 3 redirect



relayd relay configuration

46

# Layer 7 Application Layer Proxy

table <dnsserver> { 172.22.1.210, 172.22.1.217 }

dns protocol "dnsproto"

relay dnsbalance { protocol dnsproto listen on 172.22.1.206 port 53 forward to <dnsserver> check tcp}

file /etc/relayd.conf



OpenBSD relayd

47

OpenBSD Kernel

Userspace


PF-Firewall

Layer 7 proxy



OpenBSD relayd

48

OpenBSD Kernel

Userspace


PF-Firewall

probes

Layer 7 proxy



OpenBSD relayd

49

OpenBSD Kernel

Userspace


PF-Firewall

probes

OK

Layer 7 proxy



OpenBSD relayd

50

OpenBSD Kernel

Userspace


PF-Firewall

probes

OK

DNS-Query

DNS-Query

Layer 7 proxy



OpenBSD relayd

51

OpenBSD Kernel

Userspace


PF-Firewall

probes

OK

DNS-Query

DNS-Query

Layer 7 proxy

DNS-Query



OpenBSD relayd

52

OpenBSD Kernel

Userspace


relayd

PF-Firewall

probes

Layer 7 proxy



OpenBSD relayd

53

OpenBSD Kernel

Userspace

relayd

PF-Firewall

probes

Not-OK


Layer 7 proxy



OpenBSD relayd

54

OpenBSD Kernel

Userspace

relayd

PF-Firewall

probes

Not-OK

DNS-Query


Layer 7 proxy

DNS-Query



OpenBSD relayd

55

OpenBSD Kernel

Userspace

relayd

PF-Firewall

probes

Not-OK

DNS-Query

DNS-Query


Layer 7 proxy

DNS-Query



relayd demo





PowerDNS dnsdist





dnsdist

“dnsdist” is an DNS aware application level gateway

• part of PowerDNS, but DNS server agnostic (can be used with any DNS resolver or authoritative server)

• supports various load-balancing schemes (least outstanding, firstAvailable, weighted hash, weighted random, round-robin ...)

• can do more than load balancing (filter, block, rewrite DNS traffic ...)

58



dnsdist

“dnsdist” is an DNS aware application level gateway

• Lua-configuration and Lua-scriptable

• available for Linux (Debian, Raspbian, Suse, Ubuntu, CentOS), FreeBSD

• should work on other Unix-ish systems

• Free Software (GPLv2 License)

59

http://dnsdist.org


http://dnsdist.org

http://dnsdist.org


DNS-Resolver with dnsdist

60

Internet

172.22.1.210 172.22.1.217

Heartbeat172.22.1.200(dnsdist)

172.22.1.200(dnsdist)



DNS-Resolver with dnsdist

60

Internet

/etc/resolv.confnameserver 172.22.1.200

172.22.1.210 172.22.1.217

Heartbeat172.22.1.200(dnsdist)

172.22.1.200(dnsdist)



starting dnsdist

simple dnsdist startup without configuration file

# dnsdist -l 172.22.1.200 172.22.1.210 172.22.1.217

61

local IP to listen for

DNS queries

DNS server to forward

queries



dnsdist demo





dnsdist statistics demo





comparing relayd and dnsdist





relayd vs. dnsdist

•relayd -- only available on OpenBSD (FreeBSD)

•dnsdist -- available on many Linux/Unix systems





relayd vs. dnsdist

•relayd -- fast layer 3 forwarding in kernel space and userspace proxying

•dnsdist -- only userspace proxying (but still pretty fast)





relayd vs. dnsdist

•relayd -- simple health monitoring and reporting

•dnsdist -- online DNS statistics and Web-UI statistics





relayd vs. dnsdist

•relayd -- filtering with “pf” firewall

•dnsdist -- DNS aware filtering with Lua-Scripting option





relayd vs. dnsdist

•relayd -- BSD license

•dnsdist -- GPLv3 License





Men & Mice Training

• February 13 – 17 -- Redwood City, California, US Introduction to DNS & BIND Hands-On class and Introduction & Advanced DNS and BIND Topics Hands-on

•March 6 – 10, -- Amsterdam (NL) or Osnabrueck (DE) Introduction to DNS & BIND Hands-On class and Introduction & Advanced DNS and BIND Topics Hands-on

https://www.menandmice.com/support-training/training/







Webinar schedule 2017

This is our schedule for the webinars in the beginning of 2017

• 2nd Feb 2017 BIND 9 logging best practices

• 23rd March 2017 DNSSEC zone signing tutorial

• 13th April 2017 SMTP STS (Strict Transport Security) vs. SMTP with DANE

71



Webinar schedule 2017

Additional webinar topics coming in 2017

• DNSSEC key management with BIND 9 "keymgr"

• BIND 9 (and Men & Mice) on Docker (Linux)

• Men & Mice Suite on Docker with Windows 2016 Server

• How to manage DMARC-, SPF-, DKIM-, multi-part TXT-, CAA-, DANE-records in DNS zones

• DNS over TCP: new developments from the IETF

• DNS Server with SQL-Databases: PowerDNS and BIND 9

72



Thank you!

Questions? Comments?




dns high-availability tools - open-source load balancing solutions

Technology