opportunities to improve system reliability and resilience by donald belcham
DESCRIPTION
Opportunities to Improve System Reliability and Resilience Donald Belcham .NET Conf UY 2014 http://netconf.uyTRANSCRIPT
SystemReliability and Resilience
and stuff
Some things need to be cleared up first
http://en.wikipedia.org/wiki/Vedette_(cabaret)
tuple
//Initialize customer and invoiceInitialize(customer, invoice);
public void Initialize(Customer customer, Invoice
invoice){
customer.Name = “asdf”;invoice.Date = DateTime.Now;
}
Initialize(customer, invoice);//did something happen to customer// and/or invoice?
customer.Name =InitNameFrom(customer,
invoice);invoice.Date =
InitDateFrom(customer, invoice);
customer.Name =GetNameFrom(customer,
invoice);invoice.Date =
GetDateFrom(customer, invoice);
var results = Initialize(customer, invoice);
customer.Name = results.Item1;invoice.Date = results.Item2;
public tuple<string, DateTime>Initialize(customer,
invoice){
return new Tuple<string, DateTime>(“asdf”, DateTime.Now);
}
public static bool TryParse(string s, out DateTime result)
or
public static tuple<bool, DateTime?> TryParse(string s)
tuple• Avoid side effects
• Avoid out parameters
• multiple values without a specific type
null object
private ILogger _logger;public MyClass(ILogger logger) {
_logger = logger;}
…
if (_logger != null) {_logger.Debug(
“it worked on my machine!”);}
null checks for everyone!
forget one and…
public class NullLogger : ILogger {public void Debug(string text) {
//do sweet nothing}
}
private ILogger _logger = new NullLogger();
public MyClass(ILogger logger) {_logger = logger;
}
…
_logger.Debug(“it worked on my machine!”);
null object• Can eliminate null checks
• Simple to implement
Circuit Breaker
Retry
Your
Applicati
on O
ut o
f Pro
cess
Dependency
N times
Out o
f Pro
cess
Dependency
N times
*
Y clients
= Denial of
Service Attack
Limit the # of retries
N * Ybecomes
5 * Y
Y isstill a
problem
Circuit Breaker
State Machine
On :: Off
On Offwhen not healthy
Off Onmanually
Get to softwarebefore we ask you
to dance
Healthyor
Unhealthy
Out o
f Pro
cess
Dependency
State is independent of
requestor
Out o
f Pro
cess
Dependency
Your
Applicati
on
Has many independent external dependencies
Your
Applicati
on
Can throttle itself
Your
Applicati
on
Has a wait threshold
Your ApplicationExternal
DependencyCircuit Breaker
Threshold = 2Pause = 10msTimeout = 30sState = ClosedRequest
Request
Failure (i.e. HTTP 500)Failure Count = 1Pause 10ms
Request
Failure (i.e. HTTP 500)Failure Count = 2State = Open
OperationFailedException
Threshold = 2Pause = 10msTimeout = 30sState = OpenRequest
30s has not passed
CircuitBreakerOpenException
Request
30s has not passed
CircuitBreakerOpenException
System can try
to
become
healthy
for 30s
Your ApplicationExternal
DependencyCircuit Breaker
Threshold = 2Pause = 10msTimeout = 30sState = ½ OpenRequest
Request
Failure (i.e. HTTP 500)
Failure Count = 2State = Open
OperationFailedException
30s has passed
Your ApplicationExternal
DependencyCircuit Breaker
Threshold = 2Pause = 10msTimeout = 30sState = ½ OpenRequest
Request
Failure Count = 0State = Closed
Response
30s has passed
Response
Your ApplicationExternal
DependencyCircuit Breaker
ClosedOpen
½ Open
½ Open is like a
manual reset
PauseTimeout
Pausebetween calls
in the loop
Timeoutbefore you
can call again
Exceptions
OperationFailed:
AggregateException
CircuitBreakerOpen:
ApplicationException
Don’t Loose Exception Info
Always use InnerException(s)
Threshold = 3State = ClosedRequest
Request
Failure (i.e. HTTP 500)
Request
Failure (i.e. HTTP 500)Failure Count = 2
Failure Count = 0State = Closed
Response
Response
Request?
Your ApplicationExternal
DependencyCircuit Breaker
Failure Count = 1
SegregateDependencies
circuitBreaker(“database”)
circuitBreaker(“weatherservice”)
Dependency type, endpoint svc,
endpoint
Where?
Your
Applicati
on O
ut o
f Pro
cess
Dependency
Cir
cuit
Bre
aker
Pro
xy
Watch forInception
Your
Applicati
on W
eb S
erv
ice
Cir
cuit
Bre
aker
Cir
cuit
Bre
aker
Pro
xy
Data
baseR
eposi
tory
circuit breaker• retry looping
• slow down attempts
• good neighbour
¡Muchas gracias!
