SRE here, Strong disagree with do not fail readiness and liveness checks on upstream dependencies failing. There are several reason to do so and unless you have extreme start up time, what's the problem with restarting?
Maybe DNS has changed on you but you are stuck with bad local cache because you poorly respect TTLs (Looking at you Java), reseting the process will clear that cache away.
Maybe TCP connections are in stuck weird state, resetting the process generally helps with that.
Maybe someone gave you bad ENV VARs and you cannot connect to database, by refusing to progress the rollout, no outage generated.
So yea, if you are not ready to do work including critical upstream dependencies, don't lie to system and say you are.
Flamkuchlo 26 minutes ago [-]
I think the best approach is to have an endpoint which returns the current state without probing the database when you check the endpoint.
Like you have a threat which does all your checks for connectivity write it into memory and the api endpoint checks the last values written.
Then you can respond to different upstream issues differently: Your DB says 'permission denied' due to broken auth data? Your Pod is not ready.
Your backend understand what a db maintenance mode is and should still return "Maintenance", your pod stays ready.
It can be more nuanced.
But the problem with restarting is, as he stated in the blog: What if now EVERY pod restarts in parallel.
solatic 12 hours ago [-]
Unlike sibling commenters who just read about "thundering herd" problems on the Internet, as someone who spent significant time in SRE roles, I agree with you as a matter of what the default approach should be. If you have a small cluster and no more than a handful of services... what thundering herd problem is there supposed to be, exactly, with so few "cattle" in the "herd"? Meanwhile, there are serious benefits, as you describe.
Large clusters with dozens of services and traces that go several services deep, with each service owned by a different team, are a whole 'nother ballgame, especially when overall production uptime is owned by an SRE team and not by the developer teams who wrote each of those services. And even in this scenario, you're not necessarily wrong; the risk attached to the cascading failure is domain-specific and may be acceptable.
arccy 14 hours ago [-]
Thundering herd / cascading outages. You take out a large enough portion of your fleet, and the remaining load overloads your remaining nodes one by one as they restart, so you can never have enough healthy nodes.
erulabs 14 hours ago [-]
SRE team debates correctness versus availability for the 540th time this year
You're both correct, of course!
atmosx 14 hours ago [-]
What this guy said :point_up:
My personal take-away is this: whatever you choose, make sure it's consistent across services (not serviceA behaves like X and serviceB like Y) and make sure eng teams know _how_ these are configured and what can go wrong. They'll figure out the rest.
solatic 12 hours ago [-]
You'd be surprised how many engineering leaders don't understand the CAP theorem and will fail engineers on interviews for picking the one they don't agree with instead of communicating their expectations clearly (dodged a bullet on that one ...)
jaggederest 14 hours ago [-]
That's a problem for circuitbreakers on these kinds of actions, not lying on health checks.
Something like healthcheck fails -> restart -> healthcheck fails -> restart -> healthcheck fails -> circuit breaker trip, alarm raised, give up until manual intervention or X minutes have passed
deathanatos 13 hours ago [-]
That circuitbreaker exists, by default. It is "CrashloopBackoff", here, and TFA covers it. (& it's an "until X minutes have passed" kind, by default.)
dilyevsky 13 hours ago [-]
backoff is only applied to individual pods/containers not across pods. the point is at scale it's easy to get into a situation where it's not possible to recover without (usually manual) full service drain
jaggederest 4 hours ago [-]
Yeah that's why I have a manual intervention breaker that goes across all the pods/nodes etc. CrashLoopBackoff is great for selfhealing but when things go really pear shaped you want something that catches the global state. Saw it activate during an AWS outage one time where new nodes were unhealthy on start, for example.
dilyevsky 14 hours ago [-]
1. was already mentioned in sibling - cascade failures
2. you'll have massive number of restarts for various flake reasons and missing things that got papered over with restarts until you hit 1 and everything is broken. another popular version of this is "just restart when memory leaks too much"
connicpu 13 hours ago [-]
The better solution is to not have too many critical upstream services :)
figmert 13 hours ago [-]
My favourite: misconfigured Linkerd setup that causes CA certs to rotate every month :) Definitely worth restarting on that
cmckn 14 hours ago [-]
> what's the problem with restarting?
Exponential backoff can delay recovery up to kubelet’s maxContainerRestartPeriod (default 5m).
javier2 14 hours ago [-]
cascading failures on upstream services. then you get 20 different services failing instead of the single one.
peterabbitcook 14 hours ago [-]
What are your feelings about using initContainers and wait-for-it to skirt the thundering-herd problem?
sidcool 16 hours ago [-]
This does not state anything new, but explains it so much well than the kubernetes documentation.
dev_cprice 15 hours ago [-]
Sam has a real way with words when it comes to educational content.
Though he did find a legit Kubernetes bug while writing the post, so technically there was at least one new thing :)
srichard16 15 hours ago [-]
Sometimes the k8s docs remind me of google's documentation
leetrout 15 hours ago [-]
many times i have said a company could be built to just make better google documentation.
stroebs 14 hours ago [-]
I need to know how to animate things like this for internal documentation.
Maybe DNS has changed on you but you are stuck with bad local cache because you poorly respect TTLs (Looking at you Java), reseting the process will clear that cache away.
Maybe TCP connections are in stuck weird state, resetting the process generally helps with that.
Maybe someone gave you bad ENV VARs and you cannot connect to database, by refusing to progress the rollout, no outage generated.
So yea, if you are not ready to do work including critical upstream dependencies, don't lie to system and say you are.
Like you have a threat which does all your checks for connectivity write it into memory and the api endpoint checks the last values written.
Then you can respond to different upstream issues differently: Your DB says 'permission denied' due to broken auth data? Your Pod is not ready.
Your backend understand what a db maintenance mode is and should still return "Maintenance", your pod stays ready.
It can be more nuanced.
But the problem with restarting is, as he stated in the blog: What if now EVERY pod restarts in parallel.
Large clusters with dozens of services and traces that go several services deep, with each service owned by a different team, are a whole 'nother ballgame, especially when overall production uptime is owned by an SRE team and not by the developer teams who wrote each of those services. And even in this scenario, you're not necessarily wrong; the risk attached to the cascading failure is domain-specific and may be acceptable.
You're both correct, of course!
My personal take-away is this: whatever you choose, make sure it's consistent across services (not serviceA behaves like X and serviceB like Y) and make sure eng teams know _how_ these are configured and what can go wrong. They'll figure out the rest.
Something like healthcheck fails -> restart -> healthcheck fails -> restart -> healthcheck fails -> circuit breaker trip, alarm raised, give up until manual intervention or X minutes have passed
2. you'll have massive number of restarts for various flake reasons and missing things that got papered over with restarts until you hit 1 and everything is broken. another popular version of this is "just restart when memory leaks too much"
Exponential backoff can delay recovery up to kubelet’s maxContainerRestartPeriod (default 5m).
Though he did find a legit Kubernetes bug while writing the post, so technically there was at least one new thing :)
3 comments https://news.ycombinator.com/item?id=48734656