In reply to @a:oper.io
Question: I've got a statefulset running three pods with an ingress controller (AWS ALB) in front of the set. Each pod tends to have long running connections (mostly large uploads and downloads which take seconds to minutes). When I delete a pod and the statefulset brings it back up, it seems that there is no connection draining happening when there is an explicit delete. Specifically, it seems that sends the pod shutdown signal prior to draining all connections first, so the ingress controller responds with 502 codes when traffic is routed to the deleting pod because the service is shut down.
So I suppose the question is: when deleting a pod, is traffic drained by the delete action (it seems like this is not the case), or by the liveness probe failing at the threshold count?

So I suppose the question is: when deleting a pod, is traffic drained by the delete action (it seems like this is not the case), or by the liveness probe failing at the threshold count?

Both.
When pod is deleted the container should handle SIGTERM and drain connections withing terminationGracePeriod (see sequence of events).
When pod doesn't pass liveness checks it won't be included in endpoint list and won't receive new connections

Vadim Rutkovsky: Ah, yes, thanks for sending that link over! It looks like I can more intelligently handle the situation with a carefully crafted preStop hook script that monitors active socket counts, waiting for the number to reduce to 0. So in theory:

1. Execute pod delete
2. Pod goes into a "Terminating" state, which removes the pod from all services, not allowing new connections, but allowing existing.
3. preStop hook executes (my script - presumably waiting for socket counts to go down). This blocks until the script exits (within the grace period)
4. SIGTERM sent, app shuts down (normally not graceful, just killing all remaining connections, which should now be 0)
5. (rest of the process I don't care too much about)

Vadim Rutkovsky: Ah, yes, thanks for sending that link over! It looks like I can more intelligently handle the situation with a carefully crafted preStop hook script that monitors active socket counts, waiting for the number to reduce to 0. So in theory:

1. Execute pod delete
2. Pod goes into a "Terminating" state, which removes the pod from all services, not allowing new connections, but allowing existing.
3. preStop hook executes (my script - presumably waiting for socket counts to go down). This blocks until the script exits (within the grace period)
4. SIGTERM sent, app shuts down (normally not graceful, just killing all remaining connections, which should now be 0)
5. (rest of the process I don't care too much about)