ZOOKEEPER-4946: Fix Login renewal thread not exiting on shutdown

[IvanKhanas]

Problem

The reLogin() retry loop in Login never exited once the retries ran out: the else branch only logged, and retry >= 0 stayed true. hasSufficientTimeElapsed() skips an attempt only if the previous one started less than minReLoginTimeMs ago (60s by default). When the KDC drops packets instead of refusing connections, the JDK waits 30s per try and retries 3 times, so every failed login takes longer than that. The thread then calls reLogin() forever and never reaches a sleep or an interrupt check. Login.shutdown() interrupted it and joined without a timeout.

SendThread calls Login.shutdown() on exit and ClientCnxn.close() joins SendThread, so ZooKeeper.close() hung. NIOServerCnxnFactory and NettyServerCnxnFactory also call login.shutdown().

The interrupt could also be lost in Shell.runCommand(), which caught InterruptedException without restoring the interrupt status. This only affects the kinit path used with a ticket cache.

Fix

• Once retries are exhausted, the retry loop breaks out to the renewal loop. A failed reLogin() has already logged out, so no TGT is left and the loop takes the existing "No TGT found" branch: it sleeps minReLoginTimeMs and tries again. Renewal recovers when the KDC comes back.
• The renewal loop runs while (!Thread.currentThread().isInterrupted()).
• Shell restores the interrupt status in both catch (InterruptedException) blocks.
• Login.shutdown() bounds the join with zookeeper.kerberos.shutdownTimeoutMs (5s by default) and logs a warning if the thread is still alive. A login.login() call in progress cannot be interrupted; the thread is a daemon.

Tests

• KerberosTicketRenewalTest.shouldLeaveRetryLoopWhenReLoginKeepsFailing lets every reLogin() attempt run, as with a slow KDC, and checks that the thread leaves the retry loop.
• KerberosTicketRenewalTest.shouldNotBlockForeverWhenRenewalThreadDoesNotExit blocks the renewal thread in a call that ignores interrupts and checks that shutdown() still returns.
• ShellTest (new) interrupts a thread inside Shell.execCommand and checks that the interrupt status survives.

Each test fails without its production change.

[IvanKhanas]

Both precommit runs landed on the same agent and died the same way. Build #1 lost the agent almost immediately. Build #2 got through the whole suite first, Tests run: 3271, Failures: 0, Errors: 0, Skipped: 4, and then hit hudson.remoting.RequestAbortedException from Channel.terminate on hadoop16. Two other PRs built today on hadoop17 and hadoop0 and are green, so this looks like the agent rather than the change.

I pushed a rebase onto current master to get another run. If it lands on hadoop16 again it will probably fail the same way.

[kezhuw]

For example, when there is a failure in kerberos, the reLogin method call throws an exception. After retry has been set to 0, the reLogin method will be repeatedly called until successful.

Sounds like it is a dead loop bug but not interruption check things. No matter how frequently the check, we cann't interrupt Shell.execCommand and others. retry >= 0 seems to be always true.

[IvanKhanas]

For example, when there is a failure in kerberos, the reLogin method call throws an exception. After retry has been set to 0, the reLogin method will be repeatedly called until successful.

Sounds like it is a dead loop bug but not interruption check things. No matter how frequently the check, we cann't interrupt Shell.execCommand and others. retry >= 0 seems to be always true.

Thanks for reading this so carefully. I was chasing how the interrupt got lost
and never noticed that retry >= 0 stays true after the last attempt.

Fixed in 3168e2a. Once the retries run out, the loop breaks back into the
renewal loop, which sleeps before the next attempt and exits on interrupt. I also
added a test that lets every reLogin() attempt run, the way a slow KDC would,
and it fails without the fix.

[anmolnar]

Could you please document the config setting?

[IvanKhanas]

Could you please document the config setting?

Done in cdadcfa. The entry is in the admin guide next to the other kerberos.* settings, plus a shorter one in the client properties list, since Login also runs on the client.

[kezhuw]

LGTM