In the scenario of SSL connection, sometimes the listener get notified before SSL handshake is done. The root cause is we don't handle the handshake state "NOT_HANDSHAKING" properly. "NOT_HANDSHAKING" could mean handshake process is finished or doesn't begin yet. We should handle latter case: handshake process doesn't begin yet.