So I took a look at

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netconf-csit-1node-scale-only-magnesium/326/ (failed) versus https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/netconf-csit-1node-scale-only-magnesium/299/ (successful)

and one thing that is immediately noticeable is that the logs from the testtool are different: testtool--netconf-scale-txt-GetSingle in the failed case is a lot smaller and lists inability to bind port 17830 and up – which seems to indicate a failed cleanup before.

Also the Getmulti log is roughly 39% of the successful run – where success covers ~38 minutes and failure covers ~19 minutes. I strongly suspect there is something fishy in the test suite.

The other thing is we have a bunch of InterruptedByTimeoutExceptions in SessionTimeoutListener. I think these require client/server heartbeats to be enabled ... which we tracked somewhere else.

rovarga noticed there are a lot of device disconnects in the karaf.log, even in the passing cases. For reference, comparing two logs 1   2 where
the first is something recent with this regression and the 2nd is from early May (the SR1 timeframe). The first has 193 instances of "Session went down"
in the karaf.log whereas the 2nd has none.

In mid may, we got the work to shade sshd in order to disassociate with the karaf provided sshd (to fix a bug with karaf shell ssh, iirc). That timeframe
doesn't match up with the description where the failures became prevalent around July 11th. However, there was a mina-sshd upgrade on July 7th
that might be a clue.

I'm experimenting in that area with something like this to see if it gets rid of the regression.

Alright, so downgrading mina-sshd seems to have stabilized the test suite, but we still seeing timeouts. I'll try longevity locally and hopefully will be able to make some sense of this.

Okay, the two additional patches seem to stabilize sessions on the use cases tested by the CSIT suite. Let's bring them in, see if they are restoring sanity.

I ran two jobs in the sandbox since yesterday on aluminium. one is running with the distribution created from
the distro check job in the master (silicon) patch that reverted the sshd upgrade
while the other is running latest aluminium distro which doesn't have that revert. The first job failed twice in 37 jobs and the
second failed 7 in 37. I was hoping it would be more concrete with 100% passing in the first job, but it's still a big difference.

just fyi.

can you point me to the two patches?

https://git.opendaylight.org/gerrit/q/I98ef51b4346c2aee0c9c87d58035bc08ef2c77d1
https://git.opendaylight.org/gerrit/q/I2c6ed07b5d3b29f6524b0ed99067895c4dfdc020

both will be part of next autoreleases.

To throw some variance across branches and get some results overnight, https://git.opendaylight.org/gerrit/c/netconf/+/92299 will re-instate sshd-2.5.1 on Silicon and we can compare the results.

https://jenkins.opendaylight.org/releng/view/netconf/job/netconf-csit-1node-scale-only-silicon/23/ succeeded with sshd-2.5.1, the only timeouts are on Karaf SSH. I have gave it another try to see where it is.

https://jenkins.opendaylight.org/releng/view/netconf/job/netconf-csit-1node-scale-only-aluminium/181/ succeeded with sshd-2.3.0, same as above.

https://jenkins.opendaylight.org/releng/view/netconf/job/netconf-csit-1node-scale-only-magnesium/355/ seems to contain some failure – but we'll need to have it finish before we know more.

I have triggered a Sodium AR build to get a read on that branch, too.

Everything except Mg is stabilized. I will open a separate issue to track the Mg failure.