[INTTEST-118] max device count lost 50% Created: 13/Jul/20  Updated: 16/Jul/20  Resolved: 16/Jul/20

Status: Resolved
Project: integration-test
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Medium
Reporter: Jamo Luhrsen Assignee: Jamo Luhrsen
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified


 Description   

It seems that some change came in mid-June that has reduced the
max device count we can get with this job by aprox 50%

The claim is strictly coming from looking at the graphs we get in jenkins:

magnesium

aluminium

 

The problem could be anywhere, including some change in the basic infra
environment. One clue is that both jobs seemed to have dropped in the same
time frame.

 Comments   
Comment by Jamo Luhrsen [ 14/Jul/20 ]

jluhrsen will run these jobs with an older distribution to see if we can eliminate the infra being the issue.

Comment by Jamo Luhrsen [ 16/Jul/20 ]

re-running the jobs in the sandbox with a distro built back when scale results were
better still resulted in low numbers, which ruled out some bug on the controller as
a whole.

I did find something that looks like a race condition in the test logic. The fix
in in test code and the sandbox results are now coming back up in the 850 range.

Also, I think maybe this scale drop maybe started happening right when our infra
and test code was moving from python2 to python3. Maybe some performance gain in
our test logic caused the issue to be more prevalent.

closing this now.

Generated at Wed Feb 07 20:04:37 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.