https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/INTTEST-100 integration-test <p>Something is newly broken with a test tool for controller jobs, like<br/> <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/controller-csit-1node-rest-cars-perf-all-magnesium/274/robot-plugin/log.html.gz#s1-s1-t1-k2-k2" class="external-link" target="_blank" rel="nofollow noopener">this one </a></p> <p>the command line being used is this:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> python cluster_rest_script.py - -host 10.30.170.86 --port 8181 add --itemtype car --itemcount 10000 --ipr 10000 </pre> </div></div> <p>but it's dying with something like this:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> Traceback (most recent call last): File <span class="code-quote">"cluster_rest_script.py"</span>, line 7, in &lt;module&gt; </pre> </div></div> <p>the controller jobs using this tool are failing because of it (obviously), but those<br/> jobs for Aluminium are needed to see the health of MRI bump.</p> INTTEST-100

cluster_rest_script.py is aborting

Bug Highest Resolved Cannot Reproduce Robert Varga Jamo Luhrsen Tue, 9 Jun 2020 00:37:20 +0000 Fri, 4 Sep 2020 16:52:16 +0000 Fri, 4 Sep 2020 16:52:16 +0000 0 5 <p>This is a failure across all release trains. Mg detected it between</p> <p><a href="https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-1node-rest-cars-perf-all-magnesium/271/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-1node-rest-cars-perf-all-magnesium/271/</a>  (good)</p> <p><a href="https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-1node-rest-cars-perf-all-magnesium/272/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/controller/job/controller-csit-1node-rest-cars-perf-all-magnesium/272/</a>  (bad)</p> <p> </p> <p>The most obvious difference seems to be:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>@@ -281,2 +282,2 @@ - 'vm_1_image': 'ZZCI - Ubuntu 16.04 - mininet-ovs-28 - ' - '20190415-091034.881'}, + 'vm_1_image': 'ZZCI - Ubuntu 16.04 - mininet-ovs-28 - x86_64 - ' + '20200601-220226.013'}, </pre> </div></div> <p>I am not sure whether this is actually the image ODL runs on</p> <p>This seems to be pointing at <a href="https://git.opendaylight.org/gerrit/c/releng/builder/+/90196/2/jjb/integration/integration-templates.yaml" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/c/releng/builder/+/90196/2/jjb/integration/integration-templates.yaml</a></p> <p> </p> <p>Ran a job that directly launched "ZZCI - Ubuntu 16.04 - mininet-ovs-28 - x86_64 - 20200601-220226.013" and it appears to work (traceback is due to connection to an IP address that doesn't exist so ignore that).</p> <p>I'm not seeing failure to import the requests library here. So what this tells me is the base image might be okay. Perhaps something that's happening in CSIT to prepare the image to run the job results in broken dependencies or similar?</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>python test/tools/odl-mdsal-clustering-tests/scripts/cluster_rest_script.py --host 10.30.170.86 --port 8181 add --itemtype car --itemcount 10000 --ipr 10000 /home/jenkins/.local/lib/python2.7/site-packages/requests/__init__.py:83: RequestsDependencyWarning: Old version of cryptography ([1, 2, 3]) may cause slowdown. warnings.warn(warning, RequestsDependencyWarning) 2020-06-12 19:13:55,948 INFO: Add 10000 car(s) to 10.30.170.86:8181 (10000 per request) Exception in thread Thread-1: Traceback (most recent call last): File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner self.run() File "/usr/lib/python2.7/threading.py", line 754, in run self.__target(*self.__args, **self.__kwargs) File "test/tools/odl-mdsal-clustering-tests/scripts/cluster_rest_script.py", line 221, in _request_sender rsp = ses.send(prep, timeout=req_timeout) File "/home/jenkins/.local/lib/python2.7/site-packages/requests/sessions.py", line 643, in send r = adapter.send(request, **kwargs) File "/home/jenkins/.local/lib/python2.7/site-packages/requests/adapters.py", line 516, in send raise ConnectionError(e, request=request) ConnectionError: HTTPConnectionPool(host='10.30.170.86', port=8181): Max retries exceeded with url: /restconf/config/car:cars (Caused by NewConnectionError('&lt;urllib3.connection.HTTPConnection object at 0x7faf0a4e4750&gt;: Failed to establish a new connection: [Errno 111] Connection refused',)) </pre> </div></div> <p>I'm not sure how to read the CSIT robot logs but if someone can get the complete traceback from the description that might be useful. What's pasted in the description isn't enough for me to see the full picture of what is failing.</p> <p>That's not the same trouble as this:<br/> <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/controller-csit-1node-rest-cars-perf-all-magnesium/274/robot-plugin/log.html.gz#s1-s1-t1-k4" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/controller-csit-1node-rest-cars-perf-all-magnesium/274/robot-plugin/log.html.gz#s1-s1-t1-k4</a></p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java">Traceback (most recent call last): File <span class="code-quote">"cluster_rest_script.py"</span>, line 7, in &lt;module&gt; </pre> </div></div> <p>where is the link to the job where you got the above traceback. That probably means the<br/> controller is not working properly, but that would be a different thing to debug.</p> <p>I tried adding '-L TRACE' to the command but unfortunately didn't get much more details:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>18:06:38.893 TRACE Arguments: [ 'Traceback (most recent call last):\r\n File "cluster_rest_script.py", line 7, in &lt;module&gt;' ] 18:06:38.893 INFO Traceback (most recent call last): File "cluster_rest_script.py", line 7, in &lt;module&gt; 18:06:38.893 TRACE Return: None </pre> </div></div> <p>Tried to get debug logs by passing --debugfile to robot and the resulting <span class="nobr"><a href="https://jira.opendaylight.org/secure/attachment/15661/15661_robot-debug.log.gz" title="robot-debug.log.gz attached to INTTEST-100">robot-debug.log.gz^{<img class="rendericon" src="https://jira.opendaylight.org/images/icons/link_attachment_7.gif" height="7" width="7" align="absmiddle" alt="" border="0"/>}</a></span> does not appear to print out anymore useful information.</p> <p>This is a tough one since the only thing we can infer from failing on "line 7" is that something is causing requests library to fail to import but why exactly is hard to say without more information.</p> <p>do you have the .html robot file and/or the output.xml file?</p> <p>Same thing is happening to <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/bgpcep-csit-1node-throughpcep-all-aluminium/99/robot-plugin/log.html.gz" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/bgpcep-csit-1node-throughpcep-all-aluminium/99/robot-plugin/log.html.gz</a></p> <p> </p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> can you confirm to me which system the python scripts run on? is it the robot VM or the mininet VM?</p> <p>If I boot a mininet vm directly and clone integration/test repo and attempt to run the mdsal python script I get:</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>Traceback (most recent call last): File "tools/odl-mdsal-clustering-tests/scripts/cluster_rest_script.py", line 7, in &lt;module&gt; import requests ImportError: No module named requests </pre> </div></div> <p><a href="https://jenkins.opendaylight.org/sandbox/view/All/job/zxiiro-test/7/console" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/view/All/job/zxiiro-test/7/console</a></p> <p>Which is pretty similar to the CSIT job run.</p> <p>Thinking about the mininet VMs changing recently, could this be a case of the old VM having robot tools like requests pre-installed vs the new mininet VM just plain missing it? or is CSIT supposed to install requests at runtime?</p> <p>I'm not sure if the old mininet is still available but if it is maybe we can get the LF to boot one and do a pip freeze on it.</p> <p>This is very simple problem: since the issue shows when running python scripts in the TOOLS_SYSTEM VM,<br/> I just compared python modules in old image vs new image:</p> <ul class="alternate" type="square"> <li>old: ZZCI - Ubuntu 16.04 - mininet-ovs-28 - 20190415-091034.881</li> <li>new: ZZCI - Ubuntu 16.04 - mininet-ovs-28 - x86_64 - 20200601-220226.013</li> </ul> <p>The below packages are missing in the new image but present in the old one:</p> <p>chardet (2.3.0)<br/> git-review (1.25.0)<br/> ndg-httpsclient (0.4.0)<br/> requests (2.9.1)<br/> urllib3 (1.13.1)</p> <p>So until we figure how to fix the image or the robot test to install the missing modules, I think it is better to revert the originating patch:</p> <p><a href="https://git.opendaylight.org/gerrit/c/releng/builder/+/90270" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/c/releng/builder/+/90270</a></p> <p>The issue shows in the mininet/TOOLS_SYSTEM VM. It looks like new image is missing some modules including requests (see my comment below).</p> <p><a href="https://git.opendaylight.org/gerrit/c/releng/builder/+/90446" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/c/releng/builder/+/90446</a> should result it these being installed:</p> <p>git-review-1.25.0-2</p> <p>python-chardet-2.3.0-2</p> <p>python-urllib3-1.13.1-2ubuntu0.16.04.3</p> <p>python-requests-2.9.1-3ubuntu0.1</p> <p>python-ndg-httpsclient-0.4.0-3</p> <div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent"> <pre>[15/06/2020 16:56:50] &lt;rovarga&gt; zxiiro: did we switch provisioning stuff when we went to vexxhost? [15/06/2020 16:57:16] &lt;rovarga&gt; zxiiro: builder/packer/provision/baseline.sh has: [15/06/2020 16:57:18] &lt;rovarga&gt; ensure_ubuntu_install git-review [15/06/2020 16:57:28] &lt;zxiiro&gt; rovarga: bash scripts to ansible. I can't remember if we were using packer in both places (I assume so) [15/06/2020 16:57:45] &lt;rovarga&gt; ooooh [15/06/2020 16:57:52] &lt;rovarga&gt; okay, so that's how we lost these packages [15/06/2020 16:58:33] &lt;rovarga&gt; note that baseline.sh is not referenced anywhere except gbp.sh [15/06/2020 16:58:44] &lt;zxiiro&gt; Vexxhost change wasn't 2019 though. [15/06/2020 16:59:14] &lt;zxiiro&gt; So we would have discovered issues earlier than that no? [15/06/2020 17:00:05] &lt;rovarga&gt; zxiiro: are you sure? [15/06/2020 17:00:11] &lt;zxiiro&gt; Although I did not spend much time managing mininet VMs possible those VMs still used the bash provisioning scripts maybe? [15/06/2020 17:00:36] &lt;zxiiro&gt; rovarga: yes we switched to Vexxhost earlier than 2019. I can't remember the exact year. maybe tykeal remembers. [15/06/2020 17:00:43] &lt;rovarga&gt; there definitely was some work being done in March-May 2019: https://git.opendaylight.org/gerrit/c/releng/builder/+/80752 [15/06/2020 17:01:50] &lt;zxiiro&gt; rovarga: Right, it's possible that the mininet scripts were not updated when we migrated and took longer to get done. We focused on java-builder with the initial migration. [15/06/2020 17:02:08] &lt;zxiiro&gt; mininet likely lagged behind with the ansible change [15/06/2020 17:03:30] &lt;zxiiro&gt; ok so according to your comment on INTTEST the old image was dated 20190415-091034.881 that does seem plausible. [15/06/2020 17:06:11] &lt;rovarga&gt; exactly [15/06/2020 17:06:26] &lt;rovarga&gt; so this was broken by a change which was merged afterwards [15/06/2020 17:06:28] &lt;rovarga&gt; *or* [15/06/2020 17:06:32] &lt;rovarga&gt; NOT merged [15/06/2020 17:06:49] &lt;rovarga&gt; there is no equivalent of https://git.opendaylight.org/gerrit/c/releng/builder/+/80752 [15/06/2020 17:07:02] &lt;rovarga&gt; for mininet-ovs-2.8 [15/06/2020 17:07:23] &lt;rovarga&gt; nite@nitebug : ~/odl/builder/packer on master $ diff -u templates/mininet-ovs-2.6.json templates/mininet-ovs-2.8.json | diffstat [15/06/2020 17:07:23] &lt;rovarga&gt; mininet-ovs-2.8.json | 8 ++++---- [15/06/2020 17:07:23] &lt;rovarga&gt; 1 file changed, 4 insertions(+), 4 deletions(-) [15/06/2020 17:07:23] &lt;rovarga&gt; nite@nitebug : ~/odl/builder/packer on master $ diff -u provision/mininet-ovs-2.6.yaml provision/mininet-ovs-2.8.yaml | diffstat [15/06/2020 17:07:23] &lt;rovarga&gt; mininet-ovs-2.8.yaml | 108 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++--------------------------- [15/06/2020 17:07:23] &lt;rovarga&gt; 1 file changed, 81 insertions(+), 27 deletions(-) [15/06/2020 17:08:12] &lt;rovarga&gt; i.e. while the templates are mostly the same, the playbooks are vastly different [15/06/2020 17:18:48] &lt;rovarga&gt; zxiiro: and the delta of installing git-review is what broke us [15/06/2020 17:18:52] &lt;rovarga&gt; https://packages.ubuntu.com/xenial/git-review [15/06/2020 17:19:05] &lt;rovarga&gt; i.e. git-review pulls in python-requests (for python2) [15/06/2020 17:19:19] &lt;zxiiro&gt; hmm interesting. [15/06/2020 17:19:21] &lt;rovarga&gt; LuisGomez: ^^^^ [15/06/2020 17:19:46] &lt;rovarga&gt; I suspect it does not matter for other mininets, as we are not using them in this capacity [15/06/2020 17:19:57] &lt;zxiiro&gt; I don't think we should install git-review on mininet as it doesn't need it and shouldn't be doing anything with Gerrit so we should just pull in requests directly. [15/06/2020 17:20:07] &lt;rovarga&gt; yup [15/06/2020 17:20:21] &lt;rovarga&gt; we do not have the smoking gun of a commit [15/06/2020 17:20:41] &lt;rovarga&gt; but I think we have enough circumstantial evidence to explain what happened [15/06/2020 17:20:44] &lt;rovarga&gt; and how to fix it [15/06/2020 17:20:53] &lt;zxiiro&gt; agreed.</pre> </div></div> <p>The issue does not affect the Aluminium release.</p> <p>this is no longer happening in Mg or Al. I'm guessing it was fixed unknowingly. closing this ticket as unreproducible</p> Development Rank 0|i03sxz: