@@ -281,2 +282,2 @@
-                'vm_1_image': 'ZZCI - Ubuntu 16.04 - mininet-ovs-28 - '
-                              '20190415-091034.881'},
+                'vm_1_image': 'ZZCI - Ubuntu 16.04 - mininet-ovs-28 - x86_64 - '
+                              '20200601-220226.013'},

python test/tools/odl-mdsal-clustering-tests/scripts/cluster_rest_script.py --host 10.30.170.86 --port 8181 add --itemtype car --itemcount 10000 --ipr 10000
/home/jenkins/.local/lib/python2.7/site-packages/requests/__init__.py:83: RequestsDependencyWarning: Old version of cryptography ([1, 2, 3]) may cause slowdown.
  warnings.warn(warning, RequestsDependencyWarning)
2020-06-12 19:13:55,948 INFO: Add 10000 car(s) to 10.30.170.86:8181 (10000 per request)
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "test/tools/odl-mdsal-clustering-tests/scripts/cluster_rest_script.py", line 221, in _request_sender
    rsp = ses.send(prep, timeout=req_timeout)
  File "/home/jenkins/.local/lib/python2.7/site-packages/requests/sessions.py", line 643, in send
    r = adapter.send(request, **kwargs)
  File "/home/jenkins/.local/lib/python2.7/site-packages/requests/adapters.py", line 516, in send
    raise ConnectionError(e, request=request)
ConnectionError: HTTPConnectionPool(host='10.30.170.86', port=8181): Max retries exceeded with url: /restconf/config/car:cars (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7faf0a4e4750>: Failed to establish a new connection: [Errno 111] Connection refused',))

18:06:38.893	TRACE	Arguments: [ 'Traceback (most recent call last):\r\n  File "cluster_rest_script.py", line 7, in <module>' ]	
18:06:38.893	INFO	Traceback (most recent call last):
  File "cluster_rest_script.py", line 7, in <module>	
18:06:38.893	TRACE	Return: None	

Traceback (most recent call last):
  File "tools/odl-mdsal-clustering-tests/scripts/cluster_rest_script.py", line 7, in <module>
    import requests
ImportError: No module named requests

[15/06/2020 16:56:50] <rovarga> zxiiro: did we switch provisioning stuff when we went to vexxhost?
[15/06/2020 16:57:16] <rovarga> zxiiro: builder/packer/provision/baseline.sh has:
[15/06/2020 16:57:18] <rovarga> ensure_ubuntu_install git-review
[15/06/2020 16:57:28] <zxiiro> rovarga: bash scripts to ansible. I can't remember if we were using packer in both places (I assume so)
[15/06/2020 16:57:45] <rovarga> ooooh
[15/06/2020 16:57:52] <rovarga> okay, so that's how we lost these packages
[15/06/2020 16:58:33] <rovarga> note that baseline.sh is not referenced anywhere except gbp.sh
[15/06/2020 16:58:44] <zxiiro> Vexxhost change wasn't 2019 though.
[15/06/2020 16:59:14] <zxiiro> So we would have discovered issues earlier than that no?
[15/06/2020 17:00:05] <rovarga> zxiiro: are you sure?
[15/06/2020 17:00:11] <zxiiro> Although I did not spend much time managing mininet VMs possible those VMs still used the bash provisioning scripts maybe?
[15/06/2020 17:00:36] <zxiiro> rovarga: yes we switched to Vexxhost  earlier than 2019. I can't remember the exact year. maybe tykeal remembers.
[15/06/2020 17:00:43] <rovarga> there definitely was some work being done in March-May 2019: https://git.opendaylight.org/gerrit/c/releng/builder/+/80752
[15/06/2020 17:01:50] <zxiiro> rovarga: Right, it's possible that the mininet scripts were not updated when we migrated and took longer to get done. We focused on java-builder with the initial migration.
[15/06/2020 17:02:08] <zxiiro> mininet likely lagged behind with the ansible change
[15/06/2020 17:03:30] <zxiiro> ok so according to your comment on INTTEST the old image was dated 20190415-091034.881 that does seem plausible.
[15/06/2020 17:06:11] <rovarga> exactly
[15/06/2020 17:06:26] <rovarga> so this was broken by a change which was merged afterwards
[15/06/2020 17:06:28] <rovarga> *or*
[15/06/2020 17:06:32] <rovarga> NOT merged
[15/06/2020 17:06:49] <rovarga> there is no equivalent of https://git.opendaylight.org/gerrit/c/releng/builder/+/80752
[15/06/2020 17:07:02] <rovarga> for mininet-ovs-2.8
[15/06/2020 17:07:23] <rovarga> nite@nitebug : ~/odl/builder/packer on master $ diff -u templates/mininet-ovs-2.6.json templates/mininet-ovs-2.8.json | diffstat
[15/06/2020 17:07:23] <rovarga>  mininet-ovs-2.8.json |    8 ++++----
[15/06/2020 17:07:23] <rovarga>  1 file changed, 4 insertions(+), 4 deletions(-)
[15/06/2020 17:07:23] <rovarga> nite@nitebug : ~/odl/builder/packer on master $ diff -u provision/mininet-ovs-2.6.yaml provision/mininet-ovs-2.8.yaml  | diffstat
[15/06/2020 17:07:23] <rovarga>  mininet-ovs-2.8.yaml |  108 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---------------------------
[15/06/2020 17:07:23] <rovarga>  1 file changed, 81 insertions(+), 27 deletions(-)
[15/06/2020 17:08:12] <rovarga> i.e. while the templates are mostly the same, the playbooks are vastly different
[15/06/2020 17:18:48] <rovarga> zxiiro: and the delta of installing git-review is what broke us
[15/06/2020 17:18:52] <rovarga> https://packages.ubuntu.com/xenial/git-review
[15/06/2020 17:19:05] <rovarga> i.e. git-review pulls in python-requests (for python2)
[15/06/2020 17:19:19] <zxiiro> hmm interesting.
[15/06/2020 17:19:21] <rovarga> LuisGomez: ^^^^
[15/06/2020 17:19:46] <rovarga> I suspect it does not matter for other mininets, as we are not using them in this capacity
[15/06/2020 17:19:57] <zxiiro> I don't think we should install git-review on mininet as it doesn't need it and shouldn't be doing anything with Gerrit so we should just pull in requests directly.
[15/06/2020 17:20:07] <rovarga> yup
[15/06/2020 17:20:21] <rovarga> we do not have the smoking gun of a commit
[15/06/2020 17:20:41] <rovarga> but I think we have enough circumstantial evidence to explain what happened
[15/06/2020 17:20:44] <rovarga> and how to fix it
[15/06/2020 17:20:53] <zxiiro> agreed.