OpenDaylight JIRA

OpenDaylight JIRA https://jira.opendaylight.org This file is an XML representation of an issue en-us

8.20.10

820010

22-06-2022

[INTTEST-14] Use JVM monitoring in each test job https://jira.opendaylight.org/browse/INTTEST-14 integration-test <p>Issues such as high utilization can cause failures that are hard to reproduce and debug.</p> <p>Some jobs already monitor some metrics and generate graphs. There is no good reason why majority of jobs do not perform such monitoring.</p> <p>Operating System: All<br/> Platform: All</p> INTTEST-14

Use JVM monitoring in each test job

Bug Open Unresolved Unassigned Vratko Polak Mon, 29 May 2017 15:44:52 +0000 Tue, 12 Nov 2019 19:44:13 +0000 0 4 <p>> high utilization</p> <p>That was referring to Java Heap utilization, but other metrics can be useful as well, for example CPU utilization.</p> <p>For now this works consists on incorporating JVM plots instruction below:</p> <p><a href="https://wiki.opendaylight.org/view/CrossProject:Integration_Group:Adding_JVM_plots_to_Jobs" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.opendaylight.org/view/CrossProject:Integration_Group:Adding_JVM_plots_to_Jobs</a></p> <p>into the process to generate a CSIT job:</p> <p><a href="https://docs.opendaylight.org/projects/integration-test/en/latest/system-test-guide.html" class="external-link" target="_blank" rel="nofollow noopener">https://docs.opendaylight.org/projects/integration-test/en/latest/system-test-guide.html</a></p> <p>Hi Luis<br/> I was assigned to this project by my company. It seems to me that I could to add JVM monitoring to some tests (I would like to try - I am a begginer in this project). Could you please give me an advice with what test I could begin?</p> <p>Thanks</p> <p>Hi Stanislav,</p> <p>Thanks for jumping on this. It seems like the framework is not working now. If you see last test case here:</p> <p><a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/openflowplugin-csit-1node-perf-bulkomatic-only-neon/403/robot-plugin/log.html.gz#s1-s1" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/openflowplugin-csit-1node-perf-bulkomatic-only-neon/403/robot-plugin/log.html.gz#s1-s1</a></p> <p>The python robot KW <span class="error">[1]</span> throws an error: IndexError: list index out of range</p> <p>So I will have to dig a bit what is going on before we can extend the framework to other jobs.</p> <p><span class="error">[1]</span> <a href="https://git.opendaylight.org/gerrit/gitweb?p=integration/test.git;a=blob;f=csit/libraries/Appenders/ElasticsearchAppender.py;h=52cbe746c0a73a4f313674c55998153fdb0bc79e;hb=HEAD#l106" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/gitweb?p=integration/test.git;a=blob;f=csit/libraries/Appenders/ElasticsearchAppender.py;h=52cbe746c0a73a4f313674c55998153fdb0bc79e;hb=HEAD#l106</a></p> <p>There is an issue with decanter jmx collector but I figured out a workaround. See <a href="https://jira.opendaylight.org/browse/INTTEST-60" class="external-link" rel="nofollow">https://jira.opendaylight.org/browse/INTTEST-60</a>.</p> <p>Once this is fixed we can resume this ticket work.</p> <p>OK, fix is merged, you can try to add the JVM for an individual job by just following this wiki: </p> <p><a href="https://wiki.opendaylight.org/view/CrossProject:Integration_Group:Adding_JVM_plots_to_Jobs" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.opendaylight.org/view/CrossProject:Integration_Group:Adding_JVM_plots_to_Jobs</a></p> <p>If the goal is to add the JVM plots to all the jobs, I will have to take a look at how to do that.</p> <p>Maybe the best place where we could add decanter features into the <b>install-features:</b> clause is in templates?</p> <p>Then probably these features are going to be installed in all jobs which are built with that template</p> <p>Could you please confirm my thinking?</p> <p>Thanks</p> <p> </p> <p>edited:</p> <p>... or simply to add to all yaml files with definition of job to the <b>install-features:</b> clause <br/> (in the way how it was done to this moment)</p> <p>What is better?</p> <p> </p> <p>List of files with templates</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> jjb/autorelease/autorelease-templates.yaml jjb/global-jjb/jjb/lf-c-cpp-jobs.yaml jjb/global-jjb/jjb/lf-ci-jobs.yaml jjb/global-jjb/jjb/lf-docker-jobs.yaml jjb/global-jjb/jjb/lf-info-vote.yaml jjb/global-jjb/jjb/lf-maven-jobs.yaml jjb/global-jjb/jjb/lf-node-jobs.yaml jjb/global-jjb/jjb/lf-python-jobs.yaml jjb/global-jjb/jjb/lf-release-jobs.yaml jjb/global-jjb/jjb/lf-release-management-jobs.yaml jjb/global-jjb/jjb/lf-rtd-jobs.yaml jjb/global-jjb/jjb/lf-whitesource-jobs.yaml jjb/integration/distribution/distribution-templates.yaml jjb/integration/integration-templates.yaml jjb/integration/integration-test-jobs.yaml jjb/local/netvirt/netvirt-apex-images.yaml jjb/local/netvirt/netvirt-dependencies.yaml jjb/local/netvirt/netvirt-job-reports.yaml jjb/local/openci/openci-odl-daily-jobs.yaml jjb/local/opflex/opflex-custom.yaml jjb/local/opflex/opflex-thirdparty-dependencies.yaml jjb/local/opflex/opflex.yaml jjb/local/ovsdb/ovsdb-full-integration.yaml jjb/local/ovsdb/ovsdb-sonar.yaml jjb/local/packaging/packaging-ansible.yaml jjb/local/packaging/packaging-puppet.yaml jjb/local/packaging/packaging.yaml jjb/local/releng-jobs.yaml jjb/local/releng-templates-java.yaml</pre> </div></div> <p> </p> <p>List of files with project</p> <p> </p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> jjb/aaa/aaa-csit-authn.yaml jjb/aaa/aaa-patch-test.yaml jjb/aaa/aaa.yaml jjb/alt-datastores/alt-datastores.yaml jjb/alto/alto-csit-setup.yaml jjb/alto/alto.yaml jjb/ansible/ansible.yaml jjb/archetypes/archetypes.yaml jjb/autorelease/autorelease-projects.yaml jjb/autorelease/autorelease-version-management.yaml jjb/autorelease/validate-autorelease-neon.yaml jjb/autorelease/validate-autorelease-sodium.yaml jjb/bgpcep/bgpcep-csit-bgpclustering-ha.yaml jjb/bgpcep/bgpcep-csit-bgpclustering-longevity.yaml jjb/bgpcep/bgpcep-csit-bgpclustering.yaml jjb/bgpcep/bgpcep-csit-bgp-ingest-mixed.yaml jjb/bgpcep/bgpcep-csit-bgp-ingest.yaml jjb/bgpcep/bgpcep-csit-throughpcep.yaml jjb/bgpcep/bgpcep-csit-userfeatures.yaml jjb/bgpcep/bgpcep-patch-test.yaml jjb/bgpcep/bgpcep.yaml jjb/bier/bier-csit-basic.yaml jjb/bier/bier.yaml jjb/coe/coe.yaml jjb/controller/controller-csit-akka1.yaml jjb/controller/controller-csit-clustering.yaml jjb/controller/controller-csit-cs-chasing-leader.yaml jjb/controller/controller-csit-cs-partnheal.yaml jjb/controller/controller-csit-ddb-expl-lead-movement.yaml jjb/controller/controller-csit-dnb-noloss.yaml jjb/controller/controller-csit-drb-partnheal.yaml jjb/controller/controller-csit-drb-precedence.yaml jjb/controller/controller-csit-dsbenchmark-cluster.yaml jjb/controller/controller-csit-dsbenchmark.yaml jjb/controller/controller-csit-<span class="code-keyword">rest</span>-cars-perf.yaml jjb/controller/controller-csit-<span class="code-keyword">rest</span>-clust-cars-perf.yaml jjb/controller/controller-patch-test.yaml jjb/controller/controller.yaml jjb/daexim/daexim-csit-basic.yaml jjb/daexim/daexim-csit-clustering-basic.yaml jjb/daexim/daexim.yaml jjb/detnet/detnet.yaml jjb/dluxapps/dluxapps-csit-yangman.yaml jjb/dluxapps/dluxapps.yaml jjb/dlux/dlux.yaml jjb/docs/docs-rtd.yaml jjb/genius/genius-csit-genius.yaml jjb/genius/genius-patch-test.yaml jjb/genius/genius.yaml jjb/groupbasedpolicy/groupbasedpolicy-csit-1node-openstack.yaml jjb/groupbasedpolicy/groupbasedpolicy-csit-3node.yaml jjb/groupbasedpolicy/groupbasedpolicy-csit-6node.yaml jjb/groupbasedpolicy/groupbasedpolicy-csit-clustering.yaml jjb/groupbasedpolicy/groupbasedpolicy-csit-sxp.yaml jjb/groupbasedpolicy/groupbasedpolicy.yaml jjb/honeycomb/vbd/vbd.yaml jjb/infrautils/infrautils-patch-test.yaml jjb/infrautils/infrautils.yaml jjb/integration/distribution/distribution-csit-userfeatures.yaml jjb/integration/distribution/distribution-jobs.yaml jjb/integration/integration-test-jobs.yaml jjb/jsonrpc/jsonrpc-csit-basic.yaml jjb/jsonrpc/jsonrpc.yaml jjb/l2switch/l2switch-csit-host-scalability-daily.yaml jjb/l2switch/l2switch-csit-scalability.yaml jjb/l2switch/l2switch-csit-<span class="code-keyword">switch</span>.yaml jjb/l2switch/l2switch-patch-test.yaml jjb/l2switch/l2switch.yaml jjb/lf-infra/lf-infra-jobs.yaml jjb/lf-infra/zzz-test-release.yaml jjb/lfn/lfn.yaml jjb/lispflowmapping/lispflowmapping-csit-msmr.yaml jjb/lispflowmapping/lispflowmapping-csit-performance.yaml jjb/lispflowmapping/lispflowmapping.yaml jjb/local/mdsal/mdsal-csit-bindingv1.yaml jjb/local/mdsal/mdsal-patch-test.yaml jjb/local/mdsal/mdsal.yaml jjb/local/nemo/nemo-csit-engine.yaml jjb/local/nemo/nemo.yaml jjb/local/netconf/netconf-clustering-scale.yaml jjb/local/netconf/netconf-clustering.yaml jjb/local/netconf/netconf-csit-callhome.yaml jjb/local/netconf/netconf-patch-test.yaml jjb/local/netconf/netconf-scale.yaml jjb/local/netconf/netconf-userfeatures.yaml jjb/local/netconf/netconf.yaml jjb/local/netvirt/netvirt-apex-images.yaml jjb/local/netvirt/netvirt-csit-hwvtep-openstack.yaml jjb/local/netvirt/netvirt-csit-multi-apex.yaml jjb/local/netvirt/netvirt-csit-multi-openstack-upgrade.yaml jjb/local/netvirt/netvirt-csit-multi-openstack.yaml jjb/local/netvirt/netvirt-dependencies.yaml jjb/local/netvirt/netvirt-job-reports.yaml jjb/local/netvirt/netvirt-patch-test.yaml jjb/local/netvirt/netvirt.yaml jjb/local/neutron/neutron-patch-test.yaml jjb/local/neutron/neutron.yaml jjb/local/odlparent/odlparent-patch-test.yaml jjb/local/odlparent/odlparent.yaml jjb/local/odltools/odltools-csit.yaml jjb/local/odltools/odltools-patch-test.yaml jjb/local/odltools/odltools.yaml jjb/local/of-config/of-config.yaml jjb/local/openci/openci-odl-daily-jobs.yaml jjb/local/openflowplugin/openflowplugin-csit-bundle-based-reconciliation.yaml jjb/local/openflowplugin/openflowplugin-csit-cbench.yaml jjb/local/openflowplugin/openflowplugin-csit-clustering-bulkomatic.yaml jjb/local/openflowplugin/openflowplugin-csit-clustering-perf-bulkomatic.yaml jjb/local/openflowplugin/openflowplugin-csit-clustering.yaml jjb/local/openflowplugin/openflowplugin-csit-flow-services.yaml jjb/local/openflowplugin/openflowplugin-csit-longevity.yaml jjb/local/openflowplugin/openflowplugin-csit-perf-bulkomatic.yaml jjb/local/openflowplugin/openflowplugin-csit-perf-stats-collection.yaml jjb/local/openflowplugin/openflowplugin-csit-sanity.yaml jjb/local/openflowplugin/openflowplugin-csit-scale-link.yaml jjb/local/openflowplugin/openflowplugin-csit-scale-<span class="code-keyword">switch</span>.yaml jjb/local/openflowplugin/openflowplugin-csit-scale.yaml jjb/local/openflowplugin/openflowplugin-patch-test.yaml jjb/local/openflowplugin/openflowplugin.yaml jjb/local/opflex/opflex-custom.yaml jjb/local/opflex/opflex-thirdparty-dependencies.yaml jjb/local/opflex/opflex.yaml jjb/local/ovsdb/ovsdb-csit-clustering.yaml jjb/local/ovsdb/ovsdb-csit-southbound.yaml jjb/local/ovsdb/ovsdb-full-integration.yaml jjb/local/ovsdb/ovsdb-patch-test.yaml jjb/local/ovsdb/ovsdb-sonar.yaml jjb/local/ovsdb/ovsdb.yaml jjb/local/p4plugin/p4plugin.yaml jjb/local/packaging/packaging-ansible.yaml jjb/local/packaging/packaging-puppet.yaml jjb/local/packaging/packaging.yaml jjb/local/packetcable/packetcable-csit-pcmm.yaml jjb/local/packetcable/packetcable.yaml jjb/local/releng-jobs.yaml jjb/local/releng-packer-jobs.yaml jjb/local/releng-views.yaml jjb/local/serviceutils/serviceutils-patch-test.yaml jjb/local/serviceutils/serviceutils.yaml jjb/local/sfc/sfc-csit-full-deploy.yaml jjb/local/sfc/sfc-csit-<span class="code-keyword">rest</span>-basic.yaml jjb/local/sfc/sfc-patch-test.yaml jjb/local/sfc/sfc-python.yaml jjb/local/sfc/sfc.yaml jjb/local/snmp4sdn/snmp4sdn.yaml jjb/local/snmp/snmp.yaml jjb/local/sxp/sxp-csit-basic.yaml jjb/local/sxp/sxp-csit-binding-origin.yaml jjb/local/sxp/sxp-csit-cluster-binding.yaml jjb/local/sxp/sxp-csit-clustering.yaml jjb/local/sxp/sxp-csit-filtering.yaml jjb/local/sxp/sxp-csit-performance.yaml jjb/local/sxp/sxp-csit-topology.yaml jjb/local/sxp/sxp.yaml jjb/local/telemetry/telemetry-csit-basic.yaml jjb/local/telemetry/telemetry.yaml jjb/local/transportpce/transportpce.yaml jjb/local/tsdr/tsdr-csit-cassandra.yaml jjb/local/tsdr/tsdr-csit-elasticsearch.yaml jjb/local/tsdr/tsdr-csit-hbase.yaml jjb/local/tsdr/tsdr-csit-hsqldb.yaml jjb/local/tsdr/tsdr.yaml jjb/local/unimgr/unimgr.yaml jjb/local/usc/usc-csit-tcp.yaml jjb/local/usc/usc.yaml jjb/local/yangtools/yangtools-csit-system.yaml jjb/local/yangtools/yangtools-patch-test.yaml jjb/local/yangtools/yangtools.yaml </pre> </div></div> <p> </p> <p>List of files with job-group</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> jjb/global-jjb/jjb/lf-ci-job-groups.yaml jjb/global-jjb/jjb/lf-docker-job-groups.yaml jjb/global-jjb/jjb/lf-maven-job-groups.yaml jjb/global-jjb/jjb/lf-node-job-groups.yaml jjb/global-jjb/jjb/lf-python-job-groups.yaml jjb/global-jjb/jjb/lf-release-job-groups.yaml jjb/global-jjb/jjb/lf-rtd-job-groups.yaml jjb/local/releng-templates-java.yaml </pre> </div></div> <p>These contain decanter features:</p> <p>jjb/local/netconf/netconf-scale.yaml<br/> jjb/local/netvirt/netvirt-csit-hwvtep-openstack.yaml<br/> jjb/local/netvirt/netvirt-csit-multi-apex.yaml<br/> jjb/local/netvirt/netvirt-csit-multi-openstack-upgrade.yaml<br/> jjb/local/netvirt/netvirt-csit-multi-openstack.yaml<br/> jjb/local/openflowplugin/openflowplugin-csit-clustering-perf-bulkomatic.yaml<br/> jjb/local/openflowplugin/openflowplugin-csit-longevity.yaml<br/> jjb/local/openflowplugin/openflowplugin-csit-perf-bulkomatic.yaml<br/> jjb/local/openflowplugin/openflowplugin-csit-scale-link.yaml<br/> jjb/local/openflowplugin/openflowplugin-csit-scale-switch.yaml<br/> jjb/local/openflowplugin/openflowplugin-csit-scale.yaml<br/> jjb/local/ovsdb/ovsdb-csit-clustering.yaml</p> <p>So if we wanted to add the JVM plot for all jobs, we would have to move:</p> <p>1) decanter feature installation.<br/> 2) decanter & ES configuration & installation script.<br/> 3) python call to generate plots.</p> <p>to the controller install scripts:</p> <p><a href="https://git.opendaylight.org/gerrit/gitweb?p=releng/builder.git;a=blob;f=jjb/integration/common-functions.sh" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/gitweb?p=releng/builder.git;a=blob;f=jjb/integration/common-functions.sh</a><br/> <a href="https://git.opendaylight.org/gerrit/gitweb?p=releng/builder.git;a=blob;f=jjb/integration/integration-deploy-controller-run-test.sh" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/gitweb?p=releng/builder.git;a=blob;f=jjb/integration/integration-deploy-controller-run-test.sh</a><br/> <a href="https://git.opendaylight.org/gerrit/gitweb?p=releng/builder.git;a=blob;f=jjb/integration/integration-configure-clustering.sh" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/gitweb?p=releng/builder.git;a=blob;f=jjb/integration/integration-configure-clustering.sh</a></p> <p>I will check anyway in the next integration call if anyone has any objection to this work.</p> <p>no objection from me.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=stanislav-chlebec" class="user-hover" rel="stanislav-chlebec">stanislav-chlebec</a>, if you want to add the JVM report for all jobs I would suggest to go the route I suggested above: modifying 3 scripts in releng/builder vs changing individual projects settings. Let me know if you have further questions.</p> <p>Sure. Thanks.</p> <p>ERROR:root:Problem formatting with args:<br/> allow_empty:False<br/> obj: OrderedDict(<span class="error">[('name', 'ELASTICSEARCHATTRIBUTE'), ('default', '\{feature-elasticsearch-attribute}'), ('description', 'Configure the period the script should harvest JVM metrics.\nDefault is short.\nPossible values: long (its value is 120000)\n short (its value is 5000)\n any number between 5000 and 120000\n')]</span>)<br/> paramdict: OrderedDict(<span class="error">[('controller-features', ''), ('controller-debug-map', ''), ('controller-scope', ''), ('controller-use-features-boot', '')]</span>)<br/> ERROR:root:Failure formatting component ('string') data 'OrderedDict(<span class="error">[('name', 'ELASTICSEARCHATTRIBUTE'), ('default', '\{feature-elasticsearch-attribute}'), ('description', 'Configure the period the script should harvest JVM metrics.\nDefault is short.\nPossible values: long (its value is 120000)\n short (its value is 5000)\n any number between 5000 and 120000\n')]</span>)'<br/> Traceback (most recent call last):<br/> File "/tmp/tmp.bg1jmOPwmn/bin/jenkins-jobs", line 11, in <module><br/> sys.exit(main())<br/> File "/tmp/tmp.bg1jmOPwmn/lib/python2.7/site-packages/jenkins_jobs/cli/entry.py", line 165, in main<br/> jjb.execute()<br/> File "/tmp/tmp.bg1jmOPwmn/lib/python2.7/site-packages/jenkins_jobs/cli/entry.py", line 146, in execute<br/> ext.obj.execute(self.options, self.jjb_config)<br/> File "/tmp/tmp.bg1jmOPwmn/lib/python2.7/site-packages/jenkins_jobs/cli/subcommand/test.py", line 63, in execute<br/> options, jjb_config)<br/> File "/tmp/tmp.bg1jmOPwmn/lib/python2.7/site-packages/jenkins_jobs/cli/subcommand/update.py", line 115, in _generate_xmljobs<br/> xml_jobs = xml_job_generator.generateXML(job_data_list)<br/> File "/tmp/tmp.bg1jmOPwmn/lib/python2.7/site-packages/jenkins_jobs/xml_config.py", line 82, in generateXML<br/> xml_objs.append(self._getXMLForData(data))<br/> File "/tmp/tmp.bg1jmOPwmn/lib/python2.7/site-packages/jenkins_jobs/xml_config.py", line 94, in _getXMLForData<br/> self._gen_xml(xml, data)<br/> File "/tmp/tmp.bg1jmOPwmn/lib/python2.7/site-packages/jenkins_jobs/xml_config.py", line 108, in _gen_xml<br/> module.gen_xml(xml, data)<br/> File "/tmp/tmp.bg1jmOPwmn/lib/python2.7/site-packages/jenkins_jobs/modules/parameters.py", line 1070, in gen_xml<br/> self.registry.dispatch('parameter', pdefs, param)<br/> File "/tmp/tmp.bg1jmOPwmn/lib/python2.7/site-packages/jenkins_jobs/registry.py", line 253, in dispatch<br/> self.dispatch(component_type, xml_parent, b, component_data)<br/> File "/tmp/tmp.bg1jmOPwmn/lib/python2.7/site-packages/jenkins_jobs/registry.py", line 171, in dispatch<br/> self.jjb_config.yamlparser<span class="error">['allow_empty_variables']</span>)<br/> File "/tmp/tmp.bg1jmOPwmn/lib/python2.7/site-packages/jenkins_jobs/formatter.py", line 59, in deep_format<br/> deep_format(obj<span class="error">[item]</span>, paramdict, allow_empty)<br/> File "/tmp/tmp.bg1jmOPwmn/lib/python2.7/site-packages/jenkins_jobs/formatter.py", line 43, in deep_format<br/> raise JenkinsJobsException(desc)<br/> jenkins_jobs.errors.JenkinsJobsException: feature-elasticsearch-attribute parameter missing to format {feature-elasticsearch-attribute}<br/> Given:<br/> OrderedDict(<span class="error">[('controller-features', ''), ('controller-debug-map', ''), ('controller-scope', ''), ('controller-use-features-boot', '')]</span>)</p> <p>jjb/integration/integration-macros.yaml<br/> 39:121 error line too long (153 > 120 characters) (line-length)</p> <p> </p> <p>Test shell scripts with shellcheck.......................................Failed<br/> hookid: shellcheck</p> <p>In jjb/integration/common-functions.sh line 284:<br/> if [ ${type} == "script" ]; then<br/> <sup>-----</sup> SC2086: Double quote to prevent globbing and word splitting.</p> <p>Did you mean: <br/> if [ "${type}" == "script" ]; then</p> <p>In jjb/integration/common-functions.sh line 288:<br/> if [ ${ELASTICSEARCHATTRIBUTE} == "short" ]; then<br/> <sup>-----------------------</sup> SC2086: Double quote to prevent globbing and word splitting.</p> <p>Did you mean: <br/> if [ "${ELASTICSEARCHATTRIBUTE}" == "short" ]; then</p> <p>For more information:<br/> <a href="https://www.shellcheck.net/wiki/SC2086" class="external-link" target="_blank" rel="nofollow noopener">https://www.shellcheck.net/wiki/SC2086</a> – Double quote to prevent globbing ...</p> <p>yamllint.................................................................Passed<br/> ERROR: InvocationError for command /w/workspace/builder-tox-verify-master/.tox/pre-commit/bin/pre-commit run --all-files (exited with code 1)</p> <p>I made an attempt to run a job</p> <p><a href="https://jenkins.opendaylight.org/releng/job/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/job/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/</a></p> <p>which was generated from my repo "releng/builder" (where are my changes - commit (jjb) stanislav@ArmDev:~/work/odl/builder$ git log -1<br/> commit 13139b76183f9302ff4f296795cf8e58062e3724<br/> Author: Stanislav Chlebec <stanislav.chlebec@pantheon.tech><br/> Date: Fri Oct 25 19:33:42 2019 +0200)</p> <p>While trying to test this jenkins job with changes from integration/test repo I provided the value to the parameter GERRIT_REFSPEC</p> <p>refs/changes/94/85394/2</p> <p>I am not sure if my approach was right because job failed and in the log I found<br/> <b>16:06:04</b> > git checkout -f ecfaf49c97b4b7df7050ad5740cc160d5b676af5*16:06:05* Commit message: "<a href="https://jira.opendaylight.org/browse/INTTEST-61" title="Lisp Flow mapping Failing - CSIT issue" class="issue-link" data-issue-key="INTTEST-61"><del>INTTEST-61</del></a> lispflowmapping failing"<b>16:06:05</b> > git rev-parse FETCH_HEAD^{commit} # timeout=10*16:06:05* > git rev-list --no-walk fbd8d762c1fb6e6e253bec4075502cbbc4587eb0 # timeout=10<br/> <b>I</b> do not see the reference to my patch so I wonder what I did wrong</p> <p>Please could you help me and to instruct me how to test?</p> <p>Thanks<br/> <br/> </p> <p>looks like you have the right workflow here. The problem I see in your <a href="https://jenkins.opendaylight.org/sandbox/job/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/1/console" class="external-link" target="_blank" rel="nofollow noopener">sandbox job</a> is something broken with common-functions.sh. Search for "07:26:03 Executing cat..." and you'll see some syntax errors. That's where<br/> to focus at this point.</p> <p>The issue I see in last sandbox run is JVM scripts are run twice. To avoid that pick another job that does not have JVM yet or use PATCHREFSPEC=refs/changes/94/85394/2 (not GERRIT_REFSPEC)</p> <p>Thanks for comments.</p> <p><a href="https://jenkins.opendaylight.org/sandbox/job/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/6/console" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/6/console</a></p> <p>I set PATCHREFSPEC=refs/changes/94/85394/2 as Luis recommneded</p> <p>I then compared the console output to <a href="https://jenkins.opendaylight.org/releng/job/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/54/parameters/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/job/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/54/parameters/</a></p> <p>According this successfull job I adjusted the parameter BUNDLE_URL to <a href="https://nexus.opendaylight.org/content/repositories//autorelease-3465/org/opendaylight/integration/karaf/0.12.0/karaf-0.12.0.zip" class="external-link" target="_blank" rel="nofollow noopener">https://nexus.opendaylight.org/content/repositories//autorelease-3465/org/opendaylight/integration/karaf/0.12.0/karaf-0.12.0.zip</a></p> <p> </p> <p>In spite of these tweaks I got failure:<br/> <b>08:39:43</b> postplan exists!!!<b>08:39:43</b> Changing the postplan path...<b>08:39:43</b> # Place the commands in run order:<b>08:39:43</b> python3 /w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/test/csit/scripts/push_test_data.py a4ff38b99ef2c7626450543021b4c134.us-east-1.aws.found.io:9243*08:39:43* Executing python3 /w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/test/csit/scripts/push_test_data.py a4ff38b99ef2c7626450543021b4c134.us-east-1.aws.found.io:9243...<b>08:39:44</b> Traceback (most recent call last):<b>08:39:44</b> File "/w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/test/csit/scripts/push_test_data.py", line 93, in <module><b>08:39:44</b> BODY = data_gen.generate()<b>08:39:44</b> File "/w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/test/csit/scripts/data_generate.py", line 49, in generate*08:39:44* tree = ET.parse(robot_log)<b>08:39:44</b> File "/usr/lib64/python3.6/xml/etree/ElementTree.py", line 1196, in parse*08:39:44* tree.parse(source, parser)<b>08:39:44</b> File "/usr/lib64/python3.6/xml/etree/ElementTree.py", line 586, in parse*08:39:44* source = open(source, "rb")<b>08:39:44</b> FileNotFoundError: <span class="error">[Errno 2]</span> No such file or directory: '/w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/output.xml'</p> <p>Again I do not know what can be a reason. Please do you have some advice for me?</p> <p>I spent some time digging through your patches and job. Like I wrote above, the abort in your sandbox<br/> job is coming from something in common-functions. Did you see this?</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> 00:39:06 Executing /w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/test/csit/scripts/controllermem_6g.sh... 00:39:06 Executing cat... 00:39:06 /tmp/common-functions.sh: line 327: source: /usr/bin/cat: cannot execute binary file 00:39:06 Executing >... 00:39:06 /tmp/common-functions.sh: line 327: >: No such file or directory 00:39:06 Executing ${WORKSPACE}/org.apache.karaf.decanter.scheduler.simple.cfg... 00:39:06 /tmp/common-functions.sh: line 327: ${WORKSPACE}/org.apache.karaf.decanter.scheduler.simple.cfg: No such file or directory 00:39:06 Executing <<EOF... 00:39:06 /tmp/common-functions.sh: line 327: <<EOF: No such file or directory 00:39:06 Executing period=5000... 00:39:06 /tmp/common-functions.sh: line 327: period=5000: No such file or directory 00:39:06 Executing EOF... 00:39:06 /tmp/common-functions.sh: line 327: EOF: No such file or directory 00:39:06 Executing echo... 00:39:06 /tmp/common-functions.sh: line 327: source: /usr/bin/echo: cannot execute binary file 00:39:06 Executing "Copying... 00:39:06 /tmp/common-functions.sh: line 327: "Copying: No such file or directory 00:39:06 Executing config... 00:39:06 /tmp/common-functions.sh: line 327: config: No such file or directory 00:39:06 Executing files... 00:39:06 /tmp/common-functions.sh: line 327: files: No such file or directory 00:39:06 Executing to... 00:39:06 /tmp/common-functions.sh: line 327: to: No such file or directory 00:39:06 Executing ODL... 00:39:06 /tmp/common-functions.sh: line 327: ODL: No such file or directory 00:39:06 Executing Controller... 00:39:06 /tmp/common-functions.sh: line 327: Controller: No such file or directory 00:39:06 Executing folder"... 00:39:06 /tmp/common-functions.sh: line 327: folder": No such file or directory 00:39:06 Executing <span class="code-keyword">for</span>... 00:39:06 /tmp/common-functions.sh: line 327: <span class="code-keyword">for</span>: No such file or directory 00:39:06 Executing i... 00:39:06 /tmp/common-functions.sh: line 327: i: No such file or directory 00:39:06 Executing in... 00:39:06 /tmp/common-functions.sh: line 327: in: No such file or directory 00:39:06 Executing `seq... 00:39:06 /tmp/common-functions.sh: line 327: `seq: No such file or directory 00:39:06 Executing 1... 00:39:06 /tmp/common-functions.sh: line 327: 1: No such file or directory 00:39:06 Executing ${NUM_ODL_SYSTEM}`... 00:39:06 /tmp/common-functions.sh: line 327: ${NUM_ODL_SYSTEM}`: No such file or directory 00:39:06 Executing <span class="code-keyword">do</span>... 00:39:06 /tmp/common-functions.sh: line 327: <span class="code-keyword">do</span>: No such file or directory 00:39:06 Executing CONTROLLERIP=ODL_SYSTEM_${i}_IP... 00:39:06 /tmp/common-functions.sh: line 327: CONTROLLERIP=ODL_SYSTEM_${i}_IP: No such file or directory 00:39:06 Executing echo... 00:39:06 /tmp/common-functions.sh: line 327: source: /usr/bin/echo: cannot execute binary file 00:39:06 Executing "Setup... 00:39:06 /tmp/common-functions.sh: line 327: "Setup: No such file or directory 00:39:06 Executing <span class="code-object">long</span>... 00:39:06 /tmp/common-functions.sh: line 327: <span class="code-object">long</span>: No such file or directory 00:39:06 Executing duration... 00:39:06 /tmp/common-functions.sh: line 327: duration: No such file or directory 00:39:06 Executing config... 00:39:06 /tmp/common-functions.sh: line 327: config: No such file or directory 00:39:06 Executing to... 00:39:06 /tmp/common-functions.sh: line 327: to: No such file or directory 00:39:06 Executing ${!CONTROLLERIP}"... 00:39:06 /tmp/common-functions.sh: line 327: ${!CONTROLLERIP}": No such file or directory 00:39:06 Executing ssh... 00:39:06 /tmp/common-functions.sh: line 327: source: /usr/bin/ssh: cannot execute binary file 00:39:06 Executing ${!CONTROLLERIP}... 00:39:06 /tmp/common-functions.sh: line 327: ${!CONTROLLERIP}: No such file or directory 00:39:06 Executing "mkdir... 00:39:06 /tmp/common-functions.sh: line 327: "mkdir: No such file or directory 00:39:06 Executing -p... 00:39:06 /tmp/common-functions.sh: line 327: source: -p: invalid option 00:39:06 source: usage: source filename [arguments] 00:39:06 Executing /tmp/${BUNDLEFOLDER}/etc/opendaylight/karaf/"... 00:39:06 /tmp/common-functions.sh: line 327: /tmp/${BUNDLEFOLDER}/etc/opendaylight/karaf/": No such file or directory 00:39:06 Executing scp... 00:39:06 /tmp/common-functions.sh: line 327: source: /usr/bin/scp: cannot execute binary file 00:39:06 Executing ${WORKSPACE}/org.apache.karaf.decanter.scheduler.simple.cfg... 00:39:06 /tmp/common-functions.sh: line 327: ${WORKSPACE}/org.apache.karaf.decanter.scheduler.simple.cfg: No such file or directory 00:39:06 Executing ${!CONTROLLERIP}:/tmp/${BUNDLEFOLDER}/etc/... 00:39:06 /tmp/common-functions.sh: line 327: ${!CONTROLLERIP}:/tmp/${BUNDLEFOLDER}/etc/: No such file or directory 00:39:06 Executing done... 00:39:06 /tmp/common-functions.sh: line 327: done: No such file or directory 00:39:06 Executing cat... 00:39:06 /tmp/common-functions.sh: line 327: source: /usr/bin/cat: cannot execute binary file 00:39:06 Executing >... 00:39:06 /tmp/common-functions.sh: line 327: >: No such file or directory 00:39:06 Executing ${WORKSPACE}/org.apache.karaf.decanter.collector.jmx-local.cfg... 00:39:06 /tmp/common-functions.sh: line 327: ${WORKSPACE}/org.apache.karaf.decanter.collector.jmx-local.cfg: No such file or directory 00:39:06 Executing <<EOF... 00:39:06 /tmp/common-functions.sh: line 327: <<EOF: No such file or directory 00:39:06 Executing type=jmx-local... 00:39:06 /tmp/common-functions.sh: line 327: type=jmx-local: No such file or directory 00:39:06 Executing url=local... 00:39:06 /tmp/common-functions.sh: line 327: url=local: No such file or directory 00:39:06 Executing object.name=java.lang:type=*,name=*... 00:39:06 /tmp/common-functions.sh: line 327: object.name=java.lang:type=*,name=*: No such file or directory 00:39:06 Executing EOF... 00:39:06 /tmp/common-functions.sh: line 327: EOF: No such file or directory 00:39:06 Executing cat... 00:39:06 /tmp/common-functions.sh: line 327: source: /usr/bin/cat: cannot execute binary file 00:39:06 Executing >... 00:39:06 /tmp/common-functions.sh: line 327: >: No such file or directory 00:39:06 Executing ${WORKSPACE}/org.apache.karaf.decanter.collector.jmx-others.cfg... 00:39:06 /tmp/common-functions.sh: line 327: ${WORKSPACE}/org.apache.karaf.decanter.collector.jmx-others.cfg: No such file or directory 00:39:06 Executing <<EOF... 00:39:06 /tmp/common-functions.sh: line 327: <<EOF: No such file or directory 00:39:06 Executing type=jmx-local... 00:39:06 /tmp/common-functions.sh: line 327: type=jmx-local: No such file or directory 00:39:06 Executing url=local... 00:39:06 /tmp/common-functions.sh: line 327: url=local: No such file or directory 00:39:06 Executing object.name=java.lang:type=*... 00:39:06 /tmp/common-functions.sh: line 327: object.name=java.lang:type=*: No such file or directory 00:39:06 Executing EOF... 00:39:06 /tmp/common-functions.sh: line 327: EOF: No such file or directory 00:39:06 Executing <span class="code-keyword">for</span>... 00:39:06 /tmp/common-functions.sh: line 327: <span class="code-keyword">for</span>: No such file or directory 00:39:06 Executing i... 00:39:06 /tmp/common-functions.sh: line 327: i: No such file or directory 00:39:06 Executing in... 00:39:06 /tmp/common-functions.sh: line 327: in: No such file or directory 00:39:06 Executing `seq... 00:39:06 /tmp/common-functions.sh: line 327: `seq: No such file or directory 00:39:06 Executing 1... 00:39:06 /tmp/common-functions.sh: line 327: 1: No such file or directory 00:39:06 Executing ${NUM_ODL_SYSTEM}`... 00:39:06 /tmp/common-functions.sh: line 327: ${NUM_ODL_SYSTEM}`: No such file or directory 00:39:06 Executing <span class="code-keyword">do</span>... 00:39:06 /tmp/common-functions.sh: line 327: <span class="code-keyword">do</span>: No such file or directory 00:39:06 Executing CONTROLLERIP=ODL_SYSTEM_${i}_IP... 00:39:06 /tmp/common-functions.sh: line 327: CONTROLLERIP=ODL_SYSTEM_${i}_IP: No such file or directory 00:39:06 Executing CLUSTERNAME=`head... 00:39:06 /tmp/common-functions.sh: line 327: CLUSTERNAME=`head: No such file or directory 00:39:06 Executing /dev/urandom... 00:39:42 /tmp/common-functions.sh: xrealloc: cannot allocate 18446744071562068032 bytes (1196032 bytes allocated) 00:39:42 Build step <span class="code-quote">'Execute shell'</span> marked build as failure </pre> </div></div> <p>Looking closer, I think you are trying to modify some file <tt>org.apache.karaf.decanter.scheduler.simple.cfg</tt> but that<br/> file comes from the controller once it's deployed right? I'm not sure, but you can check.</p> <p>we have three types of "plans" for making configurations. <a href="https://wiki.opendaylight.org/view/CrossProject:Integration_Group:How_To_Install_test_SW#3.29_Integration_scriptplan" class="external-link" target="_blank" rel="nofollow noopener">Scriptplans, Configplans and Postplans.</a><br/> I think you are doing your work from a scriptplan which might not work for this. Maybe you need to use the configplan?</p> <p>There are 3 issues AFAICT:</p> <p>1) Decanter collector does not work in master: <a href="https://jira.opendaylight.org/projects/INTTEST/issues/INTTEST-62" class="external-link" rel="nofollow">https://jira.opendaylight.org/projects/INTTEST/issues/INTTEST-62</a><br/> 2) JVM plots do not work with cluster because cluster deploy scripts are not using common functions: <a href="https://git.opendaylight.org/gerrit/#/c/releng/builder/+/85445/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/releng/builder/+/85445/</a><br/> 3) The proposed script has some issues, I left some comments in the patch: <a href="https://git.opendaylight.org/gerrit/#/c/releng/builder/+/85398/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/releng/builder/+/85398/</a></p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> + echo <span class="code-quote">'#################################################'</span> ################################################# + echo <span class="code-quote">'## Verify Cluster is UP ##'</span> ## Verify Cluster is UP ## + echo <span class="code-quote">'#################################################'</span> ################################################# + cat ++ seq 1 3 + <span class="code-keyword">for</span> i in <span class="code-quote">'$(seq 1 <span class="code-quote">"${NUM_ODL_SYSTEM}"</span>)'</span> + CONTROLLERIP=ODL_SYSTEM_1_IP + echo <span class="code-quote">'Verifying member-1 with IP address 10.30.170.135 is UP'</span> Verifying member-1 with IP address 10.30.170.135 is UP + scp /w/workspace/stano-v2-openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/verify-cluster-is-up.sh 10.30.170.135:/tmp Warning: Permanently added <span class="code-quote">'10.30.170.135'</span> (ECDSA) to the list of known hosts. + ssh 10.30.170.135 <span class="code-quote">'bash /tmp/verify-cluster-is-up.sh 1 10.30.170.135'</span> Warning: Permanently added <span class="code-quote">'10.30.170.135'</span> (ECDSA) to the list of known hosts. Waiting <span class="code-keyword">for</span> controller to come up... 000 http:<span class="code-comment">//localhost:8181/restconf/modules </span>000 http:<span class="code-comment">//localhost:8181/jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore </span>000 http:<span class="code-comment">//localhost:8181/restconf/modules </span>000 http:<span class="code-comment">//localhost:8181/jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore </span>000 http:<span class="code-comment">//localhost:8181/restconf/modules </span>000 http:<span class="code-comment">//localhost:8181/jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore </span>000 http:<span class="code-comment">//localhost:8181/restconf/modules </span>000 http:<span class="code-comment">//localhost:8181/jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore </span>000 http:<span class="code-comment">//localhost:8181/restconf/modules </span>000 http:<span class="code-comment">//localhost:8181/jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore </span>already waited 5 seconds... ... ... ... 401 http:<span class="code-comment">//localhost:8181/restconf/modules </span><html> <head> <meta http-equiv=<span class="code-quote">"Content-Type"</span> content=<span class="code-quote">"text/html;charset=utf-8"</span>/> <title>Error 401 Unauthorized</title> </head> <body><h2>HTTP ERROR 401</h2> <p>Problem accessing /jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore. Reason: <pre> Unauthorized</pre></p> </body> </html> 401 http:<span class="code-comment">//localhost:8181/jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore </span>401 http:<span class="code-comment">//localhost:8181/restconf/modules </span><html> <head> <meta http-equiv=<span class="code-quote">"Content-Type"</span> content=<span class="code-quote">"text/html;charset=utf-8"</span>/> <title>Error 401 Unauthorized</title> </head> <body><h2>HTTP ERROR 401</h2> <p>Problem accessing /jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore. Reason: <pre> Unauthorized</pre></p> </body> </html> 401 http:<span class="code-comment">//localhost:8181/jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore </span>401 http:<span class="code-comment">//localhost:8181/restconf/modules </span><html> <head> <meta http-equiv=<span class="code-quote">"Content-Type"</span> content=<span class="code-quote">"text/html;charset=utf-8"</span>/> <title>Error 401 Unauthorized</title> </head> <body><h2>HTTP ERROR 401</h2> <p>Problem accessing /jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore. Reason: <pre> Unauthorized</pre></p> </body> </html> 401 http:<span class="code-comment">//localhost:8181/jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore </span>401 http:<span class="code-comment">//localhost:8181/restconf/modules </span><html> <head> <meta http-equiv=<span class="code-quote">"Content-Type"</span> content=<span class="code-quote">"text/html;charset=utf-8"</span>/> <title>Error 401 Unauthorized</title> </head> <body><h2>HTTP ERROR 401</h2> <p>Problem accessing /jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore. Reason: <pre> Unauthorized</pre></p> </body> </html> 401 http:<span class="code-comment">//localhost:8181/jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore </span>404 http:<span class="code-comment">//localhost:8181/restconf/modules </span>{<span class="code-quote">"request"</span>:{<span class="code-quote">"mbean"</span>:<span class="code-quote">"org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore"</span>,<span class="code-quote">"type"</span>:<span class="code-quote">"read"</span>},<span class="code-quote">"value"</span>:{<span class="code-quote">"ReadWriteTransactionCount"</span>:0,<span class="code-quote">"SnapshotIndex"</span>:1,<span class="code-quote">"InMemoryJournalLogSize"</span>:1,<span class="code-quote">"ReplicatedToAllIndex"</span>:1,<span class="code-quote">"Leader"</span>:<span class="code-quote">"member-3-shard-inventory-config"</span>,<span class="code-quote">"LastIndex"</span>:2,<span class="code-quote">"RaftState"</span>:<span class="code-quote">"Follower"</span>,<span class="code-quote">"LastApplied"</span>:2,<span class="code-quote">"LastCommittedTransactionTime"</span>:<span class="code-quote">"1970-01-01 00:00:00.000"</span>,<span class="code-quote">"PeerAddresses"</span>:<span class="code-quote">"member-2-shard-inventory-config: akka.tcp:\/\/opendaylight-cluster-data@10.30.170.171:2550\/user\/shardmanager-config\/member-2-shard-inventory-config, member-3-shard-inventory-config: akka.tcp:\/\/opendaylight-cluster-data@10.30.170.84:2550\/user\/shardmanager-config\/member-3-shard-inventory-config"</span>,<span class="code-quote">"LastLogIndex"</span>:2,<span class="code-quote">"LastLeadershipChangeTime"</span>:<span class="code-quote">"2019-11-05 10:14:06.554"</span>,<span class="code-quote">"FollowerInitialSyncStatus"</span>:<span class="code-keyword">true</span>,<span class="code-quote">"FollowerInfo"</span>:[],<span class="code-quote">"FailedReadTransactionsCount"</span>:0,<span class="code-quote">"Voting"</span>:<span class="code-keyword">true</span>,<span class="code-quote">"StatRetrievalTime"</span>:<span class="code-quote">"1.421 ms"</span>,<span class="code-quote">"CurrentTerm"</span>:2,<span class="code-quote">"LastTerm"</span>:2,<span class="code-quote">"FailedTransactionsCount"</span>:0,<span class="code-quote">"PendingTxCommitQueueSize"</span>:0,<span class="code-quote">"VotedFor"</span>:<span class="code-quote">"member-3-shard-inventory-config"</span>,<span class="code-quote">"SnapshotCaptureInitiated"</span>:<span class="code-keyword">false</span>,<span class="code-quote">"CommittedTransactionsCount"</span>:0,<span class="code-quote">"TxCohortCacheSize"</span>:0,<span class="code-quote">"PeerVotingStates"</span>:<span class="code-quote">"member-2-shard-inventory-config: <span class="code-keyword">true</span>, member-3-shard-inventory-config: <span class="code-keyword">true</span>"</span>,<span class="code-quote">"LastLogTerm"</span>:2,<span class="code-quote">"StatRetrievalError"</span>:<span class="code-keyword">null</span>,<span class="code-quote">"CommitIndex"</span>:2,<span class="code-quote">"SnapshotTerm"</span>:2,<span class="code-quote">"AbortTransactionsCount"</span>:0,<span class="code-quote">"ReadOnlyTransactionCount"</span>:0,<span class="code-quote">"ShardName"</span>:<span class="code-quote">"member-1-shard-inventory-config"</span>,<span class="code-quote">"LeadershipChangeCount"</span>:1,<span class="code-quote">"InMemoryJournalDataSize"</span>:33},<span class="code-quote">"timestamp"</span>:1572948874,<span class="code-quote">"status"</span>:200}200 http:<span class="code-comment">//localhost:8181/jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore </span>already waited 70 seconds... 200 http:<span class="code-comment">//localhost:8181/restconf/modules </span>{<span class="code-quote">"request"</span>:{<span class="code-quote">"mbean"</span>:<span class="code-quote">"org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore"</span>,<span class="code-quote">"type"</span>:<span class="code-quote">"read"</span>},<span class="code-quote">"value"</span>:{<span class="code-quote">"ReadWriteTransactionCount"</span>:0,<span class="code-quote">"SnapshotIndex"</span>:1,<span class="code-quote">"InMemoryJournalLogSize"</span>:1,<span class="code-quote">"ReplicatedToAllIndex"</span>:1,<span class="code-quote">"Leader"</span>:<span class="code-quote">"member-3-shard-inventory-config"</span>,<span class="code-quote">"LastIndex"</span>:2,<span class="code-quote">"RaftState"</span>:<span class="code-quote">"Follower"</span>,<span class="code-quote">"LastApplied"</span>:2,<span class="code-quote">"LastCommittedTransactionTime"</span>:<span class="code-quote">"1970-01-01 00:00:00.000"</span>,<span class="code-quote">"PeerAddresses"</span>:<span class="code-quote">"member-2-shard-inventory-config: akka.tcp:\/\/opendaylight-cluster-data@10.30.170.171:2550\/user\/shardmanager-config\/member-2-shard-inventory-config, member-3-shard-inventory-config: akka.tcp:\/\/opendaylight-cluster-data@10.30.170.84:2550\/user\/shardmanager-config\/member-3-shard-inventory-config"</span>,<span class="code-quote">"LastLogIndex"</span>:2,<span class="code-quote">"LastLeadershipChangeTime"</span>:<span class="code-quote">"2019-11-05 10:14:06.554"</span>,<span class="code-quote">"FollowerInitialSyncStatus"</span>:<span class="code-keyword">true</span>,<span class="code-quote">"FollowerInfo"</span>:[],<span class="code-quote">"FailedReadTransactionsCount"</span>:0,<span class="code-quote">"Voting"</span>:<span class="code-keyword">true</span>,<span class="code-quote">"StatRetrievalTime"</span>:<span class="code-quote">"1.421 ms"</span>,<span class="code-quote">"CurrentTerm"</span>:2,<span class="code-quote">"LastTerm"</span>:2,<span class="code-quote">"FailedTransactionsCount"</span>:0,<span class="code-quote">"PendingTxCommitQueueSize"</span>:0,<span class="code-quote">"VotedFor"</span>:<span class="code-quote">"member-3-shard-inventory-config"</span>,<span class="code-quote">"SnapshotCaptureInitiated"</span>:<span class="code-keyword">false</span>,<span class="code-quote">"CommittedTransactionsCount"</span>:0,<span class="code-quote">"TxCohortCacheSize"</span>:0,<span class="code-quote">"PeerVotingStates"</span>:<span class="code-quote">"member-2-shard-inventory-config: <span class="code-keyword">true</span>, member-3-shard-inventory-config: <span class="code-keyword">true</span>"</span>,<span class="code-quote">"LastLogTerm"</span>:2,<span class="code-quote">"StatRetrievalError"</span>:<span class="code-keyword">null</span>,<span class="code-quote">"CommitIndex"</span>:2,<span class="code-quote">"SnapshotTerm"</span>:2,<span class="code-quote">"AbortTransactionsCount"</span>:0,<span class="code-quote">"ReadOnlyTransactionCount"</span>:0,<span class="code-quote">"ShardName"</span>:<span class="code-quote">"member-1-shard-inventory-config"</span>,<span class="code-quote">"LeadershipChangeCount"</span>:1,<span class="code-quote">"InMemoryJournalDataSize"</span>:33},<span class="code-quote">"timestamp"</span>:1572948875,<span class="code-quote">"status"</span>:200}200 http:<span class="code-comment">//localhost:8181/jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore </span>Controller is UP </pre> </div></div> <p><a href="https://jenkins.opendaylight.org/sandbox/view/All/job/stano-v2-openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/1/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/view/All/job/stano-v2-openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/1/</a></p> <p><a href="https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/57/console" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/57/console</a></p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> + suite_list=<span class="code-quote">'/w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/test/csit/suites/openflowplugin/Performance_Bulkomatic/100K_Flows_100DPN_3Node_Cluster_Perf_Test.robot /w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/test/csit/suites/integration/Create_JVM_Plots.robot '</span> + eval <span class="code-quote">'SUITES='</span>\<span class="code-quote">''/w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/test/csit/suites/openflowplugin/Performance_Bulkomatic/100K_Flows_100DPN_3Node_Cluster_Perf_Test.robot /w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/test/csit/suites/integration/Create_JVM_Plots.robot '</span>\''' ++ SUITES=<span class="code-quote">'/w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/test/csit/suites/openflowplugin/Performance_Bulkomatic/100K_Flows_100DPN_3Node_Cluster_Perf_Test.robot /w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/test/csit/suites/integration/Create_JVM_Plots.robot '</span> </pre> </div></div> <p> </p> <p><a href="https://jenkins.opendaylight.org/sandbox/view/All/job/stano-v2-openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/1/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/view/All/job/stano-v2-openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/1/</a></p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> + suite_list=<span class="code-quote">'/w/workspace/stano-v2-openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/test/csit/suites/openflowplugin/Performance_Bulkomatic/100K_Flows_100DPN_3Node_Cluster_Perf_Test.robot '</span> + add_jvm_plots=integration/test/csit/suites/integration/Create_JVM_Plots.robot + suite_list+=<span class="code-quote">' integration/test/csit/suites/integration/Create_JVM_Plots.robot'</span> + eval <span class="code-quote">'SUITES='</span>\<span class="code-quote">''/w/workspace/stano-v2-openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/test/csit/suites/openflowplugin/Performance_Bulkomatic/100K_Flows_100DPN_3Node_Cluster_Perf_Test.robot integration/test/csit/suites/integration/Create_JVM_Plots.robot'</span>\''' ++ SUITES=<span class="code-quote">'/w/workspace/stano-v2-openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/test/csit/suites/openflowplugin/Performance_Bulkomatic/100K_Flows_100DPN_3Node_Cluster_Perf_Test.robot integration/test/csit/suites/integration/Create_JVM_Plots.robot'</span> </pre> </div></div> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> /w/workspace/stanov3-openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/test/csit/suites/openflowplugin/Performance_Bulkomatic/100K_Flows_100DPN_3Node_Cluster_Perf_Test.robot/w/workspace/stanov3-openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/test/csit/suites<span class="code-comment">//w/workspace/stanov3-openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/Create_JVM_Plots.robot </span> </pre> </div></div> <p><a href="https://jenkins.opendaylight.org/sandbox/view/All/job/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/1/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/view/All/job/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/1/</a></p> <p>This job succeeded.</p> <p>Success:</p> <p><a href="https://jenkins.opendaylight.org/sandbox/view/All/job/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/1/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/view/All/job/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/1/</a></p> <p> </p> <p>Yet testing:</p> <p><a href="https://jenkins.opendaylight.org/sandbox/view/All/job/openflowplugin-csit-1node-longevity-only-neon/1/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/view/All/job/openflowplugin-csit-1node-longevity-only-neon/1/</a></p> <p><a href="https://jenkins.opendaylight.org/sandbox/view/All/job/controller-csit-1node-notifications-longevity-only-neon/1/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/view/All/job/controller-csit-1node-notifications-longevity-only-neon/1/</a></p> <p> </p> <p>the plots didn't get created:<br/> <a href="https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/1/robot-plugin/log.html.gz#s1-s2-t1-k1-k1-k1-k1-k5" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/1/robot-plugin/log.html.gz#s1-s2-t1-k1-k1-k1-k1-k5</a></p> <p>JMX collector does not work in Magnesium: <a href="https://jira.opendaylight.org/projects/INTTEST/issues/INTTEST-62" class="external-link" rel="nofollow">https://jira.opendaylight.org/projects/INTTEST/issues/INTTEST-62</a></p> <p>it seems that it does not work also for neon</p> <p><a href="https://jenkins.opendaylight.org/sandbox/view/All/job/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-neon/1/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/view/All/job/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-neon/1/</a></p> <p>There is also need ti update documentation.</p> <p>Should I do it?</p> <p>Probably these two:</p> <p><a href="https://wiki.opendaylight.org/view/CrossProject:Integration_Group:Adding_JVM_plots_to_Jobs" class="external-link" target="_blank" rel="nofollow noopener">https://wiki.opendaylight.org/view/CrossProject:Integration_Group:Adding_JVM_plots_to_Jobs</a></p> <p><a href="https://docs.opendaylight.org/projects/integration-test/en/latest/system-test-guide.html" class="external-link" target="_blank" rel="nofollow noopener">https://docs.opendaylight.org/projects/integration-test/en/latest/system-test-guide.html</a></p> <p> </p> <p>I would just add a note on the first link saying the JVM plots are automatically added to all CSIT jobs so no need to follow this instruction.</p> <p>The second link does not go in detail to all files and logs generated by a CSIT job, so for that matter you can either leave as it is or add a note saying JVM plots will be automatically generated.</p> INTTEST-63 Development Rank 0|i01ls7: