[#INTTEST-14] Use JVM monitoring in each test job

Comment by Vratko Polak [ 29/May/17 ]

> high utilization

That was referring to Java Heap utilization, but other metrics can be useful as well, for example CPU utilization.

Comment by Luis Gomez [ 25/Jul/18 ]

For now this works consists on incorporating JVM plots instruction below:

https://wiki.opendaylight.org/view/CrossProject:Integration_Group:Adding_JVM_plots_to_Jobs

into the process to generate a CSIT job:

https://docs.opendaylight.org/projects/integration-test/en/latest/system-test-guide.html

Comment by Stanislav Chlebec [ 17/Oct/19 ]

Hi Luis
I was assigned to this project by my company. It seems to me that I could to add JVM monitoring to some tests (I would like to try - I am a begginer in this project). Could you please give me an advice with what test I could begin?

Thanks

Comment by Luis Gomez [ 18/Oct/19 ]

Hi Stanislav,

Thanks for jumping on this. It seems like the framework is not working now. If you see last test case here:

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/openflowplugin-csit-1node-perf-bulkomatic-only-neon/403/robot-plugin/log.html.gz#s1-s1

The python robot KW [1] throws an error: IndexError: list index out of range

So I will have to dig a bit what is going on before we can extend the framework to other jobs.

[1] https://git.opendaylight.org/gerrit/gitweb?p=integration/test.git;a=blob;f=csit/libraries/Appenders/ElasticsearchAppender.py;h=52cbe746c0a73a4f313674c55998153fdb0bc79e;hb=HEAD#l106

Comment by Luis Gomez [ 18/Oct/19 ]

There is an issue with decanter jmx collector but I figured out a workaround. See https://jira.opendaylight.org/browse/INTTEST-60.

Comment by Luis Gomez [ 18/Oct/19 ]

Once this is fixed we can resume this ticket work.

Comment by Luis Gomez [ 21/Oct/19 ]

OK, fix is merged, you can try to add the JVM for an individual job by just following this wiki:

https://wiki.opendaylight.org/view/CrossProject:Integration_Group:Adding_JVM_plots_to_Jobs

If the goal is to add the JVM plots to all the jobs, I will have to take a look at how to do that.

Comment by Stanislav Chlebec [ 22/Oct/19 ]

Maybe the best place where we could add decanter features into the install-features: clause is in templates?

Then probably these features are going to be installed in all jobs which are built with that template

Could you please confirm my thinking?

Thanks

edited:

... or simply to add to all yaml files with definition of job to the install-features: clause
(in the way how it was done to this moment)

What is better?

Comment by Stanislav Chlebec [ 22/Oct/19 ]

List of files with templates

 jjb/autorelease/autorelease-templates.yaml
 jjb/global-jjb/jjb/lf-c-cpp-jobs.yaml
 jjb/global-jjb/jjb/lf-ci-jobs.yaml
 jjb/global-jjb/jjb/lf-docker-jobs.yaml
 jjb/global-jjb/jjb/lf-info-vote.yaml
 jjb/global-jjb/jjb/lf-maven-jobs.yaml
 jjb/global-jjb/jjb/lf-node-jobs.yaml
 jjb/global-jjb/jjb/lf-python-jobs.yaml
 jjb/global-jjb/jjb/lf-release-jobs.yaml
 jjb/global-jjb/jjb/lf-release-management-jobs.yaml
 jjb/global-jjb/jjb/lf-rtd-jobs.yaml
 jjb/global-jjb/jjb/lf-whitesource-jobs.yaml
 jjb/integration/distribution/distribution-templates.yaml
 jjb/integration/integration-templates.yaml
 jjb/integration/integration-test-jobs.yaml
 jjb/local/netvirt/netvirt-apex-images.yaml
 jjb/local/netvirt/netvirt-dependencies.yaml
 jjb/local/netvirt/netvirt-job-reports.yaml
 jjb/local/openci/openci-odl-daily-jobs.yaml
 jjb/local/opflex/opflex-custom.yaml
 jjb/local/opflex/opflex-thirdparty-dependencies.yaml
 jjb/local/opflex/opflex.yaml
 jjb/local/ovsdb/ovsdb-full-integration.yaml
 jjb/local/ovsdb/ovsdb-sonar.yaml
 jjb/local/packaging/packaging-ansible.yaml
 jjb/local/packaging/packaging-puppet.yaml
 jjb/local/packaging/packaging.yaml
 jjb/local/releng-jobs.yaml
 jjb/local/releng-templates-java.yaml

Comment by Stanislav Chlebec [ 22/Oct/19 ]

List of files with project

jjb/aaa/aaa-csit-authn.yaml
jjb/aaa/aaa-patch-test.yaml
jjb/aaa/aaa.yaml
jjb/alt-datastores/alt-datastores.yaml
jjb/alto/alto-csit-setup.yaml
jjb/alto/alto.yaml
jjb/ansible/ansible.yaml
jjb/archetypes/archetypes.yaml
jjb/autorelease/autorelease-projects.yaml
jjb/autorelease/autorelease-version-management.yaml
jjb/autorelease/validate-autorelease-neon.yaml
jjb/autorelease/validate-autorelease-sodium.yaml
jjb/bgpcep/bgpcep-csit-bgpclustering-ha.yaml
jjb/bgpcep/bgpcep-csit-bgpclustering-longevity.yaml
jjb/bgpcep/bgpcep-csit-bgpclustering.yaml
jjb/bgpcep/bgpcep-csit-bgp-ingest-mixed.yaml
jjb/bgpcep/bgpcep-csit-bgp-ingest.yaml
jjb/bgpcep/bgpcep-csit-throughpcep.yaml
jjb/bgpcep/bgpcep-csit-userfeatures.yaml
jjb/bgpcep/bgpcep-patch-test.yaml
jjb/bgpcep/bgpcep.yaml
jjb/bier/bier-csit-basic.yaml
jjb/bier/bier.yaml
jjb/coe/coe.yaml
jjb/controller/controller-csit-akka1.yaml
jjb/controller/controller-csit-clustering.yaml
jjb/controller/controller-csit-cs-chasing-leader.yaml
jjb/controller/controller-csit-cs-partnheal.yaml
jjb/controller/controller-csit-ddb-expl-lead-movement.yaml
jjb/controller/controller-csit-dnb-noloss.yaml
jjb/controller/controller-csit-drb-partnheal.yaml
jjb/controller/controller-csit-drb-precedence.yaml
jjb/controller/controller-csit-dsbenchmark-cluster.yaml
jjb/controller/controller-csit-dsbenchmark.yaml
jjb/controller/controller-csit-rest-cars-perf.yaml
jjb/controller/controller-csit-rest-clust-cars-perf.yaml
jjb/controller/controller-patch-test.yaml
jjb/controller/controller.yaml
jjb/daexim/daexim-csit-basic.yaml
jjb/daexim/daexim-csit-clustering-basic.yaml
jjb/daexim/daexim.yaml
jjb/detnet/detnet.yaml
jjb/dluxapps/dluxapps-csit-yangman.yaml
jjb/dluxapps/dluxapps.yaml
jjb/dlux/dlux.yaml
jjb/docs/docs-rtd.yaml
jjb/genius/genius-csit-genius.yaml
jjb/genius/genius-patch-test.yaml
jjb/genius/genius.yaml
jjb/groupbasedpolicy/groupbasedpolicy-csit-1node-openstack.yaml
jjb/groupbasedpolicy/groupbasedpolicy-csit-3node.yaml
jjb/groupbasedpolicy/groupbasedpolicy-csit-6node.yaml
jjb/groupbasedpolicy/groupbasedpolicy-csit-clustering.yaml
jjb/groupbasedpolicy/groupbasedpolicy-csit-sxp.yaml
jjb/groupbasedpolicy/groupbasedpolicy.yaml
jjb/honeycomb/vbd/vbd.yaml
jjb/infrautils/infrautils-patch-test.yaml
jjb/infrautils/infrautils.yaml
jjb/integration/distribution/distribution-csit-userfeatures.yaml
jjb/integration/distribution/distribution-jobs.yaml
jjb/integration/integration-test-jobs.yaml
jjb/jsonrpc/jsonrpc-csit-basic.yaml
jjb/jsonrpc/jsonrpc.yaml
jjb/l2switch/l2switch-csit-host-scalability-daily.yaml
jjb/l2switch/l2switch-csit-scalability.yaml
jjb/l2switch/l2switch-csit-switch.yaml
jjb/l2switch/l2switch-patch-test.yaml
jjb/l2switch/l2switch.yaml
jjb/lf-infra/lf-infra-jobs.yaml
jjb/lf-infra/zzz-test-release.yaml
jjb/lfn/lfn.yaml
jjb/lispflowmapping/lispflowmapping-csit-msmr.yaml
jjb/lispflowmapping/lispflowmapping-csit-performance.yaml
jjb/lispflowmapping/lispflowmapping.yaml
jjb/local/mdsal/mdsal-csit-bindingv1.yaml
jjb/local/mdsal/mdsal-patch-test.yaml
jjb/local/mdsal/mdsal.yaml
jjb/local/nemo/nemo-csit-engine.yaml
jjb/local/nemo/nemo.yaml
jjb/local/netconf/netconf-clustering-scale.yaml
jjb/local/netconf/netconf-clustering.yaml
jjb/local/netconf/netconf-csit-callhome.yaml
jjb/local/netconf/netconf-patch-test.yaml
jjb/local/netconf/netconf-scale.yaml
jjb/local/netconf/netconf-userfeatures.yaml
jjb/local/netconf/netconf.yaml
jjb/local/netvirt/netvirt-apex-images.yaml
jjb/local/netvirt/netvirt-csit-hwvtep-openstack.yaml
jjb/local/netvirt/netvirt-csit-multi-apex.yaml
jjb/local/netvirt/netvirt-csit-multi-openstack-upgrade.yaml
jjb/local/netvirt/netvirt-csit-multi-openstack.yaml
jjb/local/netvirt/netvirt-dependencies.yaml
jjb/local/netvirt/netvirt-job-reports.yaml
jjb/local/netvirt/netvirt-patch-test.yaml
jjb/local/netvirt/netvirt.yaml
jjb/local/neutron/neutron-patch-test.yaml
jjb/local/neutron/neutron.yaml
jjb/local/odlparent/odlparent-patch-test.yaml
jjb/local/odlparent/odlparent.yaml
jjb/local/odltools/odltools-csit.yaml
jjb/local/odltools/odltools-patch-test.yaml
jjb/local/odltools/odltools.yaml
jjb/local/of-config/of-config.yaml
jjb/local/openci/openci-odl-daily-jobs.yaml
jjb/local/openflowplugin/openflowplugin-csit-bundle-based-reconciliation.yaml
jjb/local/openflowplugin/openflowplugin-csit-cbench.yaml
jjb/local/openflowplugin/openflowplugin-csit-clustering-bulkomatic.yaml
jjb/local/openflowplugin/openflowplugin-csit-clustering-perf-bulkomatic.yaml
jjb/local/openflowplugin/openflowplugin-csit-clustering.yaml
jjb/local/openflowplugin/openflowplugin-csit-flow-services.yaml
jjb/local/openflowplugin/openflowplugin-csit-longevity.yaml
jjb/local/openflowplugin/openflowplugin-csit-perf-bulkomatic.yaml
jjb/local/openflowplugin/openflowplugin-csit-perf-stats-collection.yaml
jjb/local/openflowplugin/openflowplugin-csit-sanity.yaml
jjb/local/openflowplugin/openflowplugin-csit-scale-link.yaml
jjb/local/openflowplugin/openflowplugin-csit-scale-switch.yaml
jjb/local/openflowplugin/openflowplugin-csit-scale.yaml
jjb/local/openflowplugin/openflowplugin-patch-test.yaml
jjb/local/openflowplugin/openflowplugin.yaml
jjb/local/opflex/opflex-custom.yaml
jjb/local/opflex/opflex-thirdparty-dependencies.yaml
jjb/local/opflex/opflex.yaml
jjb/local/ovsdb/ovsdb-csit-clustering.yaml
jjb/local/ovsdb/ovsdb-csit-southbound.yaml
jjb/local/ovsdb/ovsdb-full-integration.yaml
jjb/local/ovsdb/ovsdb-patch-test.yaml
jjb/local/ovsdb/ovsdb-sonar.yaml
jjb/local/ovsdb/ovsdb.yaml
jjb/local/p4plugin/p4plugin.yaml
jjb/local/packaging/packaging-ansible.yaml
jjb/local/packaging/packaging-puppet.yaml
jjb/local/packaging/packaging.yaml
jjb/local/packetcable/packetcable-csit-pcmm.yaml
jjb/local/packetcable/packetcable.yaml
jjb/local/releng-jobs.yaml
jjb/local/releng-packer-jobs.yaml
jjb/local/releng-views.yaml
jjb/local/serviceutils/serviceutils-patch-test.yaml
jjb/local/serviceutils/serviceutils.yaml
jjb/local/sfc/sfc-csit-full-deploy.yaml
jjb/local/sfc/sfc-csit-rest-basic.yaml
jjb/local/sfc/sfc-patch-test.yaml
jjb/local/sfc/sfc-python.yaml
jjb/local/sfc/sfc.yaml
jjb/local/snmp4sdn/snmp4sdn.yaml
jjb/local/snmp/snmp.yaml
jjb/local/sxp/sxp-csit-basic.yaml
jjb/local/sxp/sxp-csit-binding-origin.yaml
jjb/local/sxp/sxp-csit-cluster-binding.yaml
jjb/local/sxp/sxp-csit-clustering.yaml
jjb/local/sxp/sxp-csit-filtering.yaml
jjb/local/sxp/sxp-csit-performance.yaml
jjb/local/sxp/sxp-csit-topology.yaml
jjb/local/sxp/sxp.yaml
jjb/local/telemetry/telemetry-csit-basic.yaml
jjb/local/telemetry/telemetry.yaml
jjb/local/transportpce/transportpce.yaml
jjb/local/tsdr/tsdr-csit-cassandra.yaml
jjb/local/tsdr/tsdr-csit-elasticsearch.yaml
jjb/local/tsdr/tsdr-csit-hbase.yaml
jjb/local/tsdr/tsdr-csit-hsqldb.yaml
jjb/local/tsdr/tsdr.yaml
jjb/local/unimgr/unimgr.yaml
jjb/local/usc/usc-csit-tcp.yaml
jjb/local/usc/usc.yaml
jjb/local/yangtools/yangtools-csit-system.yaml
jjb/local/yangtools/yangtools-patch-test.yaml
jjb/local/yangtools/yangtools.yaml

Comment by Stanislav Chlebec [ 22/Oct/19 ]

List of files with job-group

jjb/global-jjb/jjb/lf-ci-job-groups.yaml
jjb/global-jjb/jjb/lf-docker-job-groups.yaml
jjb/global-jjb/jjb/lf-maven-job-groups.yaml
jjb/global-jjb/jjb/lf-node-job-groups.yaml
jjb/global-jjb/jjb/lf-python-job-groups.yaml
jjb/global-jjb/jjb/lf-release-job-groups.yaml
jjb/global-jjb/jjb/lf-rtd-job-groups.yaml
jjb/local/releng-templates-java.yaml

Comment by Stanislav Chlebec [ 22/Oct/19 ]

These contain decanter features:

jjb/local/netconf/netconf-scale.yaml
jjb/local/netvirt/netvirt-csit-hwvtep-openstack.yaml
jjb/local/netvirt/netvirt-csit-multi-apex.yaml
jjb/local/netvirt/netvirt-csit-multi-openstack-upgrade.yaml
jjb/local/netvirt/netvirt-csit-multi-openstack.yaml
jjb/local/openflowplugin/openflowplugin-csit-clustering-perf-bulkomatic.yaml
jjb/local/openflowplugin/openflowplugin-csit-longevity.yaml
jjb/local/openflowplugin/openflowplugin-csit-perf-bulkomatic.yaml
jjb/local/openflowplugin/openflowplugin-csit-scale-link.yaml
jjb/local/openflowplugin/openflowplugin-csit-scale-switch.yaml
jjb/local/openflowplugin/openflowplugin-csit-scale.yaml
jjb/local/ovsdb/ovsdb-csit-clustering.yaml

Comment by Luis Gomez [ 22/Oct/19 ]

So if we wanted to add the JVM plot for all jobs, we would have to move:

1) decanter feature installation.
2) decanter & ES configuration & installation script.
3) python call to generate plots.

to the controller install scripts:

https://git.opendaylight.org/gerrit/gitweb?p=releng/builder.git;a=blob;f=jjb/integration/common-functions.sh
https://git.opendaylight.org/gerrit/gitweb?p=releng/builder.git;a=blob;f=jjb/integration/integration-deploy-controller-run-test.sh
https://git.opendaylight.org/gerrit/gitweb?p=releng/builder.git;a=blob;f=jjb/integration/integration-configure-clustering.sh

I will check anyway in the next integration call if anyone has any objection to this work.

Comment by Jamo Luhrsen [ 22/Oct/19 ]

no objection from me.

Comment by Luis Gomez [ 24/Oct/19 ]

stanislav-chlebec, if you want to add the JVM report for all jobs I would suggest to go the route I suggested above: modifying 3 scripts in releng/builder vs changing individual projects settings. Let me know if you have further questions.

Comment by Stanislav Chlebec [ 25/Oct/19 ]

Sure. Thanks.

Comment by Stanislav Chlebec [ 28/Oct/19 ]

ERROR:root:Problem formatting with args:
allow_empty:False
obj: OrderedDict([('name', 'ELASTICSEARCHATTRIBUTE'), ('default', '\{feature-elasticsearch-attribute}'), ('description', 'Configure the period the script should harvest JVM metrics.\nDefault is short.\nPossible values: long (its value is 120000)\n short (its value is 5000)\n any number between 5000 and 120000\n')])
paramdict: OrderedDict([('controller-features', ''), ('controller-debug-map', ''), ('controller-scope', ''), ('controller-use-features-boot', '')])
ERROR:root:Failure formatting component ('string') data 'OrderedDict([('name', 'ELASTICSEARCHATTRIBUTE'), ('default', '\{feature-elasticsearch-attribute}'), ('description', 'Configure the period the script should harvest JVM metrics.\nDefault is short.\nPossible values: long (its value is 120000)\n short (its value is 5000)\n any number between 5000 and 120000\n')])'
Traceback (most recent call last):
File "/tmp/tmp.bg1jmOPwmn/bin/jenkins-jobs", line 11, in <module>
sys.exit(main())
File "/tmp/tmp.bg1jmOPwmn/lib/python2.7/site-packages/jenkins_jobs/cli/entry.py", line 165, in main
jjb.execute()
File "/tmp/tmp.bg1jmOPwmn/lib/python2.7/site-packages/jenkins_jobs/cli/entry.py", line 146, in execute
ext.obj.execute(self.options, self.jjb_config)
File "/tmp/tmp.bg1jmOPwmn/lib/python2.7/site-packages/jenkins_jobs/cli/subcommand/test.py", line 63, in execute
options, jjb_config)
File "/tmp/tmp.bg1jmOPwmn/lib/python2.7/site-packages/jenkins_jobs/cli/subcommand/update.py", line 115, in _generate_xmljobs
xml_jobs = xml_job_generator.generateXML(job_data_list)
File "/tmp/tmp.bg1jmOPwmn/lib/python2.7/site-packages/jenkins_jobs/xml_config.py", line 82, in generateXML
xml_objs.append(self._getXMLForData(data))
File "/tmp/tmp.bg1jmOPwmn/lib/python2.7/site-packages/jenkins_jobs/xml_config.py", line 94, in _getXMLForData
self._gen_xml(xml, data)
File "/tmp/tmp.bg1jmOPwmn/lib/python2.7/site-packages/jenkins_jobs/xml_config.py", line 108, in _gen_xml
module.gen_xml(xml, data)
File "/tmp/tmp.bg1jmOPwmn/lib/python2.7/site-packages/jenkins_jobs/modules/parameters.py", line 1070, in gen_xml
self.registry.dispatch('parameter', pdefs, param)
File "/tmp/tmp.bg1jmOPwmn/lib/python2.7/site-packages/jenkins_jobs/registry.py", line 253, in dispatch
self.dispatch(component_type, xml_parent, b, component_data)
File "/tmp/tmp.bg1jmOPwmn/lib/python2.7/site-packages/jenkins_jobs/registry.py", line 171, in dispatch
self.jjb_config.yamlparser['allow_empty_variables'])
File "/tmp/tmp.bg1jmOPwmn/lib/python2.7/site-packages/jenkins_jobs/formatter.py", line 59, in deep_format
deep_format(obj[item], paramdict, allow_empty)
File "/tmp/tmp.bg1jmOPwmn/lib/python2.7/site-packages/jenkins_jobs/formatter.py", line 43, in deep_format
raise JenkinsJobsException(desc)
jenkins_jobs.errors.JenkinsJobsException: feature-elasticsearch-attribute parameter missing to format {feature-elasticsearch-attribute}
Given:
OrderedDict([('controller-features', ''), ('controller-debug-map', ''), ('controller-scope', ''), ('controller-use-features-boot', '')])

Comment by Stanislav Chlebec [ 28/Oct/19 ]

jjb/integration/integration-macros.yaml
39:121 error line too long (153 > 120 characters) (line-length)

Comment by Stanislav Chlebec [ 28/Oct/19 ]

Test shell scripts with shellcheck.......................................Failed
hookid: shellcheck

In jjb/integration/common-functions.sh line 284:
if [ ${type} == "script" ]; then
^----- SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
if [ "${type}" == "script" ]; then

In jjb/integration/common-functions.sh line 288:
if [ ${ELASTICSEARCHATTRIBUTE} == "short" ]; then
^{-----------------------} SC2086: Double quote to prevent globbing and word splitting.

Did you mean:
if [ "${ELASTICSEARCHATTRIBUTE}" == "short" ]; then

For more information:
https://www.shellcheck.net/wiki/SC2086 – Double quote to prevent globbing ...

yamllint.................................................................Passed
ERROR: InvocationError for command /w/workspace/builder-tox-verify-master/.tox/pre-commit/bin/pre-commit run --all-files (exited with code 1)

Comment by Stanislav Chlebec [ 29/Oct/19 ]

I made an attempt to run a job

https://jenkins.opendaylight.org/releng/job/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/

which was generated from my repo "releng/builder" (where are my changes - commit (jjb) stanislav@ArmDev:~/work/odl/builder$ git log -1
commit 13139b76183f9302ff4f296795cf8e58062e3724
Author: Stanislav Chlebec <stanislav.chlebec@pantheon.tech>
Date: Fri Oct 25 19:33:42 2019 +0200)

While trying to test this jenkins job with changes from integration/test repo I provided the value to the parameter GERRIT_REFSPEC

refs/changes/94/85394/2

I am not sure if my approach was right because job failed and in the log I found
16:06:04 > git checkout -f ecfaf49c97b4b7df7050ad5740cc160d5b676af5*16:06:05* Commit message: "~~INTTEST-61~~ lispflowmapping failing"16:06:05 > git rev-parse FETCH_HEAD^{commit} # timeout=10*16:06:05* > git rev-list --no-walk fbd8d762c1fb6e6e253bec4075502cbbc4587eb0 # timeout=10
I do not see the reference to my patch so I wonder what I did wrong

Please could you help me and to instruct me how to test?

Thanks

Comment by Jamo Luhrsen [ 29/Oct/19 ]

looks like you have the right workflow here. The problem I see in your sandbox job is something broken with common-functions.sh. Search for "07:26:03 Executing cat..." and you'll see some syntax errors. That's where
to focus at this point.

Comment by Luis Gomez [ 29/Oct/19 ]

The issue I see in last sandbox run is JVM scripts are run twice. To avoid that pick another job that does not have JVM yet or use PATCHREFSPEC=refs/changes/94/85394/2 (not GERRIT_REFSPEC)

Comment by Stanislav Chlebec [ 30/Oct/19 ]

Thanks for comments.

https://jenkins.opendaylight.org/sandbox/job/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/6/console

I set PATCHREFSPEC=refs/changes/94/85394/2 as Luis recommneded

I then compared the console output to https://jenkins.opendaylight.org/releng/job/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/54/parameters/

According this successfull job I adjusted the parameter BUNDLE_URL to https://nexus.opendaylight.org/content/repositories//autorelease-3465/org/opendaylight/integration/karaf/0.12.0/karaf-0.12.0.zip

In spite of these tweaks I got failure:
08:39:43 postplan exists!!!08:39:43 Changing the postplan path...08:39:43 # Place the commands in run order:08:39:43 python3 /w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/test/csit/scripts/push_test_data.py a4ff38b99ef2c7626450543021b4c134.us-east-1.aws.found.io:9243*08:39:43* Executing python3 /w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/test/csit/scripts/push_test_data.py a4ff38b99ef2c7626450543021b4c134.us-east-1.aws.found.io:9243...08:39:44 Traceback (most recent call last):08:39:44 File "/w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/test/csit/scripts/push_test_data.py", line 93, in <module>08:39:44 BODY = data_gen.generate()08:39:44 File "/w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/test/csit/scripts/data_generate.py", line 49, in generate*08:39:44* tree = ET.parse(robot_log)08:39:44 File "/usr/lib64/python3.6/xml/etree/ElementTree.py", line 1196, in parse*08:39:44* tree.parse(source, parser)08:39:44 File "/usr/lib64/python3.6/xml/etree/ElementTree.py", line 586, in parse*08:39:44* source = open(source, "rb")08:39:44 FileNotFoundError: [Errno 2] No such file or directory: '/w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/output.xml'

Again I do not know what can be a reason. Please do you have some advice for me?

Comment by Jamo Luhrsen [ 30/Oct/19 ]

I spent some time digging through your patches and job. Like I wrote above, the abort in your sandbox
job is coming from something in common-functions. Did you see this?

00:39:06 Executing /w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/test/csit/scripts/controllermem_6g.sh...
00:39:06 Executing cat...
00:39:06 /tmp/common-functions.sh: line 327: source: /usr/bin/cat: cannot execute binary file
00:39:06 Executing >...
00:39:06 /tmp/common-functions.sh: line 327: >: No such file or directory
00:39:06 Executing ${WORKSPACE}/org.apache.karaf.decanter.scheduler.simple.cfg...
00:39:06 /tmp/common-functions.sh: line 327: ${WORKSPACE}/org.apache.karaf.decanter.scheduler.simple.cfg: No such file or directory
00:39:06 Executing <<EOF...
00:39:06 /tmp/common-functions.sh: line 327: <<EOF: No such file or directory
00:39:06 Executing period=5000...
00:39:06 /tmp/common-functions.sh: line 327: period=5000: No such file or directory
00:39:06 Executing EOF...
00:39:06 /tmp/common-functions.sh: line 327: EOF: No such file or directory
00:39:06 Executing echo...
00:39:06 /tmp/common-functions.sh: line 327: source: /usr/bin/echo: cannot execute binary file
00:39:06 Executing "Copying...
00:39:06 /tmp/common-functions.sh: line 327: "Copying: No such file or directory
00:39:06 Executing config...
00:39:06 /tmp/common-functions.sh: line 327: config: No such file or directory
00:39:06 Executing files...
00:39:06 /tmp/common-functions.sh: line 327: files: No such file or directory
00:39:06 Executing to...
00:39:06 /tmp/common-functions.sh: line 327: to: No such file or directory
00:39:06 Executing ODL...
00:39:06 /tmp/common-functions.sh: line 327: ODL: No such file or directory
00:39:06 Executing Controller...
00:39:06 /tmp/common-functions.sh: line 327: Controller: No such file or directory
00:39:06 Executing folder"...
00:39:06 /tmp/common-functions.sh: line 327: folder": No such file or directory
00:39:06 Executing for...
00:39:06 /tmp/common-functions.sh: line 327: for: No such file or directory
00:39:06 Executing i...
00:39:06 /tmp/common-functions.sh: line 327: i: No such file or directory
00:39:06 Executing in...
00:39:06 /tmp/common-functions.sh: line 327: in: No such file or directory
00:39:06 Executing `seq...
00:39:06 /tmp/common-functions.sh: line 327: `seq: No such file or directory
00:39:06 Executing 1...
00:39:06 /tmp/common-functions.sh: line 327: 1: No such file or directory
00:39:06 Executing ${NUM_ODL_SYSTEM}`...
00:39:06 /tmp/common-functions.sh: line 327: ${NUM_ODL_SYSTEM}`: No such file or directory
00:39:06 Executing do...
00:39:06 /tmp/common-functions.sh: line 327: do: No such file or directory
00:39:06 Executing CONTROLLERIP=ODL_SYSTEM_${i}_IP...
00:39:06 /tmp/common-functions.sh: line 327: CONTROLLERIP=ODL_SYSTEM_${i}_IP: No such file or directory
00:39:06 Executing echo...
00:39:06 /tmp/common-functions.sh: line 327: source: /usr/bin/echo: cannot execute binary file
00:39:06 Executing "Setup...
00:39:06 /tmp/common-functions.sh: line 327: "Setup: No such file or directory
00:39:06 Executing long...
00:39:06 /tmp/common-functions.sh: line 327: long: No such file or directory
00:39:06 Executing duration...
00:39:06 /tmp/common-functions.sh: line 327: duration: No such file or directory
00:39:06 Executing config...
00:39:06 /tmp/common-functions.sh: line 327: config: No such file or directory
00:39:06 Executing to...
00:39:06 /tmp/common-functions.sh: line 327: to: No such file or directory
00:39:06 Executing ${!CONTROLLERIP}"...
00:39:06 /tmp/common-functions.sh: line 327: ${!CONTROLLERIP}": No such file or directory
00:39:06 Executing ssh...
00:39:06 /tmp/common-functions.sh: line 327: source: /usr/bin/ssh: cannot execute binary file
00:39:06 Executing ${!CONTROLLERIP}...
00:39:06 /tmp/common-functions.sh: line 327: ${!CONTROLLERIP}: No such file or directory
00:39:06 Executing "mkdir...
00:39:06 /tmp/common-functions.sh: line 327: "mkdir: No such file or directory
00:39:06 Executing -p...
00:39:06 /tmp/common-functions.sh: line 327: source: -p: invalid option
00:39:06 source: usage: source filename [arguments]
00:39:06 Executing /tmp/${BUNDLEFOLDER}/etc/opendaylight/karaf/"...
00:39:06 /tmp/common-functions.sh: line 327: /tmp/${BUNDLEFOLDER}/etc/opendaylight/karaf/": No such file or directory
00:39:06 Executing scp...
00:39:06 /tmp/common-functions.sh: line 327: source: /usr/bin/scp: cannot execute binary file
00:39:06 Executing ${WORKSPACE}/org.apache.karaf.decanter.scheduler.simple.cfg...
00:39:06 /tmp/common-functions.sh: line 327: ${WORKSPACE}/org.apache.karaf.decanter.scheduler.simple.cfg: No such file or directory
00:39:06 Executing ${!CONTROLLERIP}:/tmp/${BUNDLEFOLDER}/etc/...
00:39:06 /tmp/common-functions.sh: line 327: ${!CONTROLLERIP}:/tmp/${BUNDLEFOLDER}/etc/: No such file or directory
00:39:06 Executing done...
00:39:06 /tmp/common-functions.sh: line 327: done: No such file or directory
00:39:06 Executing cat...
00:39:06 /tmp/common-functions.sh: line 327: source: /usr/bin/cat: cannot execute binary file
00:39:06 Executing >...
00:39:06 /tmp/common-functions.sh: line 327: >: No such file or directory
00:39:06 Executing ${WORKSPACE}/org.apache.karaf.decanter.collector.jmx-local.cfg...
00:39:06 /tmp/common-functions.sh: line 327: ${WORKSPACE}/org.apache.karaf.decanter.collector.jmx-local.cfg: No such file or directory
00:39:06 Executing <<EOF...
00:39:06 /tmp/common-functions.sh: line 327: <<EOF: No such file or directory
00:39:06 Executing type=jmx-local...
00:39:06 /tmp/common-functions.sh: line 327: type=jmx-local: No such file or directory
00:39:06 Executing url=local...
00:39:06 /tmp/common-functions.sh: line 327: url=local: No such file or directory
00:39:06 Executing object.name=java.lang:type=*,name=*...
00:39:06 /tmp/common-functions.sh: line 327: object.name=java.lang:type=*,name=*: No such file or directory
00:39:06 Executing EOF...
00:39:06 /tmp/common-functions.sh: line 327: EOF: No such file or directory
00:39:06 Executing cat...
00:39:06 /tmp/common-functions.sh: line 327: source: /usr/bin/cat: cannot execute binary file
00:39:06 Executing >...
00:39:06 /tmp/common-functions.sh: line 327: >: No such file or directory
00:39:06 Executing ${WORKSPACE}/org.apache.karaf.decanter.collector.jmx-others.cfg...
00:39:06 /tmp/common-functions.sh: line 327: ${WORKSPACE}/org.apache.karaf.decanter.collector.jmx-others.cfg: No such file or directory
00:39:06 Executing <<EOF...
00:39:06 /tmp/common-functions.sh: line 327: <<EOF: No such file or directory
00:39:06 Executing type=jmx-local...
00:39:06 /tmp/common-functions.sh: line 327: type=jmx-local: No such file or directory
00:39:06 Executing url=local...
00:39:06 /tmp/common-functions.sh: line 327: url=local: No such file or directory
00:39:06 Executing object.name=java.lang:type=*...
00:39:06 /tmp/common-functions.sh: line 327: object.name=java.lang:type=*: No such file or directory
00:39:06 Executing EOF...
00:39:06 /tmp/common-functions.sh: line 327: EOF: No such file or directory
00:39:06 Executing for...
00:39:06 /tmp/common-functions.sh: line 327: for: No such file or directory
00:39:06 Executing i...
00:39:06 /tmp/common-functions.sh: line 327: i: No such file or directory
00:39:06 Executing in...
00:39:06 /tmp/common-functions.sh: line 327: in: No such file or directory
00:39:06 Executing `seq...
00:39:06 /tmp/common-functions.sh: line 327: `seq: No such file or directory
00:39:06 Executing 1...
00:39:06 /tmp/common-functions.sh: line 327: 1: No such file or directory
00:39:06 Executing ${NUM_ODL_SYSTEM}`...
00:39:06 /tmp/common-functions.sh: line 327: ${NUM_ODL_SYSTEM}`: No such file or directory
00:39:06 Executing do...
00:39:06 /tmp/common-functions.sh: line 327: do: No such file or directory
00:39:06 Executing CONTROLLERIP=ODL_SYSTEM_${i}_IP...
00:39:06 /tmp/common-functions.sh: line 327: CONTROLLERIP=ODL_SYSTEM_${i}_IP: No such file or directory
00:39:06 Executing CLUSTERNAME=`head...
00:39:06 /tmp/common-functions.sh: line 327: CLUSTERNAME=`head: No such file or directory
00:39:06 Executing /dev/urandom...
00:39:42 /tmp/common-functions.sh: xrealloc: cannot allocate 18446744071562068032 bytes (1196032 bytes allocated)
00:39:42 Build step 'Execute shell' marked build as failure

Looking closer, I think you are trying to modify some file org.apache.karaf.decanter.scheduler.simple.cfg but that
file comes from the controller once it's deployed right? I'm not sure, but you can check.

we have three types of "plans" for making configurations. Scriptplans, Configplans and Postplans.
I think you are doing your work from a scriptplan which might not work for this. Maybe you need to use the configplan?

Comment by Luis Gomez [ 31/Oct/19 ]

There are 3 issues AFAICT:

1) Decanter collector does not work in master: https://jira.opendaylight.org/projects/INTTEST/issues/INTTEST-62
2) JVM plots do not work with cluster because cluster deploy scripts are not using common functions: https://git.opendaylight.org/gerrit/#/c/releng/builder/+/85445/
3) The proposed script has some issues, I left some comments in the patch: https://git.opendaylight.org/gerrit/#/c/releng/builder/+/85398/

Comment by Stanislav Chlebec [ 05/Nov/19 ]

+ echo '#################################################'
#################################################
+ echo '##         Verify Cluster is UP                ##'
##         Verify Cluster is UP                ##
+ echo '#################################################'
#################################################
+ cat
++ seq 1 3
+ for i in '$(seq 1 "${NUM_ODL_SYSTEM}")'
+ CONTROLLERIP=ODL_SYSTEM_1_IP
+ echo 'Verifying member-1 with IP address 10.30.170.135 is UP'
Verifying member-1 with IP address 10.30.170.135 is UP
+ scp /w/workspace/stano-v2-openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/verify-cluster-is-up.sh 10.30.170.135:/tmp
Warning: Permanently added '10.30.170.135' (ECDSA) to the list of known hosts.
+ ssh 10.30.170.135 'bash /tmp/verify-cluster-is-up.sh 1 10.30.170.135'
Warning: Permanently added '10.30.170.135' (ECDSA) to the list of known hosts.
Waiting for controller to come up...
000 http://localhost:8181/restconf/modules
000 http://localhost:8181/jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore
000 http://localhost:8181/restconf/modules
000 http://localhost:8181/jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore
000 http://localhost:8181/restconf/modules
000 http://localhost:8181/jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore
000 http://localhost:8181/restconf/modules
000 http://localhost:8181/jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore
000 http://localhost:8181/restconf/modules
000 http://localhost:8181/jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore
already waited 5 seconds...
...
...
...
401 http://localhost:8181/restconf/modules
<html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 401 Unauthorized</title> </head> <body><h2>HTTP ERROR 401</h2> <p>Problem accessing /jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore. Reason: <pre> Unauthorized</pre></p> </body> </html> 401 http://localhost:8181/jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore
401 http://localhost:8181/restconf/modules
<html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 401 Unauthorized</title> </head> <body><h2>HTTP ERROR 401</h2> <p>Problem accessing /jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore. Reason: <pre> Unauthorized</pre></p> </body> </html> 401 http://localhost:8181/jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore
401 http://localhost:8181/restconf/modules
<html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 401 Unauthorized</title> </head> <body><h2>HTTP ERROR 401</h2> <p>Problem accessing /jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore. Reason: <pre> Unauthorized</pre></p> </body> </html> 401 http://localhost:8181/jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore
401 http://localhost:8181/restconf/modules
<html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 401 Unauthorized</title> </head> <body><h2>HTTP ERROR 401</h2> <p>Problem accessing /jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore. Reason: <pre> Unauthorized</pre></p> </body> </html> 401 http://localhost:8181/jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore
404 http://localhost:8181/restconf/modules
{"request":{"mbean":"org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore","type":"read"},"value":{"ReadWriteTransactionCount":0,"SnapshotIndex":1,"InMemoryJournalLogSize":1,"ReplicatedToAllIndex":1,"Leader":"member-3-shard-inventory-config","LastIndex":2,"RaftState":"Follower","LastApplied":2,"LastCommittedTransactionTime":"1970-01-01 00:00:00.000","PeerAddresses":"member-2-shard-inventory-config: akka.tcp:\/\/opendaylight-cluster-data@10.30.170.171:2550\/user\/shardmanager-config\/member-2-shard-inventory-config, member-3-shard-inventory-config: akka.tcp:\/\/opendaylight-cluster-data@10.30.170.84:2550\/user\/shardmanager-config\/member-3-shard-inventory-config","LastLogIndex":2,"LastLeadershipChangeTime":"2019-11-05 10:14:06.554","FollowerInitialSyncStatus":true,"FollowerInfo":[],"FailedReadTransactionsCount":0,"Voting":true,"StatRetrievalTime":"1.421 ms","CurrentTerm":2,"LastTerm":2,"FailedTransactionsCount":0,"PendingTxCommitQueueSize":0,"VotedFor":"member-3-shard-inventory-config","SnapshotCaptureInitiated":false,"CommittedTransactionsCount":0,"TxCohortCacheSize":0,"PeerVotingStates":"member-2-shard-inventory-config: true, member-3-shard-inventory-config: true","LastLogTerm":2,"StatRetrievalError":null,"CommitIndex":2,"SnapshotTerm":2,"AbortTransactionsCount":0,"ReadOnlyTransactionCount":0,"ShardName":"member-1-shard-inventory-config","LeadershipChangeCount":1,"InMemoryJournalDataSize":33},"timestamp":1572948874,"status":200}200 http://localhost:8181/jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore
already waited 70 seconds...
200 http://localhost:8181/restconf/modules
{"request":{"mbean":"org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore","type":"read"},"value":{"ReadWriteTransactionCount":0,"SnapshotIndex":1,"InMemoryJournalLogSize":1,"ReplicatedToAllIndex":1,"Leader":"member-3-shard-inventory-config","LastIndex":2,"RaftState":"Follower","LastApplied":2,"LastCommittedTransactionTime":"1970-01-01 00:00:00.000","PeerAddresses":"member-2-shard-inventory-config: akka.tcp:\/\/opendaylight-cluster-data@10.30.170.171:2550\/user\/shardmanager-config\/member-2-shard-inventory-config, member-3-shard-inventory-config: akka.tcp:\/\/opendaylight-cluster-data@10.30.170.84:2550\/user\/shardmanager-config\/member-3-shard-inventory-config","LastLogIndex":2,"LastLeadershipChangeTime":"2019-11-05 10:14:06.554","FollowerInitialSyncStatus":true,"FollowerInfo":[],"FailedReadTransactionsCount":0,"Voting":true,"StatRetrievalTime":"1.421 ms","CurrentTerm":2,"LastTerm":2,"FailedTransactionsCount":0,"PendingTxCommitQueueSize":0,"VotedFor":"member-3-shard-inventory-config","SnapshotCaptureInitiated":false,"CommittedTransactionsCount":0,"TxCohortCacheSize":0,"PeerVotingStates":"member-2-shard-inventory-config: true, member-3-shard-inventory-config: true","LastLogTerm":2,"StatRetrievalError":null,"CommitIndex":2,"SnapshotTerm":2,"AbortTransactionsCount":0,"ReadOnlyTransactionCount":0,"ShardName":"member-1-shard-inventory-config","LeadershipChangeCount":1,"InMemoryJournalDataSize":33},"timestamp":1572948875,"status":200}200 http://localhost:8181/jolokia/read/org.opendaylight.controller:Category=Shards,name=member-1-shard-inventory-config,type=DistributedConfigDatastore
Controller is UP

https://jenkins.opendaylight.org/sandbox/view/All/job/stano-v2-openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/1/

Comment by Stanislav Chlebec [ 05/Nov/19 ]

https://jenkins.opendaylight.org/releng/view/openflowplugin/job/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/57/console

+ suite_list='/w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/test/csit/suites/openflowplugin/Performance_Bulkomatic/100K_Flows_100DPN_3Node_Cluster_Perf_Test.robot /w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/test/csit/suites/integration/Create_JVM_Plots.robot '
+ eval 'SUITES='\''/w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/test/csit/suites/openflowplugin/Performance_Bulkomatic/100K_Flows_100DPN_3Node_Cluster_Perf_Test.robot /w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/test/csit/suites/integration/Create_JVM_Plots.robot '\'''
++ SUITES='/w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/test/csit/suites/openflowplugin/Performance_Bulkomatic/100K_Flows_100DPN_3Node_Cluster_Perf_Test.robot /w/workspace/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/test/csit/suites/integration/Create_JVM_Plots.robot '

https://jenkins.opendaylight.org/sandbox/view/All/job/stano-v2-openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/1/

+ suite_list='/w/workspace/stano-v2-openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/test/csit/suites/openflowplugin/Performance_Bulkomatic/100K_Flows_100DPN_3Node_Cluster_Perf_Test.robot '
+ add_jvm_plots=integration/test/csit/suites/integration/Create_JVM_Plots.robot
+ suite_list+=' integration/test/csit/suites/integration/Create_JVM_Plots.robot'
+ eval 'SUITES='\''/w/workspace/stano-v2-openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/test/csit/suites/openflowplugin/Performance_Bulkomatic/100K_Flows_100DPN_3Node_Cluster_Perf_Test.robot  integration/test/csit/suites/integration/Create_JVM_Plots.robot'\'''
++ SUITES='/w/workspace/stano-v2-openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/test/csit/suites/openflowplugin/Performance_Bulkomatic/100K_Flows_100DPN_3Node_Cluster_Perf_Test.robot  integration/test/csit/suites/integration/Create_JVM_Plots.robot'

Comment by Stanislav Chlebec [ 05/Nov/19 ]

/w/workspace/stanov3-openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/test/csit/suites/openflowplugin/Performance_Bulkomatic/100K_Flows_100DPN_3Node_Cluster_Perf_Test.robot/w/workspace/stanov3-openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/test/csit/suites//w/workspace/stanov3-openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/Create_JVM_Plots.robot

Comment by Stanislav Chlebec [ 05/Nov/19 ]

https://jenkins.opendaylight.org/sandbox/view/All/job/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/1/

This job succeeded.

Comment by Stanislav Chlebec [ 05/Nov/19 ]

Success:

https://jenkins.opendaylight.org/sandbox/view/All/job/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/1/

Yet testing:

https://jenkins.opendaylight.org/sandbox/view/All/job/openflowplugin-csit-1node-longevity-only-neon/1/

https://jenkins.opendaylight.org/sandbox/view/All/job/controller-csit-1node-notifications-longevity-only-neon/1/

Comment by Jamo Luhrsen [ 05/Nov/19 ]

the plots didn't get created:
https://logs.opendaylight.org/sandbox/vex-yul-odl-jenkins-2/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-magnesium/1/robot-plugin/log.html.gz#s1-s2-t1-k1-k1-k1-k1-k5

Comment by Luis Gomez [ 05/Nov/19 ]

JMX collector does not work in Magnesium: https://jira.opendaylight.org/projects/INTTEST/issues/INTTEST-62

Comment by Stanislav Chlebec [ 06/Nov/19 ]

it seems that it does not work also for neon

https://jenkins.opendaylight.org/sandbox/view/All/job/openflowplugin-csit-3node-clustering-perf-bulkomatic-only-neon/1/

Comment by Stanislav Chlebec [ 12/Nov/19 ]

There is also need ti update documentation.

Should I do it?

Probably these two:

https://wiki.opendaylight.org/view/CrossProject:Integration_Group:Adding_JVM_plots_to_Jobs

https://docs.opendaylight.org/projects/integration-test/en/latest/system-test-guide.html

Comment by Luis Gomez [ 12/Nov/19 ]

I would just add a note on the first link saying the JVM plots are automatically added to all CSIT jobs so no need to follow this instruction.

The second link does not go in detail to all files and logs generated by a CSIT job, so for that matter you can either leave as it is or add a note saying JVM plots will be automatically generated.

[INTTEST-14] Use JVM monitoring in each test job Created: 29/May/17 Updated: 12/Nov/19
Status:	Open
Project:	integration-test
Component/s:	None
Affects Version/s:	None
Fix Version/s:	None

[INTTEST-14] Use JVM monitoring in each test job Created: 29/May/17 Updated: 12/Nov/19