[CARDINAL-5] cardinal.xml slows ODL boot by ~13 minutes Created: 14/Nov/16  Updated: 19/Oct/17  Resolved: 22/Nov/16

Status: Resolved
Project: cardinal
Component/s: General
Affects Version/s: unspecified
Fix Version/s: None

Type: Bug
Reporter: Vratko Polak Assignee: Unassigned
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All

External issue ID: 7175

 Description   

This affects Carbon CSIT jobs of all install, especially if they require a feature which gets installed later, such as netconf-connector. Possibly the fix to CARDINAL-3 uses way too long timeout.

Sections from an example karaf.log from Sandbox https://logs.opendaylight.org/sandbox/jenkins091/netconf-csit-1node-userfeatures-all-carbon/2/archives/karaf.log.gz

2016-11-14 11:42:31,569 | INFO | config-pusher | ConfigPusherImpl | 135 - org.opendaylight.controller.config-persister-impl - 0.6.0.SNAPSHOT | Pushing configuration snapshot cardinal.xml(odl-integration-compatible-with-all,odl-integration-compatible-with-all)
2016-11-14 11:42:31,716 | INFO | config-pusher | CardinalProvider | 360 - org.opendaylight.cardinal.impl - 0.2.0.SNAPSHOT | CardinalProvider Session Initiated
2016-11-14 11:42:31,716 | INFO | config-pusher | OdlCardinalSysInfoApis | 360 - org.opendaylight.cardinal.impl - 0.2.0.SNAPSHOT | set data broker
2016-11-14 11:42:31,717 | INFO | config-pusher | OdlCardinalKarafInfoApi | 360 - org.opendaylight.cardinal.impl - 0.2.0.SNAPSHOT | set data broker
2016-11-14 11:42:31,717 | INFO | config-pusher | CardinalModule | 360 - org.opendaylight.cardinal.impl - 0.2.0.SNAPSHOT | Starting Snmp Daemon
2016-11-14 11:42:31,717 | INFO | config-pusher | Agent | 360 - org.opendaylight.cardinal.impl - 0.2.0.SNAPSHOT | starting Html adaptor service
2016-11-14 11:42:31,732 | INFO | config-pusher | Agent | 360 - org.opendaylight.cardinal.impl - 0.2.0.SNAPSHOT | started Html adaptor service
2016-11-14 11:42:31,805 | INFO | config-pusher | CardinalModule | 360 - org.opendaylight.cardinal.impl - 0.2.0.SNAPSHOT | Fetching system information
2016-11-14 11:42:34,813 | INFO | config-pusher | SnmpSet | 360 - org.opendaylight.cardinal.impl - 0.2.0.SNAPSHOT | Error: Response PDU is null
2016-11-14 11:42:37,818 | INFO | config-pusher | SnmpSet | 360 - org.opendaylight.cardinal.impl - 0.2.0.SNAPSHOT | Error: Response PDU is null
2016-11-14 11:42:37,858 | INFO | config-pusher | SetCardinalMibValues | 360 - org.opendaylight.cardinal.impl - 0.2.0.SNAPSHOT | karaf ssh process not running...
2016-11-14 11:42:37,858 | INFO | config-pusher | SetCardinalMibValues | 360 - org.opendaylight.cardinal.impl - 0.2.0.SNAPSHOT | error of the command (if any):
2016-11-14 11:42:40,863 | INFO | config-pusher | SnmpSet | 360 - org.opendaylight.cardinal.impl - 0.2.0.SNAPSHOT | Error: Response PDU is null
2016-11-14 11:42:41,679 | WARN | ction-172-174'}} | DeadlockMonitor | 131 - org.opendaylight.controller.config-manager - 0.6.0.SNAPSHOT | ModuleIdentifier

{factoryName='cardinal', instanceName='cardinal-default'} did not finish after 9963 ms

...

2016-11-14 11:56:06,877 | WARN | ction-172-174'}} | DeadlockMonitor | 131 - org.opendaylight.controller.config-manager - 0.6.0.SNAPSHOT | ModuleIdentifier{factoryName='cardinal', instanceName='cardinal-default'}

did not finish after 815161 ms
2016-11-14 11:56:11,706 | INFO | config-pusher | OdlCardinalGet | 360 - org.opendaylight.cardinal.impl - 0.2.0.SNAPSHOT | Getting null in response.getResponse...To handle later
2016-11-14 11:56:11,710 | INFO | config-pusher | OdlCardinalKarafInfoApi | 360 - org.opendaylight.cardinal.impl - 0.2.0.SNAPSHOT | initiating read write transaction
2016-11-14 11:56:11,760 | INFO | config-pusher | ConfigPusherImpl | 135 - org.opendaylight.controller.config-persister-impl - 0.6.0.SNAPSHOT | Successfully pushed configuration snapshot cardinal.xml(odl-integration-compatible-with-all,odl-integration-compatible-with-all)

 Comments   
Comment by Vratko Polak [ 14/Nov/16 ]

Setting severity to Major as this impacts CSIT job duration in unrelated projects.

Fixing CONTROLLER-1565 would decrease this severity.

Comment by Rajani Srivastava [ 16/Nov/16 ]

Hi Vratko,

Patch submitted and merged for removing this deadlock condition :
https://git.opendaylight.org/gerrit/#/c/48385/

Please check if cardinal still affecting CSIT jobs.

Comment by Vratko Polak [ 22/Nov/16 ]

Looking at https://jenkins.opendaylight.org/releng/view/distribution/job/distribution-csit-1node-userfeatures-all-carbon/
The first successful run #95 coincides with the Cardinal fix.

Generated at Wed Feb 07 19:15:00 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.