[CONTROLLER-632] Livelock during starting bundles Created: 16/Jul/14  Updated: 25/Jul/14  Due: 25/Jul/14  Resolved: 25/Jul/14

Status: Resolved
Project: controller
Component/s: config
Affects Version/s: Helium
Fix Version/s: None

Type: Bug
Reporter: Richard Kosegi Assignee: Tony Tkacik
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All

Attachments: Text File deadlock_opendaylight.log    
External issue ID: 1378

 Description   

There is kind of lock during ODL startup:

2014-07-16 05:42:17.128 PDT [DeadlockMonitor{TransactionIdentifier{name='ConfigTransaction-64-66'}}] WARN o.o.c.c.m.i.DeadlockMonitor$DeadlockMonitorRunnable - ModuleIdentifier

{factoryName='instruction-scheduler-impl', instanceName='global-instruction-scheduler'} did not finish after 5982 ms
2014-07-16 05:42:18.129 PDT [DeadlockMonitor{TransactionIdentifier{name='ConfigTransaction-64-66'}}] WARN o.o.c.c.m.i.DeadlockMonitor$DeadlockMonitorRunnable - ModuleIdentifier{factoryName='instruction-scheduler-impl', instanceName='global-instruction-scheduler'}

did not finish after 6983 ms
2014-07-16 05:42:19.130 PDT [DeadlockMonitor{TransactionIdentifier{name='ConfigTransaction-64-66'}}] WARN o.o.c.c.m.i.DeadlockMonitor$DeadlockMonitorRunnable - ModuleIdentifier

{factoryName='instruction-scheduler-impl', instanceName='global-instruction-scheduler'} did not finish after 7984 ms
2014-07-16 05:42:20.131 PDT [DeadlockMonitor{TransactionIdentifier{name='ConfigTransaction-64-66'}}] WARN o.o.c.c.m.i.DeadlockMonitor$DeadlockMonitorRunnable - ModuleIdentifier{factoryName='instruction-scheduler-impl', instanceName='global-instruction-scheduler'}

did not finish after 8985 ms
2014-07-16 05:42:21.132 PDT [DeadlockMonitor{TransactionIdentifier{name='ConfigTransaction-64-66'}}] WARN o.o.c.c.m.i.DeadlockMonitor$DeadlockMonitorRunnable - ModuleIdentifier

{factoryName='instruction-scheduler-impl', instanceName='global-instruction-scheduler'} did not finish after 9987 ms
2014-07-16 05:42:22.133 PDT [DeadlockMonitor{TransactionIdentifier{name='ConfigTransaction-64-66'}}] WARN o.o.c.c.m.i.DeadlockMonitor$DeadlockMonitorRunnable - ModuleIdentifier{factoryName='instruction-scheduler-impl', instanceName='global-instruction-scheduler'}

did not finish after 10987 ms
2014-07-16 05:42:23.134 PDT [DeadlockMonitor{TransactionIdentifier{name='ConfigTransaction-64-66'}}] WARN o.o.c.c.m.i.DeadlockMonitor$DeadlockMonitorRunnable - ModuleIdentifier

{factoryName='instruction-scheduler-impl', instanceName='global-instruction-scheduler'} did not finish after 11988 ms
2014-07-16 05:42:24.135 PDT [DeadlockMonitor{TransactionIdentifier{name='ConfigTransaction-64-66'}}] WARN o.o.c.c.m.i.DeadlockMonitor$DeadlockMonitorRunnable - ModuleIdentifier{factoryName='instruction-scheduler-impl', instanceName='global-instruction-scheduler'}

did not finish after 12989 ms

Please look at threads fileinstall and configpusher.

Full threadump attached.

 Comments   
Comment by Richard Kosegi [ 16/Jul/14 ]

Attachment deadlock_opendaylight.log has been added with description: Threaddump

Comment by Tomas Olvecky [ 17/Jul/14 ]

I believe this is happening:
config-pusher thread pushes config, create instance of InstructionSchedulerImplModule tries to read something from md-sal, but md-sal cannot read the data because yang model is missing.

The pushed XML should declare that it is dependent on that yang model in required-capabilities section.

Furthermore, RuntimeGeneratedMappingServiceImpl.waitForSchema should be implemented such that it periodically logs the schema on which it is waiting, similar to how DeadlockMonitor in c-s works. pseudocode:

while true:

{future.get(5secs);log.warn "waiting for schema foo";}
Comment by Tony Tkacik [ 23/Jul/14 ]

Updated config subsystem to not use platform thread to do blocking transactions
when bundle is loaded.

https://git.opendaylight.org/gerrit/#/c/9262/

Generated at Wed Feb 07 19:53:30 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.