[GENIUS-19] long delays between vm boot and flow installation Created: 11/Sep/16  Updated: 12/Sep/16  Resolved: 12/Sep/16

Status: Resolved
Project: genius
Component/s: General
Affects Version/s: (unspecified)
Fix Version/s: None

Type: Bug
Reporter: Koby Aizer Assignee: Tali Ben-Meir
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All

Attachments: Text File karaf.log-long-boot-time-ofport16-filtered.log     File of-port16-port-status-and-flowmod.pcap    
External issue ID: 6689

 Description   

Causes DHCP failures.

Attaching openflow protocol pcap that indicates the amount of time between the VM port creation and the first flow install to table=0.

Also attaching ODL log.

 Comments   
Comment by Koby Aizer [ 11/Sep/16 ]

Attachment of-port16-port-status-and-flowmod.pcap has been added with description: openflow pcap

Comment by Koby Aizer [ 11/Sep/16 ]

Attachment karaf.log-long-boot-time-ofport16-filtered.log has been added with description: odl log

Comment by Tali Ben-Meir [ 12/Sep/16 ]

Reproducible in compute nodes with many allocated cpus (tested with 48)
We experience major performance degradation because of the way job-queues are handled by DataStoreJobCoordinator

The DataStoreJobCoordinator main loop:

while(true) {
for (int i = 0; i < THREADPOOL_SIZE; i++) {
Map<String, JobQueue> jobEntriesMap = jobQueueMap.get;
if (jobEntriesMap.isEmpty())

{ Thread.sleep(500); continue; }

// handle jobEntriesMap
// ...

if (jobQueueMap.isEmpty()) {
reentrantLock.lock();
try

{ waitCondition.await(); }

finally

{ reentrantLock.unlock(); }

}
}

There are 2 main issues here resulting in throttling:
1. Jobs are not distributed evenly between the entries in the jobQueueMap. As a result this thread sleeps a lot - sometimes ~10-20sec
2. the condition guarding waitCondition.await is never reached since the jobQueueMap is never empty - it always contain an entry per thread.

Comment by Tali Ben-Meir [ 12/Sep/16 ]

https://git.opendaylight.org/gerrit/#/c/45477/

Generated at Wed Feb 07 19:59:42 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.