https://jira.opendaylight.org This file is an XML representation of an issue en-us 8.20.10 820010 22-06-2022 https://jira.opendaylight.org/browse/GENIUS-166 genius <p>Genius Fluorine CSIT is randomly hitting some RESTCONF ReadTimeOuts on some of the DELETE/POST requests recently.</p> <p><a href="https://jenkins.opendaylight.org/sandbox/job/jamo-genius-csit-1node-gate-all-fluorine/" class="external-link" target="_blank" rel="nofollow noopener">https://jenkins.opendaylight.org/sandbox/job/jamo-genius-csit-1node-gate-all-fluorine/</a></p> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <th class='confluenceTh'>Documentation:</th> <td class='confluenceTd'>Send a DELETE request on the session object found using the</td> </tr> <tr> <th class='confluenceTh'>Start / End / Elapsed:</th> <td class='confluenceTd'>20180531 04:34:26.956 / 20180531 04:34:31.569 / 00:00:04.613</td> </tr> </tbody></table> </div> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <td class='confluenceTd'>04:34:27.959</td> <td class='confluenceTd'>WARN</td> <td class='confluenceTd'>Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='10.30.170.29', port=8181): Read timed out. (read timeout=1.0)",)': /restconf/config/itm:transport-zones/transport-zone/TZA/</td> <td class='confluenceTd'> </td> </tr> </tbody></table> </div> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <td class='confluenceTd'>04:34:29.162</td> <td class='confluenceTd'>WARN</td> <td class='confluenceTd'>Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='10.30.170.29', port=8181): Read timed out. (read timeout=1.0)",)': /restconf/config/itm:transport-zones/transport-zone/TZA/</td> <td class='confluenceTd'> </td> </tr> </tbody></table> </div> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <td class='confluenceTd'>04:34:30.565</td> <td class='confluenceTd'>WARN</td> <td class='confluenceTd'>Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='10.30.170.29', port=8181): Read timed out. (read timeout=1.0)",)': /restconf/config/itm:transport-zones/transport-zone/TZA/</td> <td class='confluenceTd'> </td> </tr> </tbody></table> </div> <div class='table-wrap'> <table class='confluenceTable'><tbody> <tr> <td class='confluenceTd'>04:34:31.568</td> <td class='confluenceTd'>FAIL</td> <td class='confluenceTd'>ConnectionError: HTTPConnectionPool(host='10.30.170.29', port=8181): Max retries exceeded with url: /restconf/config/itm:transport-zones/transport-zone/TZA/ (Caused by ReadTimeoutError("HTTPConnectionPool(host='10.30.170.29', port=8181): Read timed out. (read timeout=1.0)",))</td> </tr> </tbody></table> </div> <p>We see the below exception in karaf log on all failing runs, but not sure whether that is the reason for the failure though:</p> <p>     2018-05-30T15:08:06,198 | WARN  | opendaylight-cluster-data-shard-dispatcher-88 | ShardDataTree 240 - org.opendaylight.controller.sal-distributed-datastore -| 1.8.0.SNAPSHOT | member-1-shard-default-config: Current transaction member-1-datastore-config-fe-0-txn-1477-0 has timed out after 19233 ms in state CAN_COMMIT_COMPLETE</p> <p>     2018-05-30T15:08:06,198 | WARN  | opendaylight-cluster-data-shard-dispatcher-65 | ShardDataTree | 240 - org.opendaylight.controller.sal-distributed-datastore - 1.8.0.SNAPSHOT | member-1-shard-inventory-config: Current transaction member-1-datastore-config-fe-0-txn-1478-0 has timed out after 19234 ms in state READY</p> <p>     2018-05-30T15:08:06,199 | ERROR | opendaylight-cluster-data-shard-dispatcher-88 | Shard | 232 - org.opendaylight.controller.sal-clustering-commons - | 1.8.0.SNAPSHOT | member-1-shard-inventory-config: Cannot canCommit transaction member-1-datastore-config-fe-0-txn-1478-0 - no cohort entry found</p> <p>2018-05-30T15:08:06,199 | ERROR | opendaylight-cluster-data-shard-dispatcher-65 | Shard                            | 232 - org.opendaylight.controller.sal-clustering-commons - 1.8.0.SNAPSHOT | member-1-shard-default-config: Cannot commit transaction member-1-datastore-config-fe-0-txn-1477-0 - no cohort entry found</p> GENIUS-166

Genius CSIT Intermittent RESTCONF ReadTimeOut Errors for POST/DELETE requests

Bug High Resolved Done Tom Pantelis Faseela K csit:3node csit:failures Thu, 31 May 2018 00:02:51 +0000 Wed, 27 Jun 2018 12:44:40 +0000 Wed, 27 Jun 2018 12:44:40 +0000 Fluorine 0 6 <p>Perhaps this is related to &amp; caused by <a href="https://jira.opendaylight.org/browse/NETCONF-546" title="404 returning empty response" class="issue-link" data-issue-key="NETCONF-546"><del>NETCONF-546</del></a> ?</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=tpantelis" class="user-hover" rel="tpantelis">tpantelis</a> on <a href="https://lists.opendaylight.org/pipermail/genius-dev/2018-June/002903.html" class="external-link" target="_blank" rel="nofollow noopener">this reply</a> also pointed to <a href="https://git.opendaylight.org/gerrit/#/c/72525/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/72525/</a> - still happening, with that?</p> <p>Ran with the DEBUG logs asked by <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=tpantelis" class="user-hover" rel="tpantelis">tpantelis</a></p> <p>   With the logs enabled, this particular error came in one of the runs. I had to run with two suites as the issue was not coming that frequently when I ran with one suite.</p> <p> </p> <p>   after connection broken by 'ReadTimeoutError("HTTPConnectionPool(host='10.30.170.18', port=8181): Read timed out. (read timeout=1.0)",)': /restconf/config/itm:transport-zones/transport-zone/TZA/</p> <p> </p> <p>            <a href="https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/87/Karthikagenius-csit-1node-gate-all-fluorine/41/" class="external-link" target="_blank" rel="nofollow noopener">https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/builder-copy-sandbox-logs/87/Karthikagenius-csit-1node-gate-all-fluorine/41/</a></p> <p> </p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=tpantelis" class="user-hover" rel="tpantelis">tpantelis</a> after taking a look at the logs informed that "there's an indirect deadlock scenario in CDS wrt txns that span multiple shards. I'll work on a patch soon. No ETA yet..."</p> <p>Submitted <a href="https://git.opendaylight.org/gerrit/#/c/72650/" class="external-link" target="_blank" rel="nofollow noopener">https://git.opendaylight.org/gerrit/#/c/72650/</a></p> <p>The patch looks good for a stop-gap. In order to allow concurrency across shard-sets, we'd need a per-shard lock somewhere and acquire them in order. We can do that later if it proves to be a problem.</p> <p>As per <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=tpantelis" class="user-hover" rel="tpantelis">tpantelis</a> latest reply, The easiest solution to avoid this issue and probably help with performance is to just go with one shard like you've been talking about anyway - that's easy enough - just customize the modules-shards-config. Otherwise, to optimize for sharding you really have to understand and manage the app's access patterns but it seems the genius/netvirt patterns are too complex and erratic for that</p> <p> </p> <p>according to <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=tpantelis" class="user-hover" rel="tpantelis">tpantelis</a>, <a href="https://jira.opendaylight.org/browse/CONTROLLER-1836" title="Deadlock scenario with multi-shard transactions" class="issue-link" data-issue-key="CONTROLLER-1836"><del>CONTROLLER-1836</del></a> <a href="https://git.opendaylight.org/gerrit/#/c/72874/" class="external-link" target="_blank" rel="nofollow noopener">c/72874</a> fixes this.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=k.faseela" class="user-hover" rel="k.faseela">k.faseela</a> verify and close this?</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=vorburger" class="user-hover" rel="vorburger">vorburger</a> : c/72874 is not merged yet. We will run the CSIT in a loop to verify the issue is fixed or not, once the patch is merged.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=k.faseela" class="user-hover" rel="k.faseela">k.faseela</a> <a href="https://jenkins.opendaylight.org/sandbox/job/jamo-genius-csit-1node-gate-all-fluorine/" class="external-link" target="_blank" rel="nofollow noopener">this sandbox job</a> will run every 20m with the <a href="https://nexus.opendaylight.org/content/repositories/opendaylight.snapshot/org/opendaylight/integration/controller/karaf/0.9.0-SNAPSHOT/karaf-0.9.0-20180620.023551-202.zip" class="external-link" target="_blank" rel="nofollow noopener">distro</a> from <a href="https://git.opendaylight.org/gerrit/#/c/72874/" class="external-link" target="_blank" rel="nofollow noopener">c/72874 </a></p> <p>I am very curious to see what we get. I think I have stumbled across similar failures in other suites recently,<br/> so maybe this is happening more often than we think, and not just in genius.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> The patch is merged in master. <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=tpantelis" class="user-hover" rel="tpantelis">tpantelis</a> : Any plans to get this in for stable/oxygen?</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> Can u edit the sandbox job to run only Configure_ITM suite? Looks like the newly added suite has some other failures, and it might distract.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=k.faseela" class="user-hover" rel="k.faseela">k.faseela</a>, the sandbox job is edited now. It will not use the custom distro, since the patch was merged yesterday. (BTW, I thought we<br/> were going to wait for CSIT results before doing that) I also removed that Configure_ITM suite, because like you said it creates noise.<br/> You should probably have someone remove that suite and file a Jira to track whatever it is that's making it unstable.</p> <p>BUT!!!! I did look at all the failures in the sandbox from overnight, and none of them were because of this issue. Seems like we are good<br/> now.</p> <p>big thanks to <a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=tpantelis" class="user-hover" rel="tpantelis">tpantelis</a> for the fix.</p> <p><a href="https://jira.opendaylight.org/secure/ViewProfile.jspa?name=jluhrsen" class="user-hover" rel="jluhrsen">jluhrsen</a> : I think the CSIT runs are looking good atleast from this error perspective.. Should we close this Jira now?</p> <p>agreed</p> Blocks NETCONF-546 CONTROLLER-1836 Development Priority Rank 0|i03f1j: