[OPNFLWPLUG-765] No local or remote implementation available for OF plugin's RPC Created: 08/Sep/16  Updated: 27/Sep/21  Resolved: 25/Oct/16

Status: Resolved
Project: OpenFlowPlugin
Component/s: General
Affects Version/s: None
Fix Version/s: None

Type: Bug
Reporter: Hideyuki Tai Assignee: Hideyuki Tai
Resolution: Cannot Reproduce Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

Operating System: All
Platform: All

External issue ID: 6661

 Description   

The issue happened in the CSIT for the stable/boron branches on 09/07.
I've never seen this failure before, so I'm gussing this is a regression which was introduced recently.

[VTN CSIT]

The test failure was that the VTN feature failed to control OpenFlow switches.
That's because the implmentation was not available for OF plugin's RPC.

I've seen the following log messages with the issue.

2016-09-07 13:00:43,142 | WARN (snip) org.opendaylight.controller.sal-remoterpc-connector - 1.4.0.SNAPSHOT | Timed out finding routers for RouteIdentifierImpl{context=null, type=(urn:opendaylight:packet:service?revision=2013-07-09)transmit-packet, route=/(urn:opendaylight:inventory?revision=2013-08-19)nodes/node/node[

{(urn:opendaylight:inventory?revision=2013-08-19)id=openflow:1}

]}

2016-09-07 13:00:43,189 | WARN (snip) org.opendaylight.controller.sal-remoterpc-connector - 1.4.0.SNAPSHOT | Timed out finding routers for RouteIdentifierImpl{context=null, type=(urn:opendaylight:flow:service?revision=2013-08-19)remove-flow, route=/(urn:opendaylight:inventory?revision=2013-08-19)nodes/node/node[

{(urn:opendaylight:inventory?revision=2013-08-19)id=openflow:2}

]}

2016-09-07 13:00:43,189 | WARN (snip) org.opendaylight.controller.sal-remoterpc-connector - 1.4.0.SNAPSHOT | Timed out finding routers for RouteIdentifierImpl{context=null, type=(urn:opendaylight:flow:transaction?revision=2015-03-04)send-barrier, route=/(urn:opendaylight:inventory?revision=2013-08-19)nodes/node/node[

{(urn:opendaylight:inventory?revision=2013-08-19)id=openflow:2}

]}

2016-09-07 13:00:43,190 | ERROR (snip) org.opendaylight.vtn.manager.implementation - 0.5.0.SNAPSHOT | send-barrier RPC has thrown an exception: node=openflow:2
org.opendaylight.controller.md.sal.dom.api.DOMRpcImplementationNotAvailableException: No local or remote implementation available for rpc AbsoluteSchemaPath

{path=[(urn:opendaylight:flow:transaction?revision=2015-03-04)send-barrier]}

at org.opendaylight.controller.remote.rpc.RemoteRpcImplementation$1.onComplete(RemoteRpcImplementation.java:66)[167:org.opendaylight.controller.sal-remoterpc-connector:1.4.0.SNAPSHOT]

2016-09-07 13:00:43,190 | ERROR (snip) org.opendaylight.vtn.manager.implementation - 0.5.0.SNAPSHOT | Failed to remove VTN flows in openflow:2
org.opendaylight.vtn.manager.VTNException: No local or remote implementation available for rpc AbsoluteSchemaPath

{path=[(urn:opendaylight:flow:service?revision=2013-08-19)remove-flow]}

(snip)
Caused by: org.opendaylight.controller.md.sal.dom.api.DOMRpcImplementationNotAvailableException: No local or remote implementation available for rpc AbsoluteSchemaPath

{path=[(urn:opendaylight:flow:service?revision=2013-08-19)remove-flow]}

at org.opendaylight.controller.remote.rpc.RemoteRpcImplementation$1.onComplete(RemoteRpcImplementation.java:66)[167:org.opendaylight.controller.sal-remoterpc-connector:1.4.0.SNAPSHOT]
at org.opendaylight.controller.remote.rpc.RemoteRpcImplementation$1.onComplete(RemoteRpcImplementation.java:56)[167:org.opendaylight.controller.sal-remoterpc-connector:1.4.0.SNAPSHOT]

You can find the complete log messages in the following location:
https://logs.opendaylight.org/releng/jenkins092/vtn-csit-3node-manager-all-boron/66/archives/odl3_karaf.log.gz

 Comments   
Comment by Tomas Slusny [ 09/Sep/16 ]

This issue is similar to one of issues in https://bugs.opendaylight.org/show_bug.cgi?id=6176, and that was when logging was not set to ERROR, then logger was logging too much data and was blocking other threads.
But Luis already fixed this in tests and changed log level to ERROR, and also next build triggered after this log level change passed (https://jenkins.opendaylight.org/releng/view/vtn/job/vtn-csit-3node-manager-all-boron/67/), so can this one be marked as resolved?

Comment by Hideyuki Tai [ 09/Sep/16 ]

(In reply to Tomas Slusny from comment #1)
> This issue is similar to one of issues in
> https://bugs.opendaylight.org/show_bug.cgi?id=6176, and that was when
> logging was not set to ERROR, then logger was logging too much data and was
> blocking other threads.
> But Luis already fixed this in tests and changed log level to ERROR, and
> also next build triggered after this log level change passed
> (https://jenkins.opendaylight.org/releng/view/vtn/job/vtn-csit-3node-manager-
> all-boron/67/), so can this one be marked as resolved?

Thank you for your good information!!!

The following CSIT just ran two times after the failed run (build #66). So I'm thinking it's a little bit early to say the issue is fixed now. Therefore, I would like to wait to see more runs before we judge that the issue doesn't exist anymore.

https://jenkins.opendaylight.org/releng/view/vtn/job/vtn-csit-3node-manager-all-boron/

That being said, we saw this issue only once, and it has not occurred in the "3node-manager-only" job so far.
https://jenkins.opendaylight.org/releng/view/vtn/job/vtn-csit-1node-manager-only-boron/

Therefore, I'm changing the importance of the bug from "critical" to "major".

Comment by Hideyuki Tai [ 19/Sep/16 ]

We faced this issue again on Sep 16 in the "3node-manager-only" job.

https://jenkins.opendaylight.org/releng/view/vtn/job/vtn-csit-3node-manager-only-boron/112/

Please see the following log file.
https://logs.opendaylight.org/releng/jenkins092/vtn-csit-3node-manager-only-boron/112/archives/odl3_karaf.log.gz

Therefore, I think the issue is not fixed yet. Could you someone work on that?

Comment by Tomas Slusny [ 22/Sep/16 ]

From that log file you provided I think that device was for some reason disconnected.
That error "No local or remote implementation available for rpc" means that target switch was disconnected during RPC processing.
But looking at both https://jenkins.opendaylight.org/releng/view/vtn/job/vtn-csit-3node-manager-only-boron/ and https://jenkins.opendaylight.org/releng/view/vtn/job/vtn-csit-3node-manager-only-carbon/ tests are passing since 19.09, so this can maybe be only Jenkins environment issue.

Comment by Andrej Leitner [ 25/Oct/16 ]

Hi Hideyuki, is this still the issue?

Comment by Hideyuki Tai [ 25/Oct/16 ]

(In reply to Tomas Slusny from comment #4)
> From that log file you provided I think that device was for some reason
> disconnected.
> That error "No local or remote implementation available for rpc" means that
> target switch was disconnected during RPC processing.
> But looking at both
> https://jenkins.opendaylight.org/releng/view/vtn/job/vtn-csit-3node-manager-
> only-boron/ and
> https://jenkins.opendaylight.org/releng/view/vtn/job/vtn-csit-3node-manager-
> only-carbon/ tests are passing since 19.09, so this can maybe be only
> Jenkins environment issue.

Thank you for the information.

Yeah, it could be only Jenkins environment issue...

Comment by Hideyuki Tai [ 25/Oct/16 ]

(In reply to Andrej Leitner from comment #5)
> Hi Hideyuki, is this still the issue?

I've not seen this issue recently in the Jenkins environment.

So it could be only Jenkins environment temporal issue...

Therefore, I've changed the status of the bug report to the "Resolved" with "Worksforme".

If this change is not appropriate, please feel free to change the status.

If I would face the same issue again, I would reopen this bug report or would issue another new one.

Generated at Wed Feb 07 20:33:20 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.