[CONTROLLER-1944] persisted data not restored in controller CSIT Created: 08/Jun/20  Updated: 12/Jun/20  Resolved: 12/Jun/20

Status: Resolved
Project: controller
Component/s: None
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Medium
Reporter: Jamo Luhrsen Assignee: Jamo Luhrsen
Resolution: Duplicate Votes: 0
Labels: None
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified

Issue Links:
Duplicate
duplicates INTTEST-95 Jobs that transfer journal from old v... Confirmed

 Description   

example:

https://logs.opendaylight.org/releng/vex-yul-odl-jenkins-1/controller-csit-1node-akka1-all-aluminium/97/robot-plugin/log.html.gz

 

This is a stable test with magnesium.

The failure is seen by a 409  when trying a GET on /restconf/config/cars:cars

This test is using the wildcard *journal to move the segmented-journal to the
new ODL instance that should recover the persisted data. The karaf log
does have some mentions of journal data recovery. There is one exception in the
persistence area, but no idea if it's related or not:

2020-06-08T03:08:11,487 | ERROR | opendaylight-cluster-data-akka.actor.default-dispatcher-19 | AbstractMXBean                   | 295 - org.opendaylight.controller.sal-common-util - 2.0.1 | registration failed
javax.management.InstanceAlreadyExistsException: org.opendaylight.controller:type=RemoteRpcBroker,name=RemoteRpcRegistry
	at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:436) ~[?:?]
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1855) ~[?:?]
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:955) ~[?:?]
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:890) ~[?:?]
	at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:320) ~[?:?]
	at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522) ~[?:?]
	at org.opendaylight.controller.md.sal.common.util.jmx.AbstractMXBean.registerMBean(AbstractMXBean.java:107) [bundleFile:?]
	at org.opendaylight.controller.remote.rpc.registry.mbeans.AbstractRegistryMXBean.<init>(AbstractRegistryMXBean.java:43) [bundleFile:?]
	at org.opendaylight.controller.remote.rpc.registry.mbeans.RemoteRpcRegistryMXBeanImpl.<init>(RemoteRpcRegistryMXBeanImpl.java:25) [bundleFile:?]
	at org.opendaylight.controller.remote.rpc.registry.RpcRegistry.preStart(RpcRegistry.java:70) [bundleFile:?]
	at akka.actor.Actor.aroundPreStart(Actor.scala:550) [bundleFile:2.5.31]
	at akka.actor.Actor.aroundPreStart$(Actor.scala:550) [bundleFile:2.5.31]
	at akka.persistence.AbstractPersistentActor.akka$persistence$Eventsourced$$super$aroundPreStart(PersistentActor.scala:455) [bundleFile:2.5.31]
	at akka.persistence.Eventsourced.aroundPreStart(Eventsourced.scala:261) [bundleFile:2.5.31]
	at akka.persistence.Eventsourced.aroundPreStart$(Eventsourced.scala:253) [bundleFile:2.5.31]
	at akka.persistence.AbstractPersistentActor.aroundPreStart(PersistentActor.scala:455) [bundleFile:2.5.31]
	at akka.actor.ActorCell.create(ActorCell.scala:678) [bundleFile:2.5.31]
	at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:549) [bundleFile:2.5.31]
	at akka.actor.ActorCell.systemInvoke(ActorCell.scala:571) [bundleFile:2.5.31]
	at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:293) [bundleFile:2.5.31]
	at akka.dispatch.Mailbox.run(Mailbox.scala:228) [bundleFile:2.5.31]
	at akka.dispatch.Mailbox.exec(Mailbox.scala:241) [bundleFile:2.5.31]
	at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260) [bundleFile:2.5.31]
	at akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339) [bundleFile:2.5.31]
	at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979) [bundleFile:2.5.31]
	at akka.dispatch.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107) [bundleFile:2.5.31]


 Comments   
Comment by Robert Varga [ 09/Jun/20 ]

The stack trace is completely unrelated and is fixed in CONTROLLER-1943.

Comment by Robert Varga [ 11/Jun/20 ]

This is the upgrade scenario, where the journal comes from a previous version. Given that we are switching journal plugins, the old distro will be using leveldb, the new distro will be using segmented-journal – hence the data is not recovered (as the journal will be missing).

To make this pass on Aluminium, the test should trigger a snapshot before shutting down the old instance (hence journal will be empty, hence it should not matter that we are using a different plugin).

Comment by Robert Varga [ 11/Jun/20 ]

This is specific to Mg->Al transition. Future releases should work fine, as should previous ones.

Comment by Tomas Cere [ 12/Jun/20 ]

The issue for updating the tests related to Mg -> Al transition is already reported here: INTTEST-95

Comment by Robert Varga [ 12/Jun/20 ]

Let's track this in INTTEST-95 then.

Generated at Wed Feb 07 19:56:50 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.