|
(In reply to Jamo Luhrsen from comment #3)
> (In reply to Vinh Nguyen from comment #2)
> > Code review submitted:
> >
> > https://git.opendaylight.org/gerrit/#/c/37358/
>
> Vinh,
>
> I tried to test this patch, but saw the same behavior of this bug.
>
> However, I had some trouble building the patch to create a distribution.
> I was using our multipatch job in the sandbox, which will create an
> entire distribution based on any patches. I used this patch for that
> job, but it failed when building with master (Boron). The failure was
> in yangtools (warning message below) so probably nothing wrong with the
> patch.
> But pulling this patch in to the stable/beryllium branch I got a distribution
> to build. Using that distribution, I re-tested this bug and saw the
> same behavior.
>
> [WARNING] Failed to notify spy hudson.maven.Maven3Builder$JenkinsEventSpy:
> org/opendaylight/yangtools/yang2sources/plugin/YangToSourcesMojo :
> Unsupported major.minor version 52.0
Hi Jamo,
I was able to verify the bug on the stable/beryllium branch. There are the reproduction steps, please let me know if I missed something:
1) git checkout stable/beryllium
2) chery-pick the patch:
git fetch https://git.opendaylight.org/gerrit/ovsdb refs/changes/58/37358/1 && git cherry-pick FETCH_HEAD
3) build and run karaf
4) connect an ovs node to controller
sudo ovs-vsctl set-manager tcp:${IP}:6640
5) create a bridge in config store and verify it's there in operational
and on ovs node
6) reboot the ovs node
7) while ovs node disconnected, create a new bridge in config store
8) when the ovs comes back up, verify that the bridge created in 5 and 7 are present in the operational as well as on ovs node.
Thanks, Vinh
|
|
(In reply to Vinh Nguyen from comment #4)
> (In reply to Jamo Luhrsen from comment #3)
> > (In reply to Vinh Nguyen from comment #2)
> > > Code review submitted:
> > >
> > > https://git.opendaylight.org/gerrit/#/c/37358/
> >
> > Vinh,
> >
> > I tried to test this patch, but saw the same behavior of this bug.
> >
> > However, I had some trouble building the patch to create a distribution.
> > I was using our multipatch job in the sandbox, which will create an
> > entire distribution based on any patches. I used this patch for that
> > job, but it failed when building with master (Boron). The failure was
> > in yangtools (warning message below) so probably nothing wrong with the
> > patch.
> > But pulling this patch in to the stable/beryllium branch I got a distribution
> > to build. Using that distribution, I re-tested this bug and saw the
> > same behavior.
> >
> > [WARNING] Failed to notify spy hudson.maven.Maven3Builder$JenkinsEventSpy:
> > org/opendaylight/yangtools/yang2sources/plugin/YangToSourcesMojo :
> > Unsupported major.minor version 52.0
>
>
> Hi Jamo,
>
> I was able to verify the bug on the stable/beryllium branch. There are the
> reproduction steps, please let me know if I missed something:
>
>
> 1) git checkout stable/beryllium
> 2) chery-pick the patch:
> git fetch https://git.opendaylight.org/gerrit/ovsdb
> refs/changes/58/37358/1 && git cherry-pick FETCH_HEAD
> 3) build and run karaf
>
> 4) connect an ovs node to controller
> sudo ovs-vsctl set-manager tcp:${IP}:6640
>
> 5) create a bridge in config store and verify it's there in operational
> and on ovs node
>
> 6) reboot the ovs node
>
> 7) while ovs node disconnected, create a new bridge in config store
>
> 8) when the ovs comes back up, verify that the bridge created in 5 and 7 are
> present in the operational as well as on ovs node.
>
> Thanks, Vinh
Vinh,
I can try again using your steps 1-3. Could you try again as well, and
instead of doing your step 6, can you run this command on your ovs node:
iptables -A OUTPUT -p tcp --dport 6640 -j DROP"
that will simulate the ovs node "going away". While it's in that state,
ovs-vsctl should show it's not connected. Now, you can go back to your
step 7 and 8.
Also, I learned today that when a node goes away like this (iptables method)
it will take aprox 3.5m before we remove it from operational. When I was
doing this test earlier today I was not waiting that long. I wonder if
there is any difference there. I have a feeling your step 6 (reboot) will
actively close the ovs connection so the plugin will see the TCP connection
closed and might immediately remove it from operational.
Thanks,
JamO
|
|
(In reply to Jamo Luhrsen from comment #5)
> (In reply to Vinh Nguyen from comment #4)
> > (In reply to Jamo Luhrsen from comment #3)
> > > (In reply to Vinh Nguyen from comment #2)
> > > > Code review submitted:
> > > >
> > > > https://git.opendaylight.org/gerrit/#/c/37358/
> > >
> > > Vinh,
> > >
> > > I tried to test this patch, but saw the same behavior of this bug.
> > >
> > > However, I had some trouble building the patch to create a distribution.
> > > I was using our multipatch job in the sandbox, which will create an
> > > entire distribution based on any patches. I used this patch for that
> > > job, but it failed when building with master (Boron). The failure was
> > > in yangtools (warning message below) so probably nothing wrong with the
> > > patch.
> > > But pulling this patch in to the stable/beryllium branch I got a distribution
> > > to build. Using that distribution, I re-tested this bug and saw the
> > > same behavior.
> > >
> > > [WARNING] Failed to notify spy hudson.maven.Maven3Builder$JenkinsEventSpy:
> > > org/opendaylight/yangtools/yang2sources/plugin/YangToSourcesMojo :
> > > Unsupported major.minor version 52.0
> >
> >
> > Hi Jamo,
> >
> > I was able to verify the bug on the stable/beryllium branch. There are the
> > reproduction steps, please let me know if I missed something:
> >
> >
> > 1) git checkout stable/beryllium
> > 2) chery-pick the patch:
> > git fetch https://git.opendaylight.org/gerrit/ovsdb
> > refs/changes/58/37358/1 && git cherry-pick FETCH_HEAD
> > 3) build and run karaf
> >
> > 4) connect an ovs node to controller
> > sudo ovs-vsctl set-manager tcp:${IP}:6640
> >
> > 5) create a bridge in config store and verify it's there in operational
> > and on ovs node
> >
> > 6) reboot the ovs node
> >
> > 7) while ovs node disconnected, create a new bridge in config store
> >
> > 8) when the ovs comes back up, verify that the bridge created in 5 and 7 are
> > present in the operational as well as on ovs node.
> >
> > Thanks, Vinh
>
> Vinh,
>
> I can try again using your steps 1-3. Could you try again as well, and
> instead of doing your step 6, can you run this command on your ovs node:
>
> iptables -A OUTPUT -p tcp --dport 6640 -j DROP"
>
> that will simulate the ovs node "going away". While it's in that state,
> ovs-vsctl should show it's not connected. Now, you can go back to your
> step 7 and 8.
>
> Also, I learned today that when a node goes away like this (iptables method)
> it will take aprox 3.5m before we remove it from operational. When I was
> doing this test earlier today I was not waiting that long. I wonder if
> there is any difference there. I have a feeling your step 6 (reboot) will
> actively close the ovs connection so the plugin will see the TCP connection
> closed and might immediately remove it from operational.
>
> Thanks,
> JamO
Hi JamO,
I also can verify the fix using iptables command for step 6) and remove the rule in step 8)
In step 6) after about couple minutes the node is removed from the datastore.
In step 8Then (In reply to Jamo Luhrsen from comment #5)
> (In reply to Vinh Nguyen from comment #4)
> > (In reply to Jamo Luhrsen from comment #3)
> > > (In reply to Vinh Nguyen from comment #2)
> > > > Code review submitted:
> > > >
> > > > https://git.opendaylight.org/gerrit/#/c/37358/
> > >
> > > Vinh,
> > >
> > > I tried to test this patch, but saw the same behavior of this bug.
> > >
> > > However, I had some trouble building the patch to create a distribution.
> > > I was using our multipatch job in the sandbox, which will create an
> > > entire distribution based on any patches. I used this patch for that
> > > job, but it failed when building with master (Boron). The failure was
> > > in yangtools (warning message below) so probably nothing wrong with the
> > > patch.
> > > But pulling this patch in to the stable/beryllium branch I got a distribution
> > > to build. Using that distribution, I re-tested this bug and saw the
> > > same behavior.
> > >
> > > [WARNING] Failed to notify spy hudson.maven.Maven3Builder$JenkinsEventSpy:
> > > org/opendaylight/yangtools/yang2sources/plugin/YangToSourcesMojo :
> > > Unsupported major.minor version 52.0
> >
> >
> > Hi Jamo,
> >
> > I was able to verify the bug on the stable/beryllium branch. There are the
> > reproduction steps, please let me know if I missed something:
> >
> >
> > 1) git checkout stable/beryllium
> > 2) chery-pick the patch:
> > git fetch https://git.opendaylight.org/gerrit/ovsdb
> > refs/changes/58/37358/1 && git cherry-pick FETCH_HEAD
> > 3) build and run karaf
> >
> > 4) connect an ovs node to controller
> > sudo ovs-vsctl set-manager tcp:${IP}:6640
> >
> > 5) create a bridge in config store and verify it's there in operational
> > and on ovs node
> >
> > 6) reboot the ovs node
> >
> > 7) while ovs node disconnected, create a new bridge in config store
> >
> > 8) when the ovs comes back up, verify that the bridge created in 5 and 7 are
> > present in the operational as well as on ovs node.
> >
> > Thanks, Vinh
>
> Vinh,
>
> I can try again using your steps 1-3. Could you try again as well, and
> instead of doing your step 6, can you run this command on your ovs node:
>
> iptables -A OUTPUT -p tcp --dport 6640 -j DROP"
>
> that will simulate the ovs node "going away". While it's in that state,
> ovs-vsctl should show it's not connected. Now, you can go back to your
> step 7 and 8.
>
> Also, I learned today that when a node goes away like this (iptables method)
> it will take aprox 3.5m before we remove it from operational. When I was
> doing this test earlier today I was not waiting that long. I wonder if
> there is any difference there. I have a feeling your step 6 (reboot) will
> actively close the ovs connection so the plugin will see the TCP connection
> closed and might immediately remove it from operational.
>
> Thanks,
> JamO
Hi JamO,
I can verify the fix using the iptables command to add/delete firewall rules in step 6 and 8 respectively.
In step 6, the ovs is disconnected after (In reply to Jamo Luhrsen from comment #5)
> (In reply to Vinh Nguyen from comment #4)
> > (In reply to Jamo Luhrsen from comment #3)
> > > (In reply to Vinh Nguyen from comment #2)
> > > > Code review submitted:
> > > >
> > > > https://git.opendaylight.org/gerrit/#/c/37358/
> > >
> > > Vinh,
> > >
> > > I tried to test this patch, but saw the same behavior of this bug.
> > >
> > > However, I had some trouble building the patch to create a distribution.
> > > I was using our multipatch job in the sandbox, which will create an
> > > entire distribution based on any patches. I used this patch for that
> > > job, but it failed when building with master (Boron). The failure was
> > > in yangtools (warning message below) so probably nothing wrong with the
> > > patch.
> > > But pulling this patch in to the stable/beryllium branch I got a distribution
> > > to build. Using that distribution, I re-tested this bug and saw the
> > > same behavior.
> > >
> > > [WARNING] Failed to notify spy hudson.maven.Maven3Builder$JenkinsEventSpy:
> > > org/opendaylight/yangtools/yang2sources/plugin/YangToSourcesMojo :
> > > Unsupported major.minor version 52.0
> >
> >
> > Hi Jamo,
> >
> > I was able to verify the bug on the stable/beryllium branch. There are the
> > reproduction steps, please let me know if I missed something:
> >
> >
> > 1) git checkout stable/beryllium
> > 2) chery-pick the patch:
> > git fetch https://git.opendaylight.org/gerrit/ovsdb
> > refs/changes/58/37358/1 && git cherry-pick FETCH_HEAD
> > 3) build and run karaf
> >
> > 4) connect an ovs node to controller
> > sudo ovs-vsctl set-manager tcp:${IP}:6640
> >
> > 5) create a bridge in config store and verify it's there in operational
> > and on ovs node
> >
> > 6) reboot the ovs node
> >
> > 7) while ovs node disconnected, create a new bridge in config store
> >
> > 8) when the ovs comes back up, verify that the bridge created in 5 and 7 are
> > present in the operational as well as on ovs node.
> >
> > Thanks, Vinh
>
> Vinh,
>
> I can try again using your steps 1-3. Could you try again as well, and
> instead of doing your step 6, can you run this command on your ovs node:
>
> iptables -A OUTPUT -p tcp --dport 6640 -j DROP"
>
> that will simulate the ovs node "going away". While it's in that state,
> ovs-vsctl should show it's not connected. Now, you can go back to your
> step 7 and 8.
>
> Also, I learned today that when a node goes away like this (iptables method)
> it will take aprox 3.5m before we remove it from operational. When I was
> doing this test earlier today I was not waiting that long. I wonder if
> there is any difference there. I have a feeling your step 6 (reboot) will
> actively close the ovs connection so the plugin will see the TCP connection
> closed and might immediately remove it from operational.
>
> Thanks,
> JamO
(In reply to Jamo Luhrsen from comment #5)
> (In reply to Vinh Nguyen from comment #4)
> > (In reply to Jamo Luhrsen from comment #3)
> > > (In reply to Vinh Nguyen from comment #2)
> > > > Code review submitted:
> > > >
> > > > https://git.opendaylight.org/gerrit/#/c/37358/
> > >
> > > Vinh,
> > >
> > > I tried to test this patch, but saw the same behavior of this bug.
> > >
> > > However, I had some trouble building the patch to create a distribution.
> > > I was using our multipatch job in the sandbox, which will create an
> > > entire distribution based on any patches. I used this patch for that
> > > job, but it failed when building with master (Boron). The failure was
> > > in yangtools (warning message below) so probably nothing wrong with the
> > > patch.
> > > But pulling this patch in to the stable/beryllium branch I got a distribution
> > > to build. Using that distribution, I re-tested this bug and saw the
> > > same behavior.
> > >
> > > [WARNING] Failed to notify spy hudson.maven.Maven3Builder$JenkinsEventSpy:
> > > org/opendaylight/yangtools/yang2sources/plugin/YangToSourcesMojo :
> > > Unsupported major.minor version 52.0
> >
> >
> > Hi Jamo,
> >
> > I was able to verify the bug on the stable/beryllium branch. There are the
> > reproduction steps, please let me know if I missed something:
> >
> >
> > 1) git checkout stable/beryllium
> > 2) chery-pick the patch:
> > git fetch https://git.opendaylight.org/gerrit/ovsdb
> > refs/changes/58/37358/1 && git cherry-pick FETCH_HEAD
> > 3) build and run karaf
> >
> > 4) connect an ovs node to controller
> > sudo ovs-vsctl set-manager tcp:${IP}:6640
> >
> > 5) create a bridge in config store and verify it's there in operational
> > and on ovs node
> >
> > 6) reboot the ovs node
> >
> > 7) while ovs node disconnected, create a new bridge in config store
> >
> > 8) when the ovs comes back up, verify that the bridge created in 5 and 7 are
> > present in the operational as well as on ovs node.
> >
> > Thanks, Vinh
>
> Vinh,
>
> I can try again using your steps 1-3. Could you try again as well, and
> instead of doing your step 6, can you run this command on your ovs node:
>
> iptables -A OUTPUT -p tcp --dport 6640 -j DROP"
>
> that will simulate the ovs node "going away". While it's in that state,
> ovs-vsctl should show it's not connected. Now, you can go back to your
> step 7 and 8.
>
> Also, I learned today that when a node goes away like this (iptables method)
> it will take aprox 3.5m before we remove it from operational. When I was
> doing this test earlier today I was not waiting that long. I wonder if
> there is any difference there. I have a feeling your step 6 (reboot) will
> actively close the ovs connection so the plugin will see the TCP connection
> closed and might immediately remove it from operational.
>
> Thanks,
> JamO
Hi JamO,
I can verify the fix successfully using iptables command to add/delete firewall rules for step6 and 8 respectively.
In step 6, the connection is disconnected after 3-4 minutes and the configurations are removed from the datastore
In step 8 after removing the firewall, the node is reconnected and new configurations are reconciled like the reboot scenario.
Thanks, Vinh
|