[NETVIRT-1049] After live migration , ssh connection not established for instance. Created: 03/Jan/18  Updated: 25/Nov/19  Resolved: 25/Nov/19

Status: Resolved
Project: netvirt
Component/s: General
Affects Version/s: None
Fix Version/s: None

Type: Bug Priority: Medium
Reporter: Hari Prasidh Assignee: Hari Prasidh
Resolution: Won't Do Votes: 0
Labels: feedback
Remaining Estimate: Not Specified
Time Spent: Not Specified
Original Estimate: Not Specified
Environment:

OpenStack version: stable/pike
ODL version: Nitrogen-FR

3-node cluster, nat-mode "conntrack"

Attachments: Zip Archive Issue_Logs.zip     Zip Archive dump_flows.zip     Zip Archive migration-issue.zip    

 Description   

Reproduction steps:

1. Create Network 1 and Network 2
2. Create VM1 and VM2 with Network 1
3. Create VM3 and VM4 with Network 2
4.Live migration for VM1
5.Verify whether ssh connection from controller node is working (ssh connection established)
6.Live migration for VM3
7.Verify whether ssh connection from controller node is working (ssh connection failed)

 Comments   
Comment by Hari Prasidh [ 09/Jan/18 ]

I have tested same scenario with single node and observed issue can be reproducible.
by initial observation, below tables flow entries were missed w.r.t. problematic vm instance.
table-50, table-51

Comment by Hari Prasidh [ 02/Feb/18 ]

I've retried with the below steps.

1.Create Network 1 and Network 2
2. Create VM1 with Network 1
3. Create VM2 with Network 2
4.Live migration for VM1
5.Verify whether ssh connection from controller node is working (ssh connection established)
6.Live migration for VM2
7.Verify whether ssh connection from controller node is working (ssh connection failed)

If not reproduce by above steps, please do repeat VM migration one host to another host

By checking dump flows dump_flows.zip below tables are missed.

Table-36,Table-50,Table-51,Table-52,Table-55

After checking logs Issue_Logs.zip, there are some Node Connector update events missed (InterfaceInventoryStateListener.java) for failure VM instance. And the Add event received with some conflict data for success and failure Vm instances.

Successful Vm instance: (Add event)
-------------------------------

2018-02-01 01:48:55,831 | DEBUG | eChangeHandler-0 | InterfaceInventoryStateListener | 232 - org.opendaylight.genius.interfacemanager-impl - 0.3.0 | Received NodeConnector Add Event: InstanceIdentifier
{targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.flow.inventory.rev130819.FlowCapableNodeConnector, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.Nodes, org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node[key=NodeKey [_id=Uri [_value=openflow:615256118989]]], org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.node.NodeConnector[key=NodeConnectorKey [_id=Uri [_value=openflow:615256118989:5]]], org.opendaylight.yang.gen.v1.urn.opendaylight.flow.inventory.rev130819.FlowCapableNodeConnector]}
, FlowCapableNodeConnector{getAdvertisedFeatures=PortFeatures [_tenMbHd=false, _tenMbFd=false, _hundredMbHd=false, _hundredMbFd=false, _oneGbHd=false, _oneGbFd=false, _tenGbFd=false, _fortyGbFd=false, _hundredGbFd=false, _oneTbFd=false, _other=false, _copper=false, _fiber=false, _autoeng=false, _pause=false, _pauseAsym=false], getConfiguration=PortConfig [_pORTDOWN=true, _nORECV=false, _nOFWD=false, _nOPACKETIN=false], getCurrentFeature=PortFeatures [_tenMbHd=false, _tenMbFd=true, _hundredMbHd=false, _hundredMbFd=false, _oneGbHd=false, _oneGbFd=false, _tenGbFd=false, _fortyGbFd=false, _hundredGbFd=false, _oneTbFd=false, _other=false, _copper=true, _fiber=false, _autoeng=false, _pause=false, _pauseAsym=false], getCurrentSpeed=10000, getHardwareAddress=MacAddress [_value=fe:16:3e:25:00:bc], getMaximumSpeed=0, getName=tap5be6a5a8-f9, getPeerFeatures=PortFeatures [_tenMbHd=false, _tenMbFd=false, _hundredMbHd=false, _hundredMbFd=false, _oneGbHd=false, _oneGbFd=false, _tenGbFd=false, _fortyGbFd=false, _hundredGbFd=false, _oneTbFd=false, _other=false, _copper=false, _fiber=false, _autoeng=false, _pause=false, _pauseAsym=false], getPortNumber=PortNumberUni [_uint32=5], getQueue=[], getReason=Add, getState=State{isBlocked=false, isLinkDown=true, isLive=false, augmentations={}}, getSupported=PortFeatures [_tenMbHd=false, _tenMbFd=false, _hundredMbHd=false, _hundredMbFd=false, _oneGbHd=false, _oneGbFd=false, _tenGbFd=false, _fortyGbFd=false, _hundredGbFd=false, _oneTbFd=false, _other=false, _copper=false, _fiber=false, _autoeng=false, _pause=false, _pauseAsym=false]}

 

Failure Vm instance: (Add event)
---------------------------------------

2018-02-01 01:49:05,833 | DEBUG | eChangeHandler-0 | InterfaceInventoryStateListener | 232 - org.opendaylight.genius.interfacemanager-impl - 0.3.0 | Received NodeConnector Add Event: InstanceIdentifier
{targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.flow.inventory.rev130819.FlowCapableNodeConnector, path=[org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.Nodes, org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node[key=NodeKey [_id=Uri [_value=openflow:615256118989]]], org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.node.NodeConnector[key=NodeConnectorKey [_id=Uri [_value=openflow:615256118989:6]]], org.opendaylight.yang.gen.v1.urn.opendaylight.flow.inventory.rev130819.FlowCapableNodeConnector]}
, FlowCapableNodeConnector{getAdvertisedFeatures=PortFeatures [_tenMbHd=false, _tenMbFd=false, _hundredMbHd=false, _hundredMbFd=false, _oneGbHd=false, _oneGbFd=false, _tenGbFd=false, _fortyGbFd=false, _hundredGbFd=false, _oneTbFd=false, _other=false, _copper=false, _fiber=false, _autoeng=false, _pause=false, _pauseAsym=false], getConfiguration=PortConfig [_pORTDOWN=false, _nORECV=false, _nOFWD=false, _nOPACKETIN=false], getCurrentFeature=PortFeatures [_tenMbHd=false, _tenMbFd=true, _hundredMbHd=false, _hundredMbFd=false, _oneGbHd=false, _oneGbFd=false, _tenGbFd=false, _fortyGbFd=false, _hundredGbFd=false, _oneTbFd=false, _other=false, _copper=true, _fiber=false, _autoeng=false, _pause=false, _pauseAsym=false], getCurrentSpeed=10000, getHardwareAddress=MacAddress [_value=fe:16:3e:f2:05:63], getMaximumSpeed=0, getName=tapb01e4b4e-ef, getPeerFeatures=PortFeatures [_tenMbHd=false, _tenMbFd=false, _hundredMbHd=false, _hundredMbFd=false, _oneGbHd=false, _oneGbFd=false, _tenGbFd=false, _fortyGbFd=false, _hundredGbFd=false, _oneTbFd=false, _other=false, _copper=false, _fiber=false, _autoeng=false, _pause=false, _pauseAsym=false], getPortNumber=PortNumberUni [_uint32=6], getQueue=[], getReason=Update, getState=State{isBlocked=false, isLinkDown=false, isLive=false, augmentations={}}, getSupported=PortFeatures [_tenMbHd=false, _tenMbFd=false, _hundredMbHd=false, _hundredMbFd=false, _oneGbHd=false, _oneGbFd=false, _tenGbFd=false, _fortyGbFd=false, _hundredGbFd=false, _oneTbFd=false, _other=false, _copper=false, _fiber=false, _autoeng=false, _pause=false, _pauseAsym=false]}

 

Comment by Sam Hague [ 06/Apr/18 ]

HariPrasidh is this still an issue?

Comment by Venkatrangan Govindarajan [ 10/Apr/18 ]

shague Yeah Sam, It still occurs, I will create a CSIT once the basic live migration test is merged

Comment by Sam Hague [ 04/Oct/18 ]

gvranganany more issues with this?

Comment by Abhinav Gupta [ 25/Nov/19 ]

will reopen if issue seen in latest release

Generated at Wed Feb 07 20:23:06 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.