[NETVIRT-1678] multiple values under a key, use Multimaps.index. at com.google.common.collect.Maps.uniqueIndex(Maps.java:1338) ~[bundleFile:?] Created: 03/Jul/20  Updated: 17/Jul/20  Resolved: 17/Jul/20

Status: Resolved
Project: netvirt
Component/s: elanmanager
Affects Version/s: Aluminium
Fix Version/s: Aluminium

Type: Bug Priority: High
Reporter: Srinivas Rachakonda Assignee: Karthikeyan Krishnan
Resolution: Done Votes: 0
Labels: None
Remaining Estimate: 0 minutes
Time Spent: 3 days
Original Estimate: Not Specified

Attachments: Zip Archive karaf.zip    
Priority: High

 Description   

VM boot goes to error state with the below exceptions:

 

opendaylight-user@root>log:list
Logger │ Level
─────────────────────────────────────────────────┼──────
ROOT │ INFO
org.apache.aries.spifly │ WARN
org.apache.karaf.jaas.modules.audit │ INFO
org.apache.karaf.shell.support.ShellUtil │ OFF
org.apache.sshd.server.session.ServerSessionImpl │ OFF
org.opendaylight.netvirt.elan │ TRACE
org.opendaylight.netvirt.neutronvpn.api.utils │ DEBUG
opendaylight-user@root>

-----------------------------------------------------------------------
stack@control:~$ openstack hypervisor list
---------------------------------------------------------

ID Hypervisor Hostname Hypervisor Type Host IP State

---------------------------------------------------------

1 compute QEMU 192.168.31.123 up
2 compute1 QEMU 192.168.31.124 up

---------------------------------------------------------

stack@control:~$ openstack network agent list
-------------------------------------------------------------------------------------------------------------------

ID Agent Type Host Availability Zone Alive State Binary

-------------------------------------------------------------------------------------------------------------------

0419c050-cf20-4e1b-92ff-f3c68b1e6bbd ODL L2 compute1 None UP neutron-odlagent-portbinding
121ab828-d2f5-4b8c-a156-17e9530a5884 DHCP agent control nova UP neutron-dhcp-agent
a429f897-6cd9-4be8-b900-3419d38170dd ODL L2 control None UP neutron-odlagent-portbinding
e15b877e-9a5a-45d1-81a8-65e431e7fd4c Metadata agent control None UP neutron-metadata-agent
f798de97-8b37-4ebb-ab9d-f0f7b3b73766 ODL L2 compute None UP neutron-odlagent-portbinding

-------------------------------------------------------------------------------------------------------------------
stack@control:~$

stack@control:~$ openstack network list
-----------------------------------------------------------------------------------

ID Name Subnets

-----------------------------------------------------------------------------------

079fa62d-766b-489b-b1da-015b2fc3d1b2 vpn_net_2 88cc102f-185b-4781-9b12-b34f53535a52
58cd941c-7330-4361-8f26-e9f2c9a71b3a vpn_net_1 7c4e243b-0b75-4d8b-8976-5c668930e80f

-----------------------------------------------------------------------------------
stack@control:~$

stack@control:~$ openstack subnet list
----------------------------------------------------------------------------------------------+

ID Name Network Subnet

----------------------------------------------------------------------------------------------+

7c4e243b-0b75-4d8b-8976-5c668930e80f vpn_sub_1 58cd941c-7330-4361-8f26-e9f2c9a71b3a 10.1.1.0/24
88cc102f-185b-4781-9b12-b34f53535a52 vpn_sub_2 079fa62d-766b-489b-b1da-015b2fc3d1b2 20.1.1.0/24

----------------------------------------------------------------------------------------------+
stack@control:~$ openstack port list
------------------------------------------------------------------------------------------------------------------------------------------------------

ID Name MAC Address Fixed IP Addresses Status

------------------------------------------------------------------------------------------------------------------------------------------------------

1d5d2461-a50a-47ec-848a-7a841e20cb12   fa:16:3e:32:8d:37 ip_address='10.1.1.2', subnet_id='7c4e243b-0b75-4d8b-8976-5c668930e80f' ACTIVE
45f25f6f-0b3f-43c0-93a3-ef2175a2b32b   fa:16:3e:ee:b2:d1 ip_address='20.1.1.2', subnet_id='88cc102f-185b-4781-9b12-b34f53535a52' ACTIVE
5f129747-80b2-49ab-989a-49bc17fbd8a7 vpn_net_1_port_1 fa:16:3e:06:e1:b2 ip_address='10.1.1.78', subnet_id='7c4e243b-0b75-4d8b-8976-5c668930e80f' ACTIVE
88590407-e2c5-4f5c-bfcd-eb798e9d1cd7 vpn_net_2_port_1 fa:16:3e:cf:45:bd ip_address='20.1.1.32', subnet_id='88cc102f-185b-4781-9b12-b34f53535a52' ACTIVE
aa41ea55-f0e2-4648-b52c-237823cf6dcb vpn_net_1_port_2 fa:16:3e:7d:b9:2b ip_address='10.1.1.170', subnet_id='7c4e243b-0b75-4d8b-8976-5c668930e80f' DOWN
ba7949c6-9b6e-4e57-bc9b-82fd2fd750b3 vpn_net_2_port_2 fa:16:3e:5a:60:59 ip_address='20.1.1.87', subnet_id='88cc102f-185b-4781-9b12-b34f53535a52' DOWN

------------------------------------------------------------------------------------------------------------------------------------------------------
stack@control:~$

stack@control:~$ openstack port show vpn_net_1_port_2
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Field Value

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

admin_state_up UP
allowed_address_pairs ip_address='71.1.1.0/24', mac_address='fa:16:3e:7d:b9:2b'
  ip_address='72.1.1.0/24', mac_address='fa:16:3e:7d:b9:2b'
binding_host_id  
binding_profile  
binding_vif_details  
binding_vif_type unbound
binding_vnic_type normal
created_at 2020-07-03T05:41:09Z
data_plane_status None
description  
device_id  
device_owner  
dns_assignment None
dns_domain None
dns_name None
extra_dhcp_opts  
fixed_ips ip_address='10.1.1.170', subnet_id='7c4e243b-0b75-4d8b-8976-5c668930e80f'
id aa41ea55-f0e2-4648-b52c-237823cf6dcb
location Munch({'project': Munch( {'domain_id': 'default', 'id': u'd55770c454e44141bb80a2a1b8918152', 'name': 'admin', 'domain_name': None}

), 'cloud': '', 'region_name': 'RegionOne', 'zone': None})

mac_address fa:16:3e:7d:b9:2b
name vpn_net_1_port_2
network_id 58cd941c-7330-4361-8f26-e9f2c9a71b3a
port_security_enabled True
project_id d55770c454e44141bb80a2a1b8918152
propagate_uplink_status None
qos_policy_id None
resource_request None
revision_number 10
security_group_ids ddf4d314-93e6-4e2f-8f98-4905fd0354e1
status DOWN
tags  
trunk_details None
updated_at 2020-07-03T07:11:45Z

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
stack@control:~$

stack@control:~$ openstack server create --image cirros-0.4.0-x86_64-disk --flavor m1.nano --port vpn_net_1_port_2 --security-group vpn_sg --availability-zone nova:compute1 vpn_net_1_vm_2
----------------------------------------------------------------------------------------------------+

Field Value

----------------------------------------------------------------------------------------------------+

OS-DCF:diskConfig MANUAL
OS-EXT-AZ:availability_zone nova
OS-EXT-SRV-ATTR:host None
OS-EXT-SRV-ATTR:hypervisor_hostname None
OS-EXT-SRV-ATTR:instance_name  
OS-EXT-STS:power_state NOSTATE
OS-EXT-STS:task_state scheduling
OS-EXT-STS:vm_state building
OS-SRV-USG:launched_at None
OS-SRV-USG:terminated_at None
accessIPv4  
accessIPv6  
addresses  
adminPass dJY8L5XmgucU
config_drive  
created 2020-07-03T09:11:44Z
flavor m1.nano (42)
hostId  
id 9d5f06d0-df28-46d7-a2bd-922746ad2261
image cirros-0.4.0-x86_64-disk (d5dad237-60d7-43e2-9844-7c65b58e1b4e)
key_name None
name vpn_net_1_vm_2
progress 0
project_id d55770c454e44141bb80a2a1b8918152
properties  
security_groups name='ddf4d314-93e6-4e2f-8f98-4905fd0354e1'
status BUILD
updated 2020-07-03T09:11:45Z
user_id c7288f86ef4549d0a78b74044f12a5f5
volumes_attached  

----------------------------------------------------------------------------------------------------+
stack@control:~$
stack@control:~$ openstack server list
------------------------------------------------------------------------------------------------------------+

ID Name Status Networks Image Flavor

------------------------------------------------------------------------------------------------------------+

9d5f06d0-df28-46d7-a2bd-922746ad2261 vpn_net_1_vm_2 BUILD   cirros-0.4.0-x86_64-disk m1.nano
02389df0-2299-4490-a2ac-b035381542ca vpn_net_2_vm_2 ERROR   cirros-0.4.0-x86_64-disk m1.nano
fb3b400c-807a-4e20-977e-5db03c5bd9ae vpn_net_2_vm_1 ACTIVE vpn_net_2=20.1.1.32 cirros-0.4.0-x86_64-disk m1.nano
8a14e761-eeb4-4bc4-9f85-ccda3075132b vpn_net_1_vm_1 ACTIVE vpn_net_1=10.1.1.78 cirros-0.4.0-x86_64-disk m1.nano

------------------------------------------------------------------------------------------------------------+
stack@control:~$

stack@control:~$ openstack server list
------------------------------------------------------------------------------------------------------------+

ID Name Status Networks Image Flavor

------------------------------------------------------------------------------------------------------------+

9d5f06d0-df28-46d7-a2bd-922746ad2261 vpn_net_1_vm_2 ERROR   cirros-0.4.0-x86_64-disk m1.nano
02389df0-2299-4490-a2ac-b035381542ca vpn_net_2_vm_2 ERROR   cirros-0.4.0-x86_64-disk m1.nano
fb3b400c-807a-4e20-977e-5db03c5bd9ae vpn_net_2_vm_1 ACTIVE vpn_net_2=20.1.1.32 cirros-0.4.0-x86_64-disk m1.nano
8a14e761-eeb4-4bc4-9f85-ccda3075132b vpn_net_1_vm_1 ACTIVE vpn_net_1=10.1.1.78 cirros-0.4.0-x86_64-disk m1.nano

------------------------------------------------------------------------------------------------------------+
stack@control:~$ openstack server show vpn_net_1_vm_2
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

Field Value

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

OS-DCF:diskConfig MANUAL
OS-EXT-AZ:availability_zone  
OS-EXT-SRV-ATTR:host None
OS-EXT-SRV-ATTR:hypervisor_hostname None
OS-EXT-SRV-ATTR:instance_name instance-00000007
OS-EXT-STS:power_state NOSTATE
OS-EXT-STS:task_state None
OS-EXT-STS:vm_state error
OS-SRV-USG:launched_at None
OS-SRV-USG:terminated_at None
accessIPv4  
accessIPv6  
addresses  
config_drive  
created 2020-07-03T09:11:44Z
fault {u'message': u'Build of instance 9d5f06d0-df28-46d7-a2bd-922746ad2261 aborted: Failed to allocate the network(s), not rescheduling.', u'code': 500, u'details': u'Traceback (most recent call last):\n File "/opt/stack/nova/nova/compute/manager.py", line 2081, in _do_build_and_run_instance\n filter_properties, request_spec)\n File "/opt/stack/nova/nova/compute/manager.py", line 2419, in _build_and_run_instance\n reason=msg)\nBuildAbortException: Build of instance 9d5f06d0-df28-46d7-a2bd-922746ad2261 aborted: Failed to allocate the network(s), not rescheduling.\n', u'created': u'2020-07-03T09:17:02Z'}
flavor m1.nano (42)
hostId  
id 9d5f06d0-df28-46d7-a2bd-922746ad2261
image cirros-0.4.0-x86_64-disk (d5dad237-60d7-43e2-9844-7c65b58e1b4e)
key_name None
name vpn_net_1_vm_2
project_id d55770c454e44141bb80a2a1b8918152
properties  
status ERROR
updated 2020-07-03T09:17:03Z
user_id c7288f86ef4549d0a78b74044f12a5f5
volumes_attached  

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
stack@control:~$

 

 

Karaf logs are attached.

 

 

 

 

 

 Comments   
Comment by Robert Varga [ 05/Jul/20 ] 
2020-07-03T05:42:23,713 | ERROR | jobcoordinator-main-task-0 | JobCoordinatorImpl               | 269 - org.opendaylight.infrautils.jobcoordinator-impl - 1.8.0.SNAPSHOT | Job still failed on final retry: JobEntry{key='bc.group.update.58cd941c-7330-4361-8f26-e9f2c9a71b3a', jobId='J371', queueId='Q267', mainWorker=org.opendaylight.netvirt.elan.l2gw.jobs.BcGroupUpdateJob@43704603, rollbackWorker=null, retryCount=5/5, futures=[com.google.common.util.concurrent.ImmediateFuture$ImmediateFailedFuture@23102b67[status=FAILURE, cause=[java.lang.IllegalArgumentException: Multiple entries with same key: BucketKey{_bucketId=BucketId{_value=1}}=Bucket{_action={ActionKey{_order=0}=Action{_action=SetFieldCase{_setField=SetField{_tunnel=Tunnel{_tunnelId=1001, augmentation=[]}, augmentation=[]}, augmentation=[]}, _order=0, augmentation=[]}, ActionKey{_order=1}=Action{_action=NxActionRegLoadNodesNodeTableFlowApplyActionsCase{_nxRegLoad=NxRegLoad{_dst=Dst{_dstChoice=DstNxRegCase{_nxReg=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowjava.nx.match.rev140421.NxmNxReg6, augmentation=[]}, _end=31, _start=0, augmentation=[]}, _value=1280, augmentation=[]}, augmentation=[]}, _order=1, augmentation=[]}, ActionKey{_order=2}=Action{_action=NxActionResubmitRpcAddGroupCase{_nxResubmit=NxResubmit{_table=220, augmentation=[]}, augmentation=[]}, _order=2, augmentation=[]}}, _bucketId=BucketId{_value=1}, _watchGroup=4294967295, _watchPort=4294967295, _weight=0, augmentation=[]} and BucketKey{_bucketId=BucketId{_value=1}}=Bucket{_action={ActionKey{_order=0}=Action{getAction=SetFieldCase{getSetField=SetField{getTunnel=Tunnel{getTunnelId=1001, augmentation=[]}, augmentation=[]}, augmentation=[]}, getOrder=0, augmentation=[]}, ActionKey{_order=1}=Action{getAction=NxActionRegLoadNodesNodeGroupBucketsBucketActionsCase{getNxRegLoad=NxRegLoad{getDst=Dst{getDstChoice=DstNxRegCase{getNxReg=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowjava.nx.match.rev140421.NxmNxReg6, augmentation=[]}, getEnd=31, getStart=0, augmentation=[]}, getValue=512, augmentation=[]}, augmentation=[]}, getOrder=1, augmentation=[]}, ActionKey{_order=2}=Action{getAction=NxActionResubmitNodesNodeGroupBucketsBucketActionsCase{getNxResubmit=NxResubmit{getInPort=65528, getTable=220, augmentation=[]}, augmentation=[]}, getOrder=2, augmentation=[]}}, _bucketId=BucketId{_value=1}, _watchGroup=4294967295, _watchPort=4294967295, _weight=0, augmentation=[]}. To index multiple values under a key, use Multimaps.index.]]]}
java.lang.IllegalArgumentException: Multiple entries with same key: BucketKey{_bucketId=BucketId{_value=1}}=Bucket{_action={ActionKey{_order=0}=Action{_action=SetFieldCase{_setField=SetField{_tunnel=Tunnel{_tunnelId=1001, augmentation=[]}, augmentation=[]}, augmentation=[]}, _order=0, augmentation=[]}, ActionKey{_order=1}=Action{_action=NxActionRegLoadNodesNodeTableFlowApplyActionsCase{_nxRegLoad=NxRegLoad{_dst=Dst{_dstChoice=DstNxRegCase{_nxReg=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowjava.nx.match.rev140421.NxmNxReg6, augmentation=[]}, _end=31, _start=0, augmentation=[]}, _value=1280, augmentation=[]}, augmentation=[]}, _order=1, augmentation=[]}, ActionKey{_order=2}=Action{_action=NxActionResubmitRpcAddGroupCase{_nxResubmit=NxResubmit{_table=220, augmentation=[]}, augmentation=[]}, _order=2, augmentation=[]}}, _bucketId=BucketId{_value=1}, _watchGroup=4294967295, _watchPort=4294967295, _weight=0, augmentation=[]} and BucketKey{_bucketId=BucketId{_value=1}}=Bucket{_action={ActionKey{_order=0}=Action{getAction=SetFieldCase{getSetField=SetField{getTunnel=Tunnel{getTunnelId=1001, augmentation=[]}, augmentation=[]}, augmentation=[]}, getOrder=0, augmentation=[]}, ActionKey{_order=1}=Action{getAction=NxActionRegLoadNodesNodeGroupBucketsBucketActionsCase{getNxRegLoad=NxRegLoad{getDst=Dst{getDstChoice=DstNxRegCase{getNxReg=interface org.opendaylight.yang.gen.v1.urn.opendaylight.openflowjava.nx.match.rev140421.NxmNxReg6, augmentation=[]}, getEnd=31, getStart=0, augmentation=[]}, getValue=512, augmentation=[]}, augmentation=[]}, getOrder=1, augmentation=[]}, ActionKey{_order=2}=Action{getAction=NxActionResubmitNodesNodeGroupBucketsBucketActionsCase{getNxResubmit=NxResubmit{getInPort=65528, getTable=220, augmentation=[]}, augmentation=[]}, getOrder=2, augmentation=[]}}, _bucketId=BucketId{_value=1}, _watchGroup=4294967295, _watchPort=4294967295, _weight=0, augmentation=[]}. To index multiple values under a key, use Multimaps.index.
	at com.google.common.collect.Maps.uniqueIndex(Maps.java:1338) ~[bundleFile:?]
	at com.google.common.collect.Maps.uniqueIndex(Maps.java:1293) ~[bundleFile:?]
	at org.opendaylight.yangtools.yang.binding.CodeHelpers.compatMap(CodeHelpers.java:296) ~[?:?]
	at org.opendaylight.yang.gen.v1.urn.opendaylight.group.types.rev131018.group.BucketsBuilder.setBucket(BucketsBuilder.java:113) ~[?:?]
	at org.opendaylight.genius.mdsalutil.MDSALUtil.buildBucketLists(MDSALUtil.java:293) ~[?:?]
	at org.opendaylight.netvirt.elan.utils.ElanUtils.syncUpdateGroup(ElanUtils.java:1768) ~[?:?]
	at org.opendaylight.netvirt.elan.l2gw.utils.ElanL2GatewayMulticastUtils.setupStandardElanBroadcastGroups(ElanL2GatewayMulticastUtils.java:303) ~[?:?]
	at org.opendaylight.netvirt.elan.l2gw.utils.ElanL2GatewayMulticastUtils.updateRemoteBroadcastGroupForAllElanDpns(ElanL2GatewayMulticastUtils.java:265) ~[?:?]
	at org.opendaylight.netvirt.elan.l2gw.jobs.BcGroupUpdateJob.lambda$call$0(BcGroupUpdateJob.java:59) ~[?:?]
	at org.opendaylight.genius.infra.ManagedTransactionFactoryImpl.lambda$callWithNewTransactionAndSubmit$3(ManagedTransactionFactoryImpl.java:89) ~[?:?]
	at org.opendaylight.genius.infra.ManagedTransactionFactoryImpl.applyWithNewTransactionAndSubmit(ManagedTransactionFactoryImpl.java:105) ~[?:?]
	at org.opendaylight.genius.infra.ManagedTransactionFactoryImpl.callWithNewTransactionAndSubmit(ManagedTransactionFactoryImpl.java:88) ~[?:?]
	at org.opendaylight.genius.infra.ManagedNewTransactionRunnerImpl.callWithNewWriteOnlyTransactionAndSubmit(ManagedNewTransactionRunnerImpl.java:63) ~[?:?]
	at org.opendaylight.netvirt.elan.l2gw.jobs.BcGroupUpdateJob.call(BcGroupUpdateJob.java:58) ~[?:?]
	at org.opendaylight.netvirt.elan.l2gw.jobs.BcGroupUpdateJob.call(BcGroupUpdateJob.java:27) ~[?:?]
	at org.opendaylight.infrautils.jobcoordinator.internal.JobCoordinatorImpl$MainTask.run(JobCoordinatorImpl.java:416) [bundleFile:?]
	at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1426) [?:?]
	at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) [?:?]
	at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) [?:?]
	at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) [?:?]
	at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) [?:?]
	at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177) [?:?]

A modeling/keying problem perhaps?

 

Comment by Karthikeyan Krishnan [ 09/Jul/20 ]

Yes Robert. We are working on the fix to solve this issue.

Comment by Karthikeyan Krishnan [ 17/Jul/20 ]

The following gerrit links for merged into Aluminium release to solve the reported problem in netvirt proejct.

 

https://git.opendaylight.org/gerrit/c/netvirt/+/91343

https://git.opendaylight.org/gerrit/c/netvirt/+/91036

https://git.opendaylight.org/gerrit/c/netvirt/+/91344

 

Generated at Wed Feb 07 20:24:41 UTC 2024 using Jira 8.20.10#820010-sha1:ace47f9899e9ee25d7157d59aa17ab06aee30d3d.