Notification of FLOW_REMOVED is not sent when using the new OF plugin.

I run the Virtualization edition with vtn option with the new OF plugin, and I had VTN Manager install flow entries whose idle timeout was configured to 300.
After I waited 300 seconds, OpenFlow switches sent FLOW_REMOVED messages to the controller.
However, IFlowProgrammerListener.flowRemoved(Node, Flow) was not invoked, so VTN Manager didn't notice that.

Please note that when I run without the new OF plugin, but with the old OF plugin, IFlowProgrammerListener.flowRemoved(Node, Flow) was invoked correctly.

What I used
-----------

The latest version of the Virtualization Edition on 5th June.

$ pwd
/home/odp/work/integration

odc180 $ git log -3 --oneline
90e2f0c INTEGRAT-4: Remove toaster from virt edition
64e608a Adding extra TCs for RESTCONF
3516960 Uploaded working AD-SAL cluster testcases

How to produce
--------------

1. Start up the Virtualization edition with -of13 and vtn option.

$ ./run.sh -of13 -virt vtn

2. Create a virtual bridge using REST API exposed by VTN Manager.

curl --user "admin":"admin" -H "Accept: application/json" -H \
"Content-type: application/json" -X POST \
http://192.168.60.180:8080/controller/nb/v2/vtn/default/vtns/vtn1 \
-d '

{"description": "Virtual Tenat 1"}

'

curl --user "admin":"admin" -H "Accept: application/json" -H \
"Content-type: application/json" -X POST \
http://192.168.60.180:8080/controller/nb/v2/vtn/default/vtns/vtn1/vbridges/vbr1 \
-d '{}'

curl --user "admin":"admin" -H "Accept: application/json" -H \
"Content-type: application/json" -X POST \
http://192.168.60.180:8080/controller/nb/v2/vtn/default/vtns/vtn1/vbridges/vbr1/interfaces/if1 \
-d '{}'

curl --user "admin":"admin" -H "Accept: application/json" -H \
"Content-type: application/json" -X POST \
http://192.168.60.180:8080/controller/nb/v2/vtn/default/vtns/vtn1/vbridges/vbr1/interfaces/if2 \
-d '{}'

curl --user "admin":"admin" -H "Accept: application/json" -H \
"Content-type: application/json" -X PUT \
http://192.168.60.180:8080/controller/nb/v2/vtn/default/vtns/vtn1/vbridges/vbr1/interfaces/if1/portmap \
-d '{"node":

{"type":"MD_SAL","id":"openflow:3"}

, "port": {"name": "s3-eth1"}}'

curl --user "admin":"admin" -H "Accept: application/json" -H \
"Content-type: application/json" -X PUT \
http://192.168.60.180:8080/controller/nb/v2/vtn/default/vtns/vtn1/vbridges/vbr1/interfaces/if2/portmap \
-d '{"node":

{"type":"MD_SAL","id":"openflow:7"}

, "port": {"name": "s7-eth2"}}'

3. Run Mininet and run OF1.0 switches.

sudo mn --controller=remote,192.168.60.180 --topo tree,3

4. Send ping packets from h1 to h8.

mininet> h1 ping h8

At this time, VTN Manager installs ten flow entries into the switches based on the above configuration.

5. Check flow entries in mininet.

Two flow entries have "idle_timeout=300", in s3 and s7.

mininet> dpctl dump-flows

• • • s1 ------------------------------------------------------------------------
      NXST_FLOW reply (xid=0x4):
      cookie=0x0, duration=295.615s, table=0, n_packets=0, n_bytes=0, idle_age=295, priority=10,in_port=1,dl_vlan=0,dl_src=e6:e1:fe:2f:2e:ea,dl_dst=6e:0c:a2:71:13:1c actions=output:2
      cookie=0x0, duration=295.69s, table=0, n_packets=0, n_bytes=0, idle_age=295, priority=10,in_port=2,dl_vlan=0,dl_src=6e:0c:a2:71:13:1c,dl_dst=e6:e1:fe:2f:2e:ea actions=output:1
    • s2 ------------------------------------------------------------------------
      NXST_FLOW reply (xid=0x4):
      cookie=0x0, duration=295.587s, table=0, n_packets=0, n_bytes=0, idle_age=295, priority=10,in_port=1,dl_vlan=0,dl_src=e6:e1:fe:2f:2e:ea,dl_dst=6e:0c:a2:71:13:1c actions=output:3
      cookie=0x0, duration=295.657s, table=0, n_packets=0, n_bytes=0, idle_age=295, priority=10,in_port=3,dl_vlan=0,dl_src=6e:0c:a2:71:13:1c,dl_dst=e6:e1:fe:2f:2e:ea actions=output:1
    • s3 ------------------------------------------------------------------------
      NXST_FLOW reply (xid=0x4):
      cookie=0x0, duration=295.588s, table=0, n_packets=0, n_bytes=0, idle_timeout=300, idle_age=295, priority=10,in_port=1,dl_vlan=0,dl_src=e6:e1:fe:2f:2e:ea,dl_dst=6e:0c:a2:71:13:1c actions=output:3
      cookie=0x0, duration=295.732s, table=0, n_packets=0, n_bytes=0, idle_age=295, priority=10,in_port=3,dl_vlan=0,dl_src=6e:0c:a2:71:13:1c,dl_dst=e6:e1:fe:2f:2e:ea actions=output:1
    • s4 ------------------------------------------------------------------------
      NXST_FLOW reply (xid=0x4):
    • s5 ------------------------------------------------------------------------
      NXST_FLOW reply (xid=0x4):
      cookie=0x0, duration=295.615s, table=0, n_packets=0, n_bytes=0, idle_age=295, priority=10,in_port=3,dl_vlan=0,dl_src=e6:e1:fe:2f:2e:ea,dl_dst=6e:0c:a2:71:13:1c actions=output:2
      cookie=0x0, duration=295.679s, table=0, n_packets=0, n_bytes=0, idle_age=295, priority=10,in_port=2,dl_vlan=0,dl_src=6e:0c:a2:71:13:1c,dl_dst=e6:e1:fe:2f:2e:ea actions=output:3
    • s6 ------------------------------------------------------------------------
      NXST_FLOW reply (xid=0x4):
    • s7 ------------------------------------------------------------------------
      NXST_FLOW reply (xid=0x4):
      cookie=0x0, duration=295.612s, table=0, n_packets=0, n_bytes=0, idle_age=295, priority=10,in_port=3,dl_vlan=0,dl_src=e6:e1:fe:2f:2e:ea,dl_dst=6e:0c:a2:71:13:1c actions=output:2
      cookie=0x0, duration=295.65s, table=0, n_packets=0, n_bytes=0, idle_timeout=300, idle_age=295, priority=10,in_port=2,dl_vlan=0,dl_src=6e:0c:a2:71:13:1c,dl_dst=e6:e1:fe:2f:2e:ea actions=output:3

6. Wait 300 seconds.

7. Check flow entries in mininet again.

The two flow entries whose idle_timeout is configured 300 is removed.

mininet> dpctl dump-flows

• • • s1 ------------------------------------------------------------------------
      NXST_FLOW reply (xid=0x4):
      cookie=0x0, duration=365.604s, table=0, n_packets=0, n_bytes=0, idle_age=365, priority=10,in_port=1,dl_vlan=0,dl_src=e6:e1:fe:2f:2e:ea,dl_dst=6e:0c:a2:71:13:1c actions=output:2
      cookie=0x0, duration=365.679s, table=0, n_packets=0, n_bytes=0, idle_age=365, priority=10,in_port=2,dl_vlan=0,dl_src=6e:0c:a2:71:13:1c,dl_dst=e6:e1:fe:2f:2e:ea actions=output:1
    • s2 ------------------------------------------------------------------------
      NXST_FLOW reply (xid=0x4):
      cookie=0x0, duration=365.576s, table=0, n_packets=0, n_bytes=0, idle_age=365, priority=10,in_port=1,dl_vlan=0,dl_src=e6:e1:fe:2f:2e:ea,dl_dst=6e:0c:a2:71:13:1c actions=output:3
      cookie=0x0, duration=365.646s, table=0, n_packets=0, n_bytes=0, idle_age=365, priority=10,in_port=3,dl_vlan=0,dl_src=6e:0c:a2:71:13:1c,dl_dst=e6:e1:fe:2f:2e:ea actions=output:1
    • s3 ------------------------------------------------------------------------
      NXST_FLOW reply (xid=0x4):
      cookie=0x0, duration=365.721s, table=0, n_packets=0, n_bytes=0, idle_age=365, priority=10,in_port=3,dl_vlan=0,dl_src=6e:0c:a2:71:13:1c,dl_dst=e6:e1:fe:2f:2e:ea actions=output:1
    • s4 ------------------------------------------------------------------------
      NXST_FLOW reply (xid=0x4):
    • s5 ------------------------------------------------------------------------
      NXST_FLOW reply (xid=0x4):
      cookie=0x0, duration=365.603s, table=0, n_packets=0, n_bytes=0, idle_age=365, priority=10,in_port=3,dl_vlan=0,dl_src=e6:e1:fe:2f:2e:ea,dl_dst=6e:0c:a2:71:13:1c actions=output:2
      cookie=0x0, duration=365.667s, table=0, n_packets=0, n_bytes=0, idle_age=365, priority=10,in_port=2,dl_vlan=0,dl_src=6e:0c:a2:71:13:1c,dl_dst=e6:e1:fe:2f:2e:ea actions=output:3
    • s6 ------------------------------------------------------------------------
      NXST_FLOW reply (xid=0x4):
    • s7 ------------------------------------------------------------------------
      NXST_FLOW reply (xid=0x4):
      cookie=0x0, duration=365.6s, table=0, n_packets=0, n_bytes=0, idle_age=365, priority=10,in_port=3,dl_vlan=0,dl_src=e6:e1:fe:2f:2e:ea,dl_dst=6e:0c:a2:71:13:1c actions=output:2

I captured all packets on the OpenFlow channels by using tcpdump, and confirmed that switches sent FLOW_REMOVED messages to the controller.
However, IFlowProgrammerListener.flowRemoved(Node, Flow) was not invoked, so VTN Manager didn't notice that.