Operating System: All
Platform: All

> It is possible that this is just a consequence of the preceding BGPCEP-683 symptoms.

It is not, as [3] fixes BGPCEP-683 but this symptom is still visible in gate job tool log [4].

Also, Claudio says this is a Restconf bug, so changing component.

[3] https://git.opendaylight.org/gerrit/#/c/62636/1
[4] https://logs.opendaylight.org/releng/jenkins092/bgpcep-csit-1node-gate-userfeatures-only-nitrogen/7/bgp_app_peer_initial_post_tc1.log.gz

> gate job tool log [4].

Restconf request details copied here.

method: POST

URL: 10.29.12.209:8181/restconf/config/bgp-rib:application-rib/10.29.12.209/tables/bgp-types:ipv4-address-family/bgp-types:unicast-subsequent-address-family/

XML data:
<ipv4-routes xmlns="urn:opendaylight:params:xml:ns:yang:bgp-inet">
<ipv4-route>
<prefix>8.0.1.0/28</prefix>
<path-id>0</path-id>
<attributes>
<ipv4-next-hop>
<global>192.0.2.1</global>
</ipv4-next-hop><as-path/>
<multi-exit-disc>
<med>0</med>
</multi-exit-disc>
<local-pref>
<pref>100</pref>
</local-pref>
<originator-id>
<originator>41.41.41.41</originator>
</originator-id>
<origin>
<value>igp</value>
</origin>
<cluster-id>
<cluster>40.40.40.40</cluster>
</cluster-id>
</attributes>
</ipv4-route>
<ipv4-route>
<prefix>8.0.1.16/28</prefix>
<path-id>0</path-id>
<attributes>
<ipv4-next-hop>
<global>192.0.2.1</global>
</ipv4-next-hop><as-path/>
<multi-exit-disc>
<med>0</med>
</multi-exit-disc>
<local-pref>
<pref>100</pref>
</local-pref>
<originator-id>
<originator>41.41.41.41</originator>
</originator-id>
<origin>
<value>igp</value>
</origin>
<cluster-id>
<cluster>40.40.40.40</cluster>
</cluster-id>
</attributes>
</ipv4-route>
<ipv4-route>
<prefix>8.0.1.32/28</prefix>
<path-id>0</path-id>
<attributes>
<ipv4-next-hop>
<global>192.0.2.1</global>
</ipv4-next-hop><as-path/>
<multi-exit-disc>
<med>0</med>
</multi-exit-disc>
<local-pref>
<pref>100</pref>
</local-pref>
<originator-id>
<originator>41.41.41.41</originator>
</originator-id>
<origin>
<value>igp</value>
</origin>
<cluster-id>
<cluster>40.40.40.40</cluster>
</cluster-id>
</attributes>
</ipv4-route>
</ipv4-routes>

I have tried to reproduce this manually, but it did not happen for me.
I think Robot suite catches ODL in some transient state before it finishes booting up.

I will continue my investigations on Sandbox.

> I will continue my investigations on Sandbox.

When run with a logging change [5], the test passed [6].

Meanwhile, Robert proposed [7] a guard (so no increased logging as with my change), running that now.

[5] https://git.opendaylight.org/gerrit/#/c/62670/1
[6] https://logs.opendaylight.org/sandbox/jenkins091/bgpcep-csit-1node-userfeatures-only-nitrogen/1/bgp_app_peer_initial_post_tc1.log.gz
[7] https://git.opendaylight.org/gerrit/#/c/62699/1

Testing with logging change [8] shows the reason is XML codec getting apparently empty input stream.

By "shows" I mean karaf.log [9] from sandbox contains the incriminating line:

2017-09-06 12:56:22,244 | WARN | qtp444030204-142 | XmlNormalizedNodeBodyReader | 314 - org.opendaylight.netconf.sal-rest-connector - 1.6.0.SNAPSHOT | Reading empty payload!

[8] https://git.opendaylight.org/gerrit/#/c/62752/3
[9] https://logs.opendaylight.org/sandbox/jenkins091/bgpcep-csit-1node-userfeatures-only-nitrogen/5/odl1_karaf.log.gz

One difference from other suites is that the Restconf request is constructed by a Python utility [10] (instead of a Robot library) which sends data in chunks.
Perhaps Carbon http server was joining this chunks for Restconf, but Nitrogen passes the input stream without reading all the data?

[10] https://github.com/opendaylight/integration-test/blob/master/tools/fastbgp/bgp_app_peer.py

NETCONF-467 might be a duplicate, or at least a related problem
https://bugs.opendaylight.org/show_bug.cgi?id=9125

(In reply to Vratko Polák from comment #6)
> One difference from other suites is that the Restconf request is constructed
> by a Python utility [10] (instead of a Robot library) which sends data in
> chunks.
> Perhaps Carbon http server was joining this chunks for Restconf, but
> Nitrogen passes the input stream without reading all the data?

Yeah, it looks like this is the case. Our implementation checks for empty payload incorrectly. In a case that the first chunk of payload's data is unavailable during processing request by our implementations of MessageBodyReader, it prematurely sorts out non-empty payload as empty payload. We read NormalizedNodeContext with null data and this ends with NPE from md-sal, because we want to store null data.

Fix for Nitrogen:
https://git.opendaylight.org/gerrit/#/c/63048/

Oxygen:
https://git.opendaylight.org/gerrit/#/c/63055/