I got by the "A message body writer..." error by adding:
<dependency>
<groupId>com.sun.jersey</groupId>
<artifactId>jersey-json</artifactId>
</dependency>
to the odl-neutron-northbound-api feature pom based on https://stackoverflow.com/questions/13108161/a-message-body-writer-for-java-class-not-found but now I get:
2018-01-13T02:18:44,772 | ERROR | qtp1602373834-139 | ContainerResponse | 35 - com.sun.jersey.jersey-server - 1.17.0 | Mapped exception to response: 500 (Internal Server Error)
javax.ws.rs.WebApplicationException: javax.xml.bind.JAXBException
- with linked exception:
[java.lang.ClassNotFoundException: com.sun.xml.bind.v2.ContextFactory cannot be found by org.opendaylight.neutron.northbound-api_0.10.0.SNAPSHOT]
at com.sun.jersey.core.provider.jaxb.AbstractRootElementProvider.writeTo(AbstractRootElementProvider.java:159) [34:com.sun.jersey.core:1.17.0]
at com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:306) [35:com.sun.jersey.jersey-server:1.17.0]
I got by that by adding com.sun.xml.bind.v2 to Import-Package but now get another failure:
java.lang.AssertionError: E2E Tests Failed - Collection not Array
at org.opendaylight.neutron.e2etest.ITNeutronE2E.test_fetch_collection_response(ITNeutronE2E.java:276)
at org.opendaylight.neutron.e2etest.ITNeutronE2E.test_fetch_with_one_query_item(ITNeutronE2E.java:296)
at org.opendaylight.neutron.e2etest.ITNeutronE2E.test(ITNeutronE2E.java:95)
This is probably related to https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.8 and more specifically to https://github.com/FasterXML/jackson-jaxrs-providers/issues/22 .
https://git.opendaylight.org/gerrit/#/c/67165/ from skitt fixes the ConfigPusher (which has nothing at all directly to do with this, but initially seemed to be the reason for the ITNeutronE2E failure, but turned out not to be).
The "Collection not Array" is because that jsonElementValue now suddenly is not a JSOn array anymore, so somehow Jersey is behaving differently after the bump; I've added logging to the test and see that it now is this: "id":"4e8e5957-649f-477b-9e5b-f1f75b21c03c","tenant_id":"9bacb3c5d39d41a79512987f338cf177","name":"net1","admin_state_up":"true","status":"ACTIVE","shared":"false","external":"false","network_type":"flat","segments":[null,null]. I know nothing at all about neutron code, but will try to understand if this is some... mapping annotation that needs to be adjusted?
Well, I think this has something to do with MOXyJsonProvider, as that seems to be the thing that is supposed to take care of JSON data. I suspect adding jersey-json changes the provider behavior...
Right, so the full response we get froma GET on http://127.0.0.1:8181/controller/nb/v2/neutron/networks?status=ACTIVE is:
{"networks":{"id":"4e8e5957-649f-477b-9e5b-f1f75b21c03c","tenant_id":"9bacb3c5d39d41a79512987f338cf177","name":"net1","admin_state_up":"true","status":"ACTIVE","shared":"false","external":"false","network_type":"flat","segments":[null,null]}}
which in the code comes from the class NeutronNetworksNorthbound which returns a NeutronNetworkRequest (yup, you read that right, that appears to be the Response despite the class being named Request!). Now that NeutronNetworkRequest has an @XmlElement(name = "networks") List<NeutronNetwork> which due to some change in the JSON serialization with this bump is no longer serialized as a JSON array despite being a List. Hm.
Today's free fun fact: If we hack an bulk.add(new NeutronNetwork()); in line 42 of NeutronNetworkRequest then it is a JSON array! I'm obviously not suggesting that be the fix, but proving that there is some... wrong optimization in whatever JSON serializer that is being use here that thinks that is a List contains only 1 element it can be written out as just JSON object instead of array. Huh.
Perhaps the better next step here would be to truly understand how the Big Bump of odlparent and yangtools is causing this change...
Yeah, that is codec weirdness/optimization of flattening a collection to a single element? At any rate we should figure out why moxy no longer works at it should.
Another thing is the question of whether moxy was ever used at all the breakage actually comes from upgraded jackson 
So to sum this up:
- in default configuration MOXy does not seem to be used to perform serialization (anymore?, that needs to be confirmed vs. nitrogen)
- with jersey-json we get serialization working, but it is broken around single-item lists
Hence we either need to get moxy working or replace it with a different JSON provider – for AAA we have the proposal to use GSON at https://git.opendaylight.org/gerrit/#/c/66056/.
https://www.eclipse.org/eclipselink/api/2.4/org/eclipse/persistence/jaxb/MarshallerProperties.html#JSON_REDUCE_ANY_ARRAYS looks interesting... but yes rovarga you raise a good point, this seems to be a in complete mess now - I'm myself confused which JSON serializer (codec) is actually used here now.
Note the presence of neutron/northbound-api/src/main/resources/org/opendaylight/controller/networkconfig/neutron/northbound/jaxb.properties, but that is in a weird package, given that NeutronNetworkRequest and Co. are in org.opendaylight.neutron.northbound.api, no? Was this project (neutron) originally using Moxy and by adding jersey-json we've made it use Jackson with (non-Moxy) JAXB instead, causing this (and probably other...) issues? Perhaps instead of fixing JSON array problem in isolation, we should take a step back and understand better what's really going on here. The jersey-json dependency may well not actually be the right solution here.
But the real question probably is what actually changed in odlparent 3.0.2 compared to 2.0.x to cause the original issue... suggestions, anyone?
> Hence we either need to get moxy working or replace it with a different JSON provider – for AAA we have the proposal to use GSON
given that the world is broken and a lot of people cannot work on master, I think the priority here should be to get neutron working as it used to before the odlparent bump, which (I think) means understanding what makes Jersey use Moxy and how that broke compared to how it worked before the bump. I'm not clear if that means that jersey-json is a bad idea - we probably need to understand how jersey can be made to work with moxy; how it did before, why it broken, and how to get it back working?
Actually replacing it with a different JSON serializer, like GSON, seems safer to do separately later - who knows what other surprises that may lead to.
The only related change I am aware of the upgrade from jackson-2.3.2 to jackson-2.8.9...
Given that https://git.opendaylight.org/gerrit/#/c/67168/ , I am not sure the setup story in this project was completely correct to begin with.
So the NeutronNorthboundRSApplication configures a MOXyJsonProvider, so Neutron clearly originally wanted to use Moxy instead Jackson...
jaxb.properties seems to be correct accordking to https://www.eclipse.org/eclipselink/documentation/2.6/moxy/runtime001.htm#CACFEGHC .
> Given that https://git.opendaylight.org/gerrit/#/c/67168/ , I am not sure the setup story in this project was completely correct to begin with.
Interesting but my suspicion would be that's more of a minor oversight leading to that WARN log, but not the real cause of / related to the problem we're chasing here? The MOXyJsonProvider seems to ahave been configured both as class as well as instance, and it looks like if it's configured as instance then one does not need to configure it as class, which makes sense.
Let us remove the jersey-json dependency again, and try to understand the real cause of the initial WebApplicationException: com.sun.jersey.api.MessageException - despite the configuration of the MOXyJsonProvider ?
One more possibility is that we have a clash on javax.ws.rs(.ext) package and we end up not interpreting MOXyJsonProvider as a correct Provider...
Yup, confirmed here:
javax.ws.rs.client │ 2.0.1 │ 68 │ javax.ws.rs-api
javax.ws.rs.container │ 2.0.1 │ 68 │ javax.ws.rs-api
javax.ws.rs.core │ 1.1.1 │ 34 │ com.sun.jersey.core
javax.ws.rs.core │ 2.0.1 │ 68 │ javax.ws.rs-api
javax.ws.rs.ext │ 1.1.1 │ 34 │ com.sun.jersey.core
javax.ws.rs.ext │ 2.0.1 │ 68 │ javax.ws.rs-api
javax.ws.rs │ 1.1.1 │ 34 │ com.sun.jersey.core
javax.ws.rs │ 2.0.1 │ 68 │ javax.ws.rs-api
> jaxb.properties seems to be correct accordking to https://www.eclipse.org/eclipselink/documentation/2.6/moxy/runtime001.htm#CACFEGHC .
I don't think so; note "the same package (directory) in which your model classes reside", and it's in an old org/opendaylight/controller/networkconfig/neutron/northbound/ instead of in org/opendaylight/neutron/northbound/api/ where it should be now... but that's not our problem here, that javax.ws.rs.WebApplicationException and com.sun.jersey.api.MessageException where Jersey isn't able to find a Codec for MIME application/json most probably hits us and fails long before any JAXB related stuff? And if the MOXyJsonProvider did work, I suspect that probably has the equivalent of javax.xml.bind.context.factory=org.eclipse.persistence.jaxb.JAXBContextFactory in jaxb.properties in code instead, so I bet that was just wrong since year and we could probably completely remove that jaxb.properties file, wherever it is, and it will still work (once it uses the MOXyJsonProvider).
> One more possibility is that we have a clash on javax.ws.rs(.ext) package and we end up not interpreting MOXyJsonProvider as a correct Provider...
> Yup, confirmed here:
Bloody OSGi! So wait, what does this mean? We got the same package in two bundles with different JAX RS versions? And something sets some stupid static in the wrong one of the two? Phew.
Looking at moxy's headers, it does not specify a version for its javax.ws.rs import:
clipseLink MOXy (176)
----------------------
Archiver-Version = Plexus Archiver
Build-Jdk = 1.7.0_80
Built-By = genie.eclipselink
Created-By = 1.6.0_21 (Sun Microsystems Inc.)
HK2-Bundle-Name = org.eclipse.persistence:org.eclipse.persistence.moxy
Manifest-Version = 1.0
Bundle-ManifestVersion = 2
Bundle-Name = EclipseLink MOXy
Bundle-SymbolicName = org.eclipse.persistence.moxy
Bundle-Vendor = Eclipse.org - EclipseLink Project
Bundle-Version = 2.6.2.v20151217-774c696
Export-Package =
org.eclipse.persistence.internal.jaxb;version=2.6.2,
org.eclipse.persistence.internal.jaxb.many;version=2.6.2,
org.eclipse.persistence.jaxb;version=2.6.2,
org.eclipse.persistence.jaxb.attachment;version=2.6.2,
org.eclipse.persistence.jaxb.compiler;version=2.6.2,
org.eclipse.persistence.jaxb.dynamic;version=2.6.2,
org.eclipse.persistence.jaxb.dynamic.metadata;version=2.6.2,
org.eclipse.persistence.jaxb.javamodel;version=2.6.2,
org.eclipse.persistence.jaxb.javamodel.oxm;version=2.6.2,
org.eclipse.persistence.jaxb.javamodel.reflection;version=2.6.2,
org.eclipse.persistence.jaxb.metadata;version=2.6.2,
org.eclipse.persistence.jaxb.rs;version=2.6.2,
org.eclipse.persistence.jaxb.xmlmodel;version=2.6.2
Import-Package =
com.sun.xml.bind;resolution:=optional,
com.sun.xml.bind.annotation;resolution:=optional,
com.sun.xml.bind.api;resolution:=optional,
com.sun.xml.bind.api.impl;resolution:=optional,
com.sun.codemodel;resolution:=optional;version="[2.2.11,3)",
com.sun.xml.xsom;resolution:=optional,
com.sun.xml.xsom.impl;resolution:=optional,
com.sun.xml.xsom.impl.parser;resolution:=optional,
com.sun.tools.xjc;resolution:=optional;version="[2.2.11,3)",
com.sun.tools.xjc.model;resolution:=optional;version="[2.2.11,3)",
com.sun.tools.xjc.outline;resolution:=optional;version="[2.2.11,3)",
javax.activation;resolution:=optional,
javax.json;resolution:=optional,
javax.json.stream;resolution:=optional,
javax.naming;resolution:=optional,
javax.validation;resolution:=optional;version=1.1.0,
javax.validation.constraints;resolution:=optional;version=1.1.0,
javax.validation.groups;resolution:=optional;version=1.1.0,
javax.ws.rs;resolution:=optional,
javax.ws.rs.core;resolution:=optional,
javax.ws.rs.ext;resolution:=optional,
Hence I suspect it binds to 2.0.1 and jersey no longer recognizes it.
Yup, it is precisely that:
karaf@root()> bundle:requirements 176
org.eclipse.persistence.moxy_2.6.2.v20151217-774c696 [176] requires:
--------------------------------------------------------------------
osgi.wiring.package; (osgi.wiring.package=javax.activation) resolved by:
osgi.wiring.package; javax.activation 1.1.0 from org.eclipse.osgi_3.11.3.v20170209-1843 [0]
osgi.wiring.package; (osgi.wiring.package=javax.json) resolved by:
osgi.wiring.package; javax.json 1.0.0 from org.glassfish.javax.json_1.0.4 [179]
osgi.wiring.package; (osgi.wiring.package=javax.json.stream) resolved by:
osgi.wiring.package; javax.json.stream 1.0.0 from org.glassfish.javax.json_1.0.4 [179]
osgi.wiring.package; (osgi.wiring.package=javax.naming) resolved by:
osgi.wiring.package; javax.naming 0.0.0 from org.eclipse.osgi_3.11.3.v20170209-1843 [0]
osgi.wiring.package; (&(osgi.wiring.package=javax.validation)(version>=1.1.0)) resolved by:
osgi.wiring.package; javax.validation 1.1.0.Final from javax.validation.api_1.1.0.Final [66]
osgi.wiring.package; (&(osgi.wiring.package=javax.validation.constraints)(version>=1.1.0)) resolved by:
osgi.wiring.package; javax.validation.constraints 1.1.0.Final from javax.validation.api_1.1.0.Final [66]
osgi.wiring.package; (&(osgi.wiring.package=javax.validation.groups)(version>=1.1.0)) resolved by:
osgi.wiring.package; javax.validation.groups 1.1.0.Final from javax.validation.api_1.1.0.Final [66]
osgi.wiring.package; (osgi.wiring.package=javax.ws.rs) resolved by:
osgi.wiring.package; javax.ws.rs 2.0.1 from javax.ws.rs-api_2.0.1 [68]
osgi.wiring.package; (osgi.wiring.package=javax.ws.rs.core) resolved by:
osgi.wiring.package; javax.ws.rs.core 2.0.1 from javax.ws.rs-api_2.0.1 [68]
osgi.wiring.package; (osgi.wiring.package=javax.ws.rs.ext) resolved by:
osgi.wiring.package; javax.ws.rs.ext 2.0.1 from javax.ws.rs-api_2.0.1 [68]
Looking at moxy 2.7.1, it will explicitly bind to version 2.0.1, so upgrading it will not solve our problem. Looking at jersey, 1.19.4 no longer packages javax.ws.rs, so that may be path out of this. Alternatively we wrap moxy and force it to resolve javax.ws.rs to 1.1.x.
Confirmed to be present.
> Looking at moxy 2.7.1, it will explicitly bind to version 2.0.1,
was just looking at that as well... we're apparently on moxy 2.6.2, so attempting a bump to 2.7.1 seems reasonable?
So if we want to avoid surprises down the road on the next round of bumps and get this over with, that would be the right thing to do here, IMHO...
> so upgrading it will not solve our problem. Looking at jersey, 1.19.4 no longer packages javax.ws.rs, so that may be path out of this.
that sounds like worth a try?
> Alternatively we wrap moxy and force it to resolve javax.ws.rs to 1.1.x.
yeah, short term alternative; but then we have to remember to remove this work-around again sooner or later, no?
Yeah, so to unblock netvirt I propose to just munge moxy into northbound-api (yeah, it is ugly) and discuss the jersey situation on the kernel projects call. At this point, everything for the upgrade to 2.x should be ready from odlparent and is really all about aaa+netconf and other downstreams...
This just shows how technical debt can explode at the worst possible of times.
As for jersey-1.19 – is probably the middle ground here, but given how much non-progress we have made over the years in moving from 1.17 I would strongly suggest biting the bullet (again) and cleaning the mess once and hopefully for ever.
Just catching up - had to step out for a while. Nice find Robert and Michael.
Thanks for working hard during weekend.
For Florine, let's fix it right uniformly among projects.
FTR: NETCONF-502 is hitting something very similar...