Let's start leaving comments here about the state of the latest build linked  in the "ODL Last FULL Build:" field...

Proposed format: #XXX : failing in ... due to ... . @user is looking into it / please look into it ASAP. For example:

#45 failing in genius/commons/testutils (due to mdsal impacts?) I'm looking into it ASAP.

I noticed that previous multipatch ran with the quick profile (-Pq), which IMHO is "cheating" ...  so let's separate:

full #52: bgpcep ConfigLoaderImplTest (prev. AddPathBasePathsTest) failure, just raised on bgpcep-dev.  While waiting for that, I will try to manually override multipatch's build order to see what we hit next (see ongoing #55).

quick #54: sfc fails to build, just raised on sfc-dev.

full #55: lispflowmapping SFT failure, just raised on lispflowmapping-dev.  While waiting for that, we could rerun and manually override multipatch's build order again and put lispflowmapping bgpcep after netvirt to see how far we get- but we need to get SFC fixed first (SFC-229; if I find time I'll see if I can help).

quick #56: after hacking SFC, let's see if we get at least a full quick build to go through now...

Full builds are pointless to retry until we hear back from all of 1. bgpcep, 2. lispflowmapping & 3. sfc.

full #58: currently running; following clarification from lispflowmapping (thanks ljakab!), includes mdsal 3.0.0 final (non-SNAPSHOT) adoption in ovsdb and openflowplugin and daexim and genius. I'm expecting it to fail in SFC. Custom builder: infrautils mdsal controller serviceutils aaa netconf daexim ovsdb neutron openflowplugin genius lispflowmapping sfc coe netvirt bgpcep

quick #56: now hits compilation build failure in netvirt; just raised on netvirt-dev.

> Full builds are pointless to retry until we hear back from all of 1. bgpcep, 2. lispflowmapping & 3. sfc.
> full #58: currently running (...) I'm expecting it to fail in SFC

full #58: actually already failed on openflowplugin (test failures related to Mockito); I've contacted openflowplugin-dev.

quick #59 passed, following build compilation failure fixed reported by netvirt.

With the blocking problem of Neutron actually have been completely broken finally out of the way (see NEUTRON-197 & NEUTRON-200), we are now reaching the stage where it is worth to start trying out to run netvirt CSIT tests. We can probably do this on both the quick and full builds? I'm unclear if  1. bgpcep, 2. lispflowmapping, 3. sfc, 4. netvirt fixed all tests meanwhile, so I have kicked off both a new quick build (#60) with CSIT and a new full build (#61; no CSIT) to see.

quick #60 failed again in netvirt; curious, see netvirt-dev thread (full #61 likely to fail earlier in some test failure; let's see).

full #61 failed in neutron NeutronE2ETest AssertionError: L2 Gateway Delete Failed expected:<204> but was:<404> - that is not the same problem as the original broken Neutron in NEUTRON-197 (what was Singleton Network Post Failed NB expected:<201> but was:<400> in the IT ITNeutronE2E (not the standalone NeutronE2ETest) ... this passed locally, curious; wondering if it could be a timing issue hitting us on slow multipatch?

quick #62 kicked off, following skitt's (thanks) netvirt fix, now let's see how CSIT looks...

old quick #62 killed due to this (as further CSIT are useless anyway)

new quick #63 with CSIT scheduled, with SFC's broken feature now fixed, let's see what's next

new full #64 also scheduled (also with CSIT, but it will never get that far); NeutronE2ETest disabled, see NEUTRON-200

quick #63 CSIT has failures related to bouncycastle and AAA; see list email.

full #64 fails in a SFT in neutron, exact same problem re. com.google.common.base 23.6.1 VS 25.1.0 I hit 48h in NEUTRON-197 - where it went away (and locally passes) after re-installing pre-req. projects. This makes no sense; it's almost as if the multipatch job, or really our Maven builds, somehow are not really isolated enough and while we build a topic like Neon-MRI we get master artifacts?!

full #65 (no CSIT) is verifying openflowplugin's fixes for test failures; neutron may fail again, need RELENG-135.

quick #66 CSIT let's see if we get past the bouncycastle/AAA integration problem following c/76817.

full #67 without CSIT with integration/distribution added to BUILD_ORDER (as in #66), following the realization that the com.google.common.base 23.6.1 VS 25.1.0 in neutron were due to c/76089 having lost the neon-mri topic accidentially.

quick #68 with CSIT after manually cancelling previous quick #66 due to above.

quick #69 with CSIT after fixing build problem on daexim (due to infrautils) affecting master (not Neon-MRI specific).

full #70 without CSIT after manually cancelling previous full #67 due to above.

quick #71 with CSIT after fixing build problem on ovsdb (due to infrautils) affecting master.

full #72 without CSIT due to above.

quick #75 with CSIT started, after #71 failed due to what https://git.opendaylight.org/gerrit/#/c/76839/ should fix; I've added integration/distribution to BUILD_ORDER again anyway, just to be safe. (Hopefully it won't hit the same maven-antrun-plugin failure re. Directory does not exist: /w/workspace/integration-multipatch-test-neon/patch_tester/distribution/karaf/target/assembly/system/mysql like https://jenkins.opendaylight.org/sandbox/job/integration-multipatch-test-neon/1.)

full #72 failed due to FindBugs error in openflowplugin; just raised on lispflowmapping-dev.

I imagine the int/dist build will fail again because of the mysql workaround; I’ve updated the Neon MRI patch to drop that (it’s obsolete with Karaf 4.2).

quick #76 with CSIT started, to get skitt's fix above (TX) and because #75 failed, probably due to having added integration/distribution to BUILD_ORDER (now no longer going to do that).

I'm signing off off helping with this (going on PTO) but it looks like the efforts over the last week are starting to pay off!

All patches have been updated to pick up yangtools-2.1.2-SNAPSHOT and mdsal-3.0.1-SNAPSHOT, build #81 triggered.

Build #76 is being re-tried as #82.

Patches updated to yangtools-2.1.2/mdsal-3.0.1, builds retriggered: full #83, quick #84.

Build 91 (full with CSIT) failed on bgpcep; I’m guessing this is fixed by the latest updates merged overnight.

Triggered 92 (fast with CSIT), 93 (full without CSIT).

Not yet, more work is needed

ariel.adam full build is complete, what else do we need to merge the patch series?