Operating System: All
Platform: All

On the failure build (#63), many RESTCONF requests to VTN Manager failed on 404 errors.
https://jenkins.opendaylight.org/releng/view/vtn/job/vtn-csit-1node-manager-all-beryllium/63/

I've checked the karaf log file on the failure build (#63).
Then, I've found out that the initialization of the VTN Manager was not completed even in 3 minutes after the controller started up.
It seems to me that ODL controller was busy on something wrong so that it didn't reach to the initialization process of the VTN Manager yet.
I think that's why all test cases failed.

That being said, I'm still not sure why the ODL controller took such a lot of time to just install features.

Jamo has gave us his investigation result on the ML.

https://lists.opendaylight.org/pipermail/vtn-dev/2016-February/001294.html

"I tested locally and found this:
with RC2.1, there is a > 7m lag between when I see AAA Initialized and
when vtn sees it's config-pusher Succcessful log message. My assumption
here is that if AAA is Initialized, that's aprox when I'll see a 200
back from /restconf/modules. This appears to be consistent and repeatable."

This means AAA is initialized. You would see 401 or 404 otherwise. This isn't a AAA issue.

Additional Jamo's comment:
"to double check, I ran two more test jobs.
#65 with RC1.2 which fails again.
#66 with latest stable/Be available and it passes.
I think something is broken just in RC2.1."

https://lists.opendaylight.org/pipermail/vtn-dev/2016-February/001302.html

#65 build (CSIT): https://jenkins.opendaylight.org/releng/view/vtn/job/vtn-csit-1node-manager-all-beryllium/65/
#66 build (CSIT): https://jenkins.opendaylight.org/releng/view/vtn/job/vtn-csit-1node-manager-all-beryllium/66/

Luis's comment on the issue:

"
Hi Jamo, I am looking at the karaf log for fail/pass runs, and in both cases the controller is not full up when the test starts, so regardless any other issue, I think we should patch our test code to increase the sleep when we load all compatible features. Something like 3 minutes instead of 1:

• single feature test: 1 min (already there)
• all compatible features test: 3 mins (new patch)
• all features test: 5 mins (already there)

BR/Luis
"

Reference: https://lists.opendaylight.org/pipermail/vtn-dev/2016-February/001306.html

(In reply to Hideyuki Tai from comment #5)
> Luis's comment on the issue:
>
> "
> Hi Jamo, I am looking at the karaf log for fail/pass runs, and in both cases
> the controller is not full up when the test starts, so regardless any other
> issue, I think we should patch our test code to increase the sleep when we
> load all compatible features. Something like 3 minutes instead of 1:
>
> - single feature test: 1 min (already there)
> - all compatible features test: 3 mins (new patch)
> - all features test: 5 mins (already there)
>
> BR/Luis
> "
>
> Reference:
> https://lists.opendaylight.org/pipermail/vtn-dev/2016-February/001306.html

Closing this bug. The RC2.2 distro did not see this issue. I have added a
patch [0] that will give more time for features to come up to prevent false
failures in the future.

https://git.opendaylight.org/gerrit/#/c/34525/

RC2.2 test:
https://jenkins.opendaylight.org/releng/job/vtn-csit-1node-manager-all-beryllium/69/