Topics

R3 OOM - APPC CDT certificate_unknown #appc #oom #casablanca


jkzcristiano
 

Dear all,

After some time working with casablanca release (deployed few days before tag 3.0.2), now I am having issues with CDT. Specifically, when I go to the CDT Web UI and clik on MY VNFS an error appears at the bottom of the screen as follows:



From appc-appc-0 logs it seems another issue with certificates¿?

2019-06-04T14:30:01.725+0000 ERROR [cadi]
2019-06-04T14:30:01.725+0000 ERROR [cadi]
2019-06-04T14:31:01.725+0000 ERROR [cadi]
2019-06-04T14:31:01.725+0000 ERROR [cadi]
2019-06-04T14:31:07.614+0000 AUDIT [cadi] ID admin converted to appc@appc.onap.org
2019-06-04T14:31:07.762+0000 ERROR [cadi] javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown [Ljava.lang.Object;@44782c35
org.onap.aaf.misc.env.APIException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
    at org.onap.aaf.cadi.http.HClient.send(HClient.java:153)
    at org.onap.aaf.cadi.client.Rcli.read(Rcli.java:386)
    at org.onap.aaf.cadi.aaf.v2_0.AAFAuthn$AAFCachedPrincipal.revalidate(AAFAuthn.java:143)
    at org.onap.aaf.cadi.aaf.v2_0.AAFAuthn.validate(AAFAuthn.java:108)
    at org.onap.aaf.cadi.aaf.v2_0.AAFAuthn.validate(AAFAuthn.java:76)
    at org.onap.aaf.cadi.shiro.AAFRealm.doGetAuthenticationInfo(AAFRealm.java:151)
    at org.apache.shiro.realm.AuthenticatingRealm.getAuthenticationInfo(AuthenticatingRealm.java:568)
    at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doSingleRealmAuthentication(ModularRealmAuthenticator.java:180)
    at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doAuthenticate(ModularRealmAuthenticator.java:267)
    at org.apache.shiro.authc.AbstractAuthenticator.authenticate(AbstractAuthenticator.java:198)
    at org.apache.shiro.mgt.AuthenticatingSecurityManager.authenticate(AuthenticatingSecurityManager.java:106)
    at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:270)
    at org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:256)
    at org.apache.shiro.web.filter.authc.AuthenticatingFilter.executeLogin(AuthenticatingFilter.java:53)
    at org.apache.shiro.web.filter.authc.BasicHttpAuthenticationFilter.onAccessDenied(BasicHttpAuthenticationFilter.java:227)
    at org.apache.shiro.web.filter.AccessControlFilter.onAccessDenied(AccessControlFilter.java:133)
    at org.apache.shiro.web.filter.AccessControlFilter.onPreHandle(AccessControlFilter.java:162)
    at org.apache.shiro.web.filter.PathMatchingFilter.isFilterChainContinued(PathMatchingFilter.java:203)
    at org.apache.shiro.web.filter.PathMatchingFilter.preHandle(PathMatchingFilter.java:178)
    at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:131)
    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
    at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
    at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
    at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
    at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
    at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
    at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1751)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
    at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:71)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
    at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:284)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:80)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
    at org.eclipse.jetty.server.Server.handle(Server.java:534)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
    at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
    at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
    at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
    at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
    at java.lang.Thread.run(Thread.java:748)
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
    at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2020)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1127)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
    at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
    at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:162)
    at org.onap.aaf.cadi.http.HClient.send(HClient.java:147)
    ... 54 more
2019-06-04T14:31:12.964+0000 AUDIT [cadi] ID admin converted to appc@appc.onap.org
2019-06-04T14:31:12.983+0000 ERROR [cadi] java.net.SocketException: Broken pipe (Write failed) [Ljava.lang.Object;@127dc39f
org.onap.aaf.misc.env.APIException: java.net.SocketException: Broken pipe (Write failed)
    at org.onap.aaf.cadi.http.HClient.send(HClient.java:153)
    at org.onap.aaf.cadi.client.Rcli.read(Rcli.java:386)


I tried to remove cdt pod, appc-appc-0 pod and aaf-locate pod but it seems a certificate issue. The information in appc-db-0 about past onboarded VNF seems to be present and LCM actions still work. But I cannot add new LCM from CDT anymore nor check onboarded VNFs (it is empty with the error shown above).

I hope some light on this issue!

Kind regards,
Xoan


Taka Cho
 

Xoan,

 

We just found a p12 certificate for AAF authentication that expired by 6/1/2019. We are going to release a R3 SNAPSHOT appc docker image for a new p12 certificate as the highest priority. Once we have done that. I will post on onap-discuss.

 

Taka

 

From: onap-discuss@... <onap-discuss@...> On Behalf Of jkzcristiano
Sent: Tuesday, June 4, 2019 12:26 PM
To: onap-discuss@...
Subject: [onap-discuss] R3 OOM - APPC CDT certificate_unknown #appc #casablanca #oom

 

Dear all,

After some time working with casablanca release (deployed few days before tag 3.0.2), now I am having issues with CDT. Specifically, when I go to the CDT Web UI and clik on MY VNFS an error appears at the bottom of the screen as follows:



From appc-appc-0 logs it seems another issue with certificates¿?

 

2019-06-04T14:30:01.725+0000 ERROR [cadi]

 

2019-06-04T14:30:01.725+0000 ERROR [cadi]

 

2019-06-04T14:31:01.725+0000 ERROR [cadi]

 

2019-06-04T14:31:01.725+0000 ERROR [cadi]

 

2019-06-04T14:31:07.614+0000 AUDIT [cadi] ID admin converted to appc@...

 

2019-06-04T14:31:07.762+0000 ERROR [cadi] javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown [Ljava.lang.Object;@44782c35

 

org.onap.aaf.misc.env.APIException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown

 

    at org.onap.aaf.cadi.http.HClient.send(HClient.java:153)

 

    at org.onap.aaf.cadi.client.Rcli.read(Rcli.java:386)

 

    at org.onap.aaf.cadi.aaf.v2_0.AAFAuthn$AAFCachedPrincipal.revalidate(AAFAuthn.java:143)

 

    at org.onap.aaf.cadi.aaf.v2_0.AAFAuthn.validate(AAFAuthn.java:108)

 

    at org.onap.aaf.cadi.aaf.v2_0.AAFAuthn.validate(AAFAuthn.java:76)

 

    at org.onap.aaf.cadi.shiro.AAFRealm.doGetAuthenticationInfo(AAFRealm.java:151)

 

    at org.apache.shiro.realm.AuthenticatingRealm.getAuthenticationInfo(AuthenticatingRealm.java:568)

 

    at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doSingleRealmAuthentication(ModularRealmAuthenticator.java:180)

 

    at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doAuthenticate(ModularRealmAuthenticator.java:267)

 

    at org.apache.shiro.authc.AbstractAuthenticator.authenticate(AbstractAuthenticator.java:198)

 

    at org.apache.shiro.mgt.AuthenticatingSecurityManager.authenticate(AuthenticatingSecurityManager.java:106)

 

    at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:270)

 

    at org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:256)

 

    at org.apache.shiro.web.filter.authc.AuthenticatingFilter.executeLogin(AuthenticatingFilter.java:53)

 

    at org.apache.shiro.web.filter.authc.BasicHttpAuthenticationFilter.onAccessDenied(BasicHttpAuthenticationFilter.java:227)

 

    at org.apache.shiro.web.filter.AccessControlFilter.onAccessDenied(AccessControlFilter.java:133)

 

    at org.apache.shiro.web.filter.AccessControlFilter.onPreHandle(AccessControlFilter.java:162)

 

    at org.apache.shiro.web.filter.PathMatchingFilter.isFilterChainContinued(PathMatchingFilter.java:203)

 

    at org.apache.shiro.web.filter.PathMatchingFilter.preHandle(PathMatchingFilter.java:178)

 

    at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:131)

 

    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)

 

    at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)

 

    at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)

 

    at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)

 

    at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)

 

    at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)

 

    at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)

 

    at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)

 

    at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)

 

    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1751)

 

    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)

 

    at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:71)

 

    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)

 

    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)

 

    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)

 

    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)

 

    at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:284)

 

    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)

 

    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)

 

    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)

 

    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

 

    at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:80)

 

    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)

 

    at org.eclipse.jetty.server.Server.handle(Server.java:534)

 

    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)

 

    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)

 

    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)

 

    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)

 

    at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)

 

    at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)

 

    at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)

 

    at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)

 

    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)

 

    at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)

 

    at java.lang.Thread.run(Thread.java:748)

 

Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown

 

    at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)

 

    at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)

 

    at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2020)

 

    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1127)

 

    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)

 

    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)

 

    at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)

 

    at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)

 

    at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)

 

    at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:162)

 

    at org.onap.aaf.cadi.http.HClient.send(HClient.java:147)

 

    ... 54 more

 

2019-06-04T14:31:12.964+0000 AUDIT [cadi] ID admin converted to appc@...

 

2019-06-04T14:31:12.983+0000 ERROR [cadi] java.net.SocketException: Broken pipe (Write failed) [Ljava.lang.Object;@127dc39f

 

org.onap.aaf.misc.env.APIException: java.net.SocketException: Broken pipe (Write failed)

 

    at org.onap.aaf.cadi.http.HClient.send(HClient.java:153)

 

    at org.onap.aaf.cadi.client.Rcli.read(Rcli.java:386)

 



I tried to remove cdt pod, appc-appc-0 pod and aaf-locate pod but it seems a certificate issue. The information in appc-db-0 about past onboarded VNF seems to be present and LCM actions still work. But I cannot add new LCM from CDT anymore nor check onboarded VNFs (it is empty with the error shown above).

I hope some light on this issue!

Kind regards,
Xoan


jkzcristiano
 

Dear Taka Cho,

thank you so much for your quick response and help! Hope to see the new snapshot!

Kind regards,
Xoan


Taka Cho
 

Xoan,

 

We made a wiki page https://wiki.onap.org/display/DW/Modify+APPC+Helm+Chart+to+override+the+pk12+certificate can use APPC helm chart to override the expired pk12 AAF client certificate.

 

You would just follow the wiki to override the certificate.

 

Hope that helps.

 

Taka

 

From: jkzcristiano <jkzcristiano@...>
Sent: Tuesday, June 4, 2019 2:25 PM
To: CHO, TAKAMUNE <tc012c@...>; onap-discuss@...
Subject: Re: [onap-discuss] R3 OOM - APPC CDT certificate_unknown #appc #casablanca #oom

 

Dear Taka Cho,

thank you so much for your quick response and help! Hope to see the new snapshot!

Kind regards,
Xoan


jkzcristiano
 

Thank you so much Taka,

with your instructions (+ make appc; make onap; helm deploy dev-appc local/onap --namespace onap -f environments/my-onap-config.yaml) the certificate_unknown issue does not appear now. However, I am still facing the "same issue" when I click on "MY VNFS" in CDT. Logs below:

2019-06-05T14:12:54.269+0000 AUDIT [cadi] ID admin converted to appc@...
2019-06-05T14:12:54.298+0000: Error connecting https://aaf-locate.onap:8095/locate/AAF_NS.service:2.0 for location. 
org.onap.aaf.misc.env.APIException: java.net.SocketException: Broken pipe (Write failed)
at org.onap.aaf.cadi.http.HClient.send(HClient.java:153)
at org.onap.aaf.cadi.aaf.v2_0.AAFLocator.refresh(AAFLocator.java:109)
at org.onap.aaf.cadi.aaf.v2_0.AbsAAFLocator._refresh(AbsAAFLocator.java:210)
at org.onap.aaf.cadi.aaf.v2_0.AbsAAFLocator.hasItems(AbsAAFLocator.java:247)
at org.onap.aaf.cadi.aaf.v2_0.AbsAAFLocator.best(AbsAAFLocator.java:272)
at org.onap.aaf.cadi.aaf.v2_0.AAFConHttp.initURI(AAFConHttp.java:202)
at org.onap.aaf.cadi.aaf.v2_0.AAFCon.client(AAFCon.java:209)
at org.onap.aaf.cadi.aaf.v2_0.AAFCon.client(AAFCon.java:196)
at org.onap.aaf.cadi.aaf.v2_0.AAFAuthn$AAFCachedPrincipal.revalidate(AAFAuthn.java:142)
at org.onap.aaf.cadi.aaf.v2_0.AAFAuthn.validate(AAFAuthn.java:108)
at org.onap.aaf.cadi.aaf.v2_0.AAFAuthn.validate(AAFAuthn.java:76)
at org.onap.aaf.cadi.shiro.AAFRealm.doGetAuthenticationInfo(AAFRealm.java:151)
at org.apache.shiro.realm.AuthenticatingRealm.getAuthenticationInfo(AuthenticatingRealm.java:568)
at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doSingleRealmAuthentication(ModularRealmAuthenticator.java:180)
at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doAuthenticate(ModularRealmAuthenticator.java:267)
at org.apache.shiro.authc.AbstractAuthenticator.authenticate(AbstractAuthenticator.java:198)
at org.apache.shiro.mgt.AuthenticatingSecurityManager.authenticate(AuthenticatingSecurityManager.java:106)
at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:270)
at org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:256)
at org.apache.shiro.web.filter.authc.AuthenticatingFilter.executeLogin(AuthenticatingFilter.java:53)
at org.apache.shiro.web.filter.authc.BasicHttpAuthenticationFilter.onAccessDenied(BasicHttpAuthenticationFilter.java:227)
at org.apache.shiro.web.filter.AccessControlFilter.onAccessDenied(AccessControlFilter.java:133)
at org.apache.shiro.web.filter.AccessControlFilter.onPreHandle(AccessControlFilter.java:162)
at org.apache.shiro.web.filter.PathMatchingFilter.isFilterChainContinued(PathMatchingFilter.java:203)
at org.apache.shiro.web.filter.PathMatchingFilter.preHandle(PathMatchingFilter.java:178)
at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:131)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1751)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:71)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:284)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:80)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.net.SocketException: Broken pipe (Write failed)
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
at sun.security.ssl.OutputRecord.writeBuffer(OutputRecord.java:431)
at sun.security.ssl.OutputRecord.write(OutputRecord.java:417)
at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:879)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:850)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:720)
at sun.security.ssl.Handshaker.sendChangeCipherSpec(Handshaker.java:1144)
at sun.security.ssl.ClientHandshaker.sendChangeCipherAndFinish(ClientHandshaker.java:1280)
2019-06-05T14:12:54.300+0000 ERROR [cadi] No Entries found for 'https://aaf-locate.onap:8095/locate/AAF_NS.service:2.0' [Ljava.lang.Object;@793d31ca
at sun.security.ssl.ClientHandshaker.serverHelloDone(ClientHandshaker.java:1190)
at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:369)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
at sun.security.ssl.Handshaker.process_record(Handshaker.java:965)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:162)
at org.onap.aaf.cadi.http.HClient.send(HClient.java:147)
... 60 more
org.onap.aaf.cadi.LocatorException: No Entries found for 'https://aaf-locate.onap:8095/locate/AAF_NS.service:2.0'
at org.onap.aaf.cadi.aaf.v2_0.AbsAAFLocator.best(AbsAAFLocator.java:273)
at org.onap.aaf.cadi.aaf.v2_0.AAFConHttp.initURI(AAFConHttp.java:202)
at org.onap.aaf.cadi.aaf.v2_0.AAFCon.client(AAFCon.java:209)
at org.onap.aaf.cadi.aaf.v2_0.AAFCon.client(AAFCon.java:196)
at org.onap.aaf.cadi.aaf.v2_0.AAFAuthn$AAFCachedPrincipal.revalidate(AAFAuthn.java:142)
at org.onap.aaf.cadi.aaf.v2_0.AAFAuthn.validate(AAFAuthn.java:108)
at org.onap.aaf.cadi.aaf.v2_0.AAFAuthn.validate(AAFAuthn.java:76)
at org.onap.aaf.cadi.shiro.AAFRealm.doGetAuthenticationInfo(AAFRealm.java:151)
at org.apache.shiro.realm.AuthenticatingRealm.getAuthenticationInfo(AuthenticatingRealm.java:568)
at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doSingleRealmAuthentication(ModularRealmAuthenticator.java:180)
at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doAuthenticate(ModularRealmAuthenticator.java:267)
at org.apache.shiro.authc.AbstractAuthenticator.authenticate(AbstractAuthenticator.java:198)
at org.apache.shiro.mgt.AuthenticatingSecurityManager.authenticate(AuthenticatingSecurityManager.java:106)
at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:270)
at org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:256)
at org.apache.shiro.web.filter.authc.AuthenticatingFilter.executeLogin(AuthenticatingFilter.java:53)
at org.apache.shiro.web.filter.authc.BasicHttpAuthenticationFilter.onAccessDenied(BasicHttpAuthenticationFilter.java:227)
at org.apache.shiro.web.filter.AccessControlFilter.onAccessDenied(AccessControlFilter.java:133)
at org.apache.shiro.web.filter.AccessControlFilter.onPreHandle(AccessControlFilter.java:162)
at org.apache.shiro.web.filter.PathMatchingFilter.isFilterChainContinued(PathMatchingFilter.java:203)
at org.apache.shiro.web.filter.PathMatchingFilter.preHandle(PathMatchingFilter.java:178)
at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:131)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1751)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:71)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:284)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:80)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
2019-06-05T14:12:54.301+0000 ERROR [cadi] Must wait at least 10 seconds for Locator Refresh
2019-06-05T14:12:54.301+0000 ERROR [cadi] org.onap.aaf.cadi.LocatorException: No Entries found for 'https://aaf-locate.onap:8095/locate/AAF_NS.service:2.0' [Ljava.lang.Object;@5bc631ee
at java.lang.Thread.run(Thread.java:748)
org.onap.aaf.cadi.CadiException: org.onap.aaf.cadi.LocatorException: No Entries found for 'https://aaf-locate.onap:8095/locate/AAF_NS.service:2.0'
at org.onap.aaf.cadi.aaf.v2_0.AAFConHttp.rclient(AAFConHttp.java:140)
at org.onap.aaf.cadi.aaf.v2_0.AAFCon.client(AAFCon.java:209)
at org.onap.aaf.cadi.aaf.v2_0.AAFCon.client(AAFCon.java:196)
at org.onap.aaf.cadi.aaf.v2_0.AAFAuthn$AAFCachedPrincipal.revalidate(AAFAuthn.java:142)
at org.onap.aaf.cadi.aaf.v2_0.AAFAuthn.validate(AAFAuthn.java:108)
at org.onap.aaf.cadi.aaf.v2_0.AAFAuthn.validate(AAFAuthn.java:76)
at org.onap.aaf.cadi.shiro.AAFRealm.doGetAuthenticationInfo(AAFRealm.java:151)
at org.apache.shiro.realm.AuthenticatingRealm.getAuthenticationInfo(AuthenticatingRealm.java:568)
at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doSingleRealmAuthentication(ModularRealmAuthenticator.java:180)
at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doAuthenticate(ModularRealmAuthenticator.java:267)
at org.apache.shiro.authc.AbstractAuthenticator.authenticate(AbstractAuthenticator.java:198)
at org.apache.shiro.mgt.AuthenticatingSecurityManager.authenticate(AuthenticatingSecurityManager.java:106)
at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:270)
at org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:256)
at org.apache.shiro.web.filter.authc.AuthenticatingFilter.executeLogin(AuthenticatingFilter.java:53)
at org.apache.shiro.web.filter.authc.BasicHttpAuthenticationFilter.onAccessDenied(BasicHttpAuthenticationFilter.java:227)
at org.apache.shiro.web.filter.AccessControlFilter.onAccessDenied(AccessControlFilter.java:133)
at org.apache.shiro.web.filter.AccessControlFilter.onPreHandle(AccessControlFilter.java:162)
at org.apache.shiro.web.filter.PathMatchingFilter.isFilterChainContinued(PathMatchingFilter.java:203)
at org.apache.shiro.web.filter.PathMatchingFilter.preHandle(PathMatchingFilter.java:178)
at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:131)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)
at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)
at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)
at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)
at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)
at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)
at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1751)
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:71)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:284)
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:80)
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)
at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.onap.aaf.cadi.LocatorException: No Entries found for 'https://aaf-locate.onap:8095/locate/AAF_NS.service:2.0'
at org.onap.aaf.cadi.aaf.v2_0.AbsAAFLocator.best(AbsAAFLocator.java:273)
at org.onap.aaf.cadi.aaf.v2_0.AAFConHttp.rclient(AAFConHttp.java:138)
... 55 more


I might need to undeploy/deploy APPC.

Kind regards,
Xoan


Taka Cho
 

Is it AAF log ?

 

Taka

 

From: onap-discuss@... <onap-discuss@...> On Behalf Of jkzcristiano
Sent: Wednesday, June 5, 2019 10:22 AM
To: CHO, TAKAMUNE <tc012c@...>; onap-discuss@...
Subject: Re: [onap-discuss] R3 OOM - APPC CDT certificate_unknown #appc #casablanca #oom

 

Thank you so much Taka,

with your instructions (+ make appc; make onap; helm deploy dev-appc local/onap --namespace onap -f environments/my-onap-config.yaml) the certificate_unknown issue does not appear now. However, I am still facing the "same issue" when I click on "MY VNFS" in CDT. Logs below:

2019-06-05T14:12:54.269+0000 AUDIT [cadi] ID admin converted to appc@...

2019-06-05T14:12:54.298+0000: Error connecting https://aaf-locate.onap:8095/locate/AAF_NS.service:2.0 for location. 

org.onap.aaf.misc.env.APIException: java.net.SocketException: Broken pipe (Write failed)

at org.onap.aaf.cadi.http.HClient.send(HClient.java:153)

at org.onap.aaf.cadi.aaf.v2_0.AAFLocator.refresh(AAFLocator.java:109)

at org.onap.aaf.cadi.aaf.v2_0.AbsAAFLocator._refresh(AbsAAFLocator.java:210)

at org.onap.aaf.cadi.aaf.v2_0.AbsAAFLocator.hasItems(AbsAAFLocator.java:247)

at org.onap.aaf.cadi.aaf.v2_0.AbsAAFLocator.best(AbsAAFLocator.java:272)

at org.onap.aaf.cadi.aaf.v2_0.AAFConHttp.initURI(AAFConHttp.java:202)

at org.onap.aaf.cadi.aaf.v2_0.AAFCon.client(AAFCon.java:209)

at org.onap.aaf.cadi.aaf.v2_0.AAFCon.client(AAFCon.java:196)

at org.onap.aaf.cadi.aaf.v2_0.AAFAuthn$AAFCachedPrincipal.revalidate(AAFAuthn.java:142)

at org.onap.aaf.cadi.aaf.v2_0.AAFAuthn.validate(AAFAuthn.java:108)

at org.onap.aaf.cadi.aaf.v2_0.AAFAuthn.validate(AAFAuthn.java:76)

at org.onap.aaf.cadi.shiro.AAFRealm.doGetAuthenticationInfo(AAFRealm.java:151)

at org.apache.shiro.realm.AuthenticatingRealm.getAuthenticationInfo(AuthenticatingRealm.java:568)

at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doSingleRealmAuthentication(ModularRealmAuthenticator.java:180)

at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doAuthenticate(ModularRealmAuthenticator.java:267)

at org.apache.shiro.authc.AbstractAuthenticator.authenticate(AbstractAuthenticator.java:198)

at org.apache.shiro.mgt.AuthenticatingSecurityManager.authenticate(AuthenticatingSecurityManager.java:106)

at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:270)

at org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:256)

at org.apache.shiro.web.filter.authc.AuthenticatingFilter.executeLogin(AuthenticatingFilter.java:53)

at org.apache.shiro.web.filter.authc.BasicHttpAuthenticationFilter.onAccessDenied(BasicHttpAuthenticationFilter.java:227)

at org.apache.shiro.web.filter.AccessControlFilter.onAccessDenied(AccessControlFilter.java:133)

at org.apache.shiro.web.filter.AccessControlFilter.onPreHandle(AccessControlFilter.java:162)

at org.apache.shiro.web.filter.PathMatchingFilter.isFilterChainContinued(PathMatchingFilter.java:203)

at org.apache.shiro.web.filter.PathMatchingFilter.preHandle(PathMatchingFilter.java:178)

at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:131)

at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)

at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)

at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)

at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)

at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)

at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)

at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)

at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)

at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1751)

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)

at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:71)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)

at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)

at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:284)

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:80)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)

at org.eclipse.jetty.server.Server.handle(Server.java:534)

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)

at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)

at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)

at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)

at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)

at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)

at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)

at java.lang.Thread.run(Thread.java:748)

Caused by: java.net.SocketException: Broken pipe (Write failed)

at java.net.SocketOutputStream.socketWrite0(Native Method)

at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)

at java.net.SocketOutputStream.write(SocketOutputStream.java:155)

at sun.security.ssl.OutputRecord.writeBuffer(OutputRecord.java:431)

at sun.security.ssl.OutputRecord.write(OutputRecord.java:417)

at sun.security.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:879)

at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:850)

at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:720)

at sun.security.ssl.Handshaker.sendChangeCipherSpec(Handshaker.java:1144)

at sun.security.ssl.ClientHandshaker.sendChangeCipherAndFinish(ClientHandshaker.java:1280)

2019-06-05T14:12:54.300+0000 ERROR [cadi] No Entries found for 'https://aaf-locate.onap:8095/locate/AAF_NS.service:2.0' [Ljava.lang.Object;@793d31ca

at sun.security.ssl.ClientHandshaker.serverHelloDone(ClientHandshaker.java:1190)

at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:369)

at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)

at sun.security.ssl.Handshaker.process_record(Handshaker.java:965)

at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064)

at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)

at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)

at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)

at sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)

at sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)

at sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:162)

at org.onap.aaf.cadi.http.HClient.send(HClient.java:147)

... 60 more

org.onap.aaf.cadi.LocatorException: No Entries found for 'https://aaf-locate.onap:8095/locate/AAF_NS.service:2.0'

at org.onap.aaf.cadi.aaf.v2_0.AbsAAFLocator.best(AbsAAFLocator.java:273)

at org.onap.aaf.cadi.aaf.v2_0.AAFConHttp.initURI(AAFConHttp.java:202)

at org.onap.aaf.cadi.aaf.v2_0.AAFCon.client(AAFCon.java:209)

at org.onap.aaf.cadi.aaf.v2_0.AAFCon.client(AAFCon.java:196)

at org.onap.aaf.cadi.aaf.v2_0.AAFAuthn$AAFCachedPrincipal.revalidate(AAFAuthn.java:142)

at org.onap.aaf.cadi.aaf.v2_0.AAFAuthn.validate(AAFAuthn.java:108)

at org.onap.aaf.cadi.aaf.v2_0.AAFAuthn.validate(AAFAuthn.java:76)

at org.onap.aaf.cadi.shiro.AAFRealm.doGetAuthenticationInfo(AAFRealm.java:151)

at org.apache.shiro.realm.AuthenticatingRealm.getAuthenticationInfo(AuthenticatingRealm.java:568)

at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doSingleRealmAuthentication(ModularRealmAuthenticator.java:180)

at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doAuthenticate(ModularRealmAuthenticator.java:267)

at org.apache.shiro.authc.AbstractAuthenticator.authenticate(AbstractAuthenticator.java:198)

at org.apache.shiro.mgt.AuthenticatingSecurityManager.authenticate(AuthenticatingSecurityManager.java:106)

at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:270)

at org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:256)

at org.apache.shiro.web.filter.authc.AuthenticatingFilter.executeLogin(AuthenticatingFilter.java:53)

at org.apache.shiro.web.filter.authc.BasicHttpAuthenticationFilter.onAccessDenied(BasicHttpAuthenticationFilter.java:227)

at org.apache.shiro.web.filter.AccessControlFilter.onAccessDenied(AccessControlFilter.java:133)

at org.apache.shiro.web.filter.AccessControlFilter.onPreHandle(AccessControlFilter.java:162)

at org.apache.shiro.web.filter.PathMatchingFilter.isFilterChainContinued(PathMatchingFilter.java:203)

at org.apache.shiro.web.filter.PathMatchingFilter.preHandle(PathMatchingFilter.java:178)

at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:131)

at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)

at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)

at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)

at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)

at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)

at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)

at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)

at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)

at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1751)

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)

at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:71)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)

at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)

at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:284)

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:80)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)

at org.eclipse.jetty.server.Server.handle(Server.java:534)

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)

at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)

at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)

at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)

at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)

at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)

at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)

2019-06-05T14:12:54.301+0000 ERROR [cadi] Must wait at least 10 seconds for Locator Refresh

2019-06-05T14:12:54.301+0000 ERROR [cadi] org.onap.aaf.cadi.LocatorException: No Entries found for 'https://aaf-locate.onap:8095/locate/AAF_NS.service:2.0' [Ljava.lang.Object;@5bc631ee

at java.lang.Thread.run(Thread.java:748)

org.onap.aaf.cadi.CadiException: org.onap.aaf.cadi.LocatorException: No Entries found for 'https://aaf-locate.onap:8095/locate/AAF_NS.service:2.0'

at org.onap.aaf.cadi.aaf.v2_0.AAFConHttp.rclient(AAFConHttp.java:140)

at org.onap.aaf.cadi.aaf.v2_0.AAFCon.client(AAFCon.java:209)

at org.onap.aaf.cadi.aaf.v2_0.AAFCon.client(AAFCon.java:196)

at org.onap.aaf.cadi.aaf.v2_0.AAFAuthn$AAFCachedPrincipal.revalidate(AAFAuthn.java:142)

at org.onap.aaf.cadi.aaf.v2_0.AAFAuthn.validate(AAFAuthn.java:108)

at org.onap.aaf.cadi.aaf.v2_0.AAFAuthn.validate(AAFAuthn.java:76)

at org.onap.aaf.cadi.shiro.AAFRealm.doGetAuthenticationInfo(AAFRealm.java:151)

at org.apache.shiro.realm.AuthenticatingRealm.getAuthenticationInfo(AuthenticatingRealm.java:568)

at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doSingleRealmAuthentication(ModularRealmAuthenticator.java:180)

at org.apache.shiro.authc.pam.ModularRealmAuthenticator.doAuthenticate(ModularRealmAuthenticator.java:267)

at org.apache.shiro.authc.AbstractAuthenticator.authenticate(AbstractAuthenticator.java:198)

at org.apache.shiro.mgt.AuthenticatingSecurityManager.authenticate(AuthenticatingSecurityManager.java:106)

at org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:270)

at org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:256)

at org.apache.shiro.web.filter.authc.AuthenticatingFilter.executeLogin(AuthenticatingFilter.java:53)

at org.apache.shiro.web.filter.authc.BasicHttpAuthenticationFilter.onAccessDenied(BasicHttpAuthenticationFilter.java:227)

at org.apache.shiro.web.filter.AccessControlFilter.onAccessDenied(AccessControlFilter.java:133)

at org.apache.shiro.web.filter.AccessControlFilter.onPreHandle(AccessControlFilter.java:162)

at org.apache.shiro.web.filter.PathMatchingFilter.isFilterChainContinued(PathMatchingFilter.java:203)

at org.apache.shiro.web.filter.PathMatchingFilter.preHandle(PathMatchingFilter.java:178)

at org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:131)

at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)

at org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)

at org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449)

at org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365)

at org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90)

at org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83)

at org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383)

at org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362)

at org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)

at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1751)

at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)

at org.ops4j.pax.web.service.jetty.internal.HttpServiceServletHandler.doHandle(HttpServiceServletHandler.java:71)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)

at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)

at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)

at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)

at org.ops4j.pax.web.service.jetty.internal.HttpServiceContext.doHandle(HttpServiceContext.java:284)

at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)

at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)

at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)

at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

at org.ops4j.pax.web.service.jetty.internal.JettyServerHandlerCollection.handle(JettyServerHandlerCollection.java:80)

at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)

at org.eclipse.jetty.server.Server.handle(Server.java:534)

at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:333)

at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)

at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:283)

at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:108)

at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)

at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)

at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)

at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)

at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)

at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)

at java.lang.Thread.run(Thread.java:748)

Caused by: org.onap.aaf.cadi.LocatorException: No Entries found for 'https://aaf-locate.onap:8095/locate/AAF_NS.service:2.0'

at org.onap.aaf.cadi.aaf.v2_0.AbsAAFLocator.best(AbsAAFLocator.java:273)

at org.onap.aaf.cadi.aaf.v2_0.AAFConHttp.rclient(AAFConHttp.java:138)

... 55 more


I might need to undeploy/deploy APPC.

Kind regards,
Xoan


jkzcristiano
 

Hi Taka,

these are logs from appc container in dev-appc-appc-0 pod.

Xoan


Taka Cho
 

It sounds like appc can not connect to that URL... based on appc’s cadi.properties file.

 

2019-06-05T14:12:54.298+0000: Error connecting https://aaf-locate.onap:8095/locate/AAF_NS.service:2.0 for location. 

 

Taka

 

 

From: onap-discuss@... <onap-discuss@...> On Behalf Of jkzcristiano
Sent: Wednesday, June 5, 2019 1:09 PM
To: CHO, TAKAMUNE <tc012c@...>; onap-discuss@...
Subject: Re: [onap-discuss] R3 OOM - APPC CDT certificate_unknown #appc #casablanca #oom

 

Hi Taka,

these are logs from appc container in dev-appc-appc-0 pod.

Xoan


jkzcristiano
 

Dear Taka and all,

some feedback here.

The first time I followed your wiki.

The original file is (case A):

ubuntu@rancher:~/oom/kubernetes$ cat appc/templates/secrets.yaml
# Copyright © 2018  AT&T, Amdocs, Bell Canada Intellectual Property.  All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#       http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
 
apiVersion: v1
kind: Secret
metadata:
  name: {{ include "common.fullname" . }}
  namespace: {{ include "common.namespace" . }}
  labels:
    app: {{ include "common.fullname" . }}
    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
type: Opaque
data:
  db-root-password: {{ .Values.config.mariadbRootPassword | b64enc | quote }}


But I changed it by (case B):

ubuntu@rancher:~/oom/kubernetes$ cat appc/templates/secrets.yaml
# Copyright © 2018  AT&T, Amdocs, Bell Canada Intellectual Property.  All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#       http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
 
apiVersion: v1
kind: Secret
metadata:
   name: {{ include "common.fullname" . }}-certs
   namespace: {{ include "common.namespace" . }}
   labels:
     app: {{ include "common.name" . }}
     chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
     release: {{ .Release.Name }}
     heritage: {{ .Release.Service }}
     type: Opaque
data:
{{ tpl (.Files.Glob "resources/config/certs/*").AsSecrets . | indent 2 }}


and make appc, make onap and deployed (update) APPC. This worked in the sense the unknown_certificate issue was solved.

However, I still had the error I told you (some connectivity issue with AAF).

Then I tried to repeat the process by using this file instead:

ubuntu@rancher:~/oom/kubernetes$ cat appc/templates/secrets.yaml
# Copyright © 2018  AT&T, Amdocs, Bell Canada Intellectual Property.  All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#       http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
 
apiVersion: v1
kind: Secret
metadata:
  name: {{ include "common.fullname" . }}
  namespace: {{ include "common.namespace" . }}
  labels:
    app: {{ include "common.fullname" . }}
    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
    release: {{ .Release.Name }}
    heritage: {{ .Release.Service }}
type: Opaque
data:
  db-root-password: {{ .Values.config.mariadbRootPassword | b64enc | quote }}
---
apiVersion: v1
kind: Secret
metadata:
   name: {{ include "common.fullname" . }}-certs
   namespace: {{ include "common.namespace" . }}
   labels:
     app: {{ include "common.name" . }}
     chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
     release: {{ .Release.Name }}
     heritage: {{ .Release.Service }}
     type: Opaque
data:
{{ tpl (.Files.Glob "resources/config/certs/*").AsSecrets . | indent 2 }}


When I tried to update APPC (with make appc, make onap, helm deploy dev-appc ...)  some error came up (kubectl describe pod/dev-appc-appc-0 -n onap):

Events:
  Type     Reason            Age              From               Message
  ----     ------            ----             ----               -------
  Warning  FailedScheduling  6m               default-scheduler  AssumePod failed: pod 3e3c9094-884f-11e9-884b-02394a5c4c27 is in the cache, so can't be assumed
  Normal   Scheduled         6m               default-scheduler  Successfully assigned onap/dev-appc-appc-0 to k8s-dev
  Warning  FailedScheduling  6m (x2 over 6m)  default-scheduler  pod has unbound PersistentVolumeClaims
  Normal   Pulled            5m               kubelet, k8s-dev   Container image "oomk8s/readiness-check:2.0.0" already present on machine
  Normal   Created           5m               kubelet, k8s-dev   Created container
  Normal   Started           5m               kubelet, k8s-dev   Started container
  Warning  Failed            4m               kubelet, k8s-dev   Error: failed to start container "appc": Error response from daemon: oci runtime error: container_linux.go:247: starting container process caued "process_linux.go:359: container init caused \"rootfs_linux.go:54: mounting \\\"/var/lib/kubelet/pods/3e3c9094-884f-11e9-884b-02394a5c4c27/volume-subpaths/certs/appc/23\\\" to rootfs \\\"/var/lib/docker/afs/mnt/13cb603827f10995f6afad95b98b77ff1959f6cd5c4ae60253909d0e16155403\\\" at \\\"/var/lib/docker/aufs/mnt/13cb603827f10995f6afad95b98b77ff1959f6cd5c4ae60253909d0e16155403/opt/onap/appc/data/stores/org.onapappc.p12\\\" caused \\\"not a directory\\\"\""
: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type
 
I tried to create that directory (oom/kubernetes/appc/resources/config/appc/opt/onap/appc/data/stores/
and locating also the file "org.onapappc.p12" there but the same error happened during helm deploy.

So I finally removed all the steps done in your wiki, undeployed/deployed appc with the option

...
appc:
  enabled: true
  config:
    enableAAF: false
...

in the overriding file. I lost some configuration of course but I only had one VNF in CDT so no issue. No APPC is not using AAF.

Kind regards and thank you for your kind help!
Xoan


Taka Cho
 

I uploaded stateful.yaml and secrets.yaml in the wiki page, those two yamls are working in SB02 in windriver lab. The p12 file should be located at resources/config/certs

 

Can you go into APPC docker try curl https://aaf-locate.onap:8095/locate/AAF_NS.service:2.0 ? or ping that? I am thinking something DNS is wrong in your k8s

 

So the “enableAAF: false” setting is working for you?

 

Taka

 

From: jkzcristiano <jkzcristiano@...>
Sent: Thursday, June 6, 2019 10:53 AM
To: CHO, TAKAMUNE <tc012c@...>; onap-discuss@...
Subject: Re: [onap-discuss] R3 OOM - APPC CDT certificate_unknown #appc #casablanca #oom

 

Dear Taka and all,

some feedback here.

The first time I followed your wiki.

The original file is (case A):

ubuntu@rancher:~/oom/kubernetes$ cat appc/templates/secrets.yaml

# Copyright © 2018  AT&T, Amdocs, Bell Canada Intellectual Property.  All rights reserved.

#

# Licensed under the Apache License, Version 2.0 (the "License");

# you may not use this file except in compliance with the License.

# You may obtain a copy of the License at

#

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

 

apiVersion: v1

kind: Secret

metadata:

  name: {{ include "common.fullname" . }}

  namespace: {{ include "common.namespace" . }}

  labels:

    app: {{ include "common.fullname" . }}

    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}

    release: {{ .Release.Name }}

    heritage: {{ .Release.Service }}

type: Opaque

data:

  db-root-password: {{ .Values.config.mariadbRootPassword | b64enc | quote }}



But I changed it by (case B):

ubuntu@rancher:~/oom/kubernetes$ cat appc/templates/secrets.yaml

# Copyright © 2018  AT&T, Amdocs, Bell Canada Intellectual Property.  All rights reserved.

#

# Licensed under the Apache License, Version 2.0 (the "License");

# you may not use this file except in compliance with the License.

# You may obtain a copy of the License at

#

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

 

apiVersion: v1
kind: Secret
metadata:
   name: {{ include "common.fullname" . }}-certs
   namespace: {{ include "common.namespace" . }}
   labels:
     app: {{ include "common.name" . }}
     chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
     release: {{ .Release.Name }}
     heritage: {{ .Release.Service }}
     type: Opaque
data:
{{ tpl (.Files.Glob "resources/config/certs/*").AsSecrets . | indent 2 }}



and make appc, make onap and deployed (update) APPC. This worked in the sense the unknown_certificate issue was solved.

However, I still had the error I told you (some connectivity issue with AAF).

Then I tried to repeat the process by using this file instead:

ubuntu@rancher:~/oom/kubernetes$ cat appc/templates/secrets.yaml

# Copyright © 2018  AT&T, Amdocs, Bell Canada Intellectual Property.  All rights reserved.

#

# Licensed under the Apache License, Version 2.0 (the "License");

# you may not use this file except in compliance with the License.

# You may obtain a copy of the License at

#

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

 

apiVersion: v1

kind: Secret

metadata:

  name: {{ include "common.fullname" . }}

  namespace: {{ include "common.namespace" . }}

  labels:

    app: {{ include "common.fullname" . }}

    chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}

    release: {{ .Release.Name }}

    heritage: {{ .Release.Service }}

type: Opaque

data:

  db-root-password: {{ .Values.config.mariadbRootPassword | b64enc | quote }}
---
apiVersion: v1
kind: Secret
metadata:
   name: {{ include "common.fullname" . }}-certs
   namespace: {{ include "common.namespace" . }}
   labels:
     app: {{ include "common.name" . }}
     chart: {{ .Chart.Name }}-{{ .Chart.Version | replace "+" "_" }}
     release: {{ .Release.Name }}
     heritage: {{ .Release.Service }}
     type: Opaque
data:
{{ tpl (.Files.Glob "resources/config/certs/*").AsSecrets . | indent 2 }}


When I tried to update APPC (with make appc, make onap, helm deploy dev-appc ...)  some error came up (kubectl describe pod/dev-appc-appc-0 -n onap):

Events:

  Type     Reason            Age              From               Message

  ----     ------            ----             ----               -------

  Warning  FailedScheduling  6m               default-scheduler  AssumePod failed: pod 3e3c9094-884f-11e9-884b-02394a5c4c27 is in the cache, so can't be assumed

  Normal   Scheduled         6m               default-scheduler  Successfully assigned onap/dev-appc-appc-0 to k8s-dev

  Warning  FailedScheduling  6m (x2 over 6m)  default-scheduler  pod has unbound PersistentVolumeClaims

  Normal   Pulled            5m               kubelet, k8s-dev   Container image "oomk8s/readiness-check:2.0.0" already present on machine

  Normal   Created           5m               kubelet, k8s-dev   Created container

  Normal   Started           5m               kubelet, k8s-dev   Started container

  Warning  Failed            4m               kubelet, k8s-dev   Error: failed to start container "appc": Error response from daemon: oci runtime error: container_linux.go:247: starting container process caued "process_linux.go:359: container init caused \"rootfs_linux.go:54: mounting \\\"/var/lib/kubelet/pods/3e3c9094-884f-11e9-884b-02394a5c4c27/volume-subpaths/certs/appc/23\\\" to rootfs \\\"/var/lib/docker/afs/mnt/13cb603827f10995f6afad95b98b77ff1959f6cd5c4ae60253909d0e16155403\\\" at \\\"/var/lib/docker/aufs/mnt/13cb603827f10995f6afad95b98b77ff1959f6cd5c4ae60253909d0e16155403/opt/onap/appc/data/stores/org.onapappc.p12\\\" caused \\\"not a directory\\\"\""

: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

 

I tried to create that directory (oom/kubernetes/appc/resources/config/appc/opt/onap/appc/data/stores/
and locating also the file "org.onapappc.p12" there but the same error happened during helm deploy.

So I finally removed all the steps done in your wiki, undeployed/deployed appc with the option

...
appc:
  enabled: true
  config:
    enableAAF: false
...

in the overriding file. I lost some configuration of course but I only had one VNF in CDT so no issue. No APPC is not using AAF.

Kind regards and thank you for your kind help!
Xoan


jkzcristiano
 

Dear Taka,

thank you for updating the wiki!

I tried to ping & curl from appc container before without success. I was able to ping to pod's IP (10.42...) but when you ping to "aaf-locate.onap" it is resolving the service IP (10.43...).
Yes it seems my setup entered in some inconsistent state in that regard.

Yes, the "enableAAF: false" setting allows me to work with CDT fine again.

Thank you Taka!
Xoan


Taka Cho
 

Great!

 

Appreciated your work for APPC

 

Taka

 

From: jkzcristiano <jkzcristiano@...>
Sent: Thursday, June 6, 2019 12:40 PM
To: CHO, TAKAMUNE <tc012c@...>; onap-discuss@...
Subject: Re: [onap-discuss] R3 OOM - APPC CDT certificate_unknown #appc #casablanca #oom

 

Dear Taka,

thank you for updating the wiki!

I tried to ping & curl from appc container before without success. I was able to ping to pod's IP (10.42...) but when you ping to "aaf-locate.onap" it is resolving the service IP (10.43...).
Yes it seems my setup entered in some inconsistent state in that regard.

Yes, the "enableAAF: false" setting allows me to work with CDT fine again.

Thank you Taka!
Xoan