R3 OOM - Policy certificate issue #oom #casablanca #policyr


jkzcristiano
 

Dear all,

I've been working on my ~3.0.2 ONAP instance 49 days. Policy was working fine until last week. Today, I tried to run a CL operation but alarms do not appear in network.log anymore (they are being published, checked from Postman). Seems policy is trying to get messages from Dmaap without success due to a SSLException.

Policy debug.log file is showing these errors periorically for several topics:


[2019-06-10T13:51:35.772+00:00|INFO|ROOT|qtp1528923159-182] 2019-06-10T13:51:35.772+0000 DEBUG [cadi] DenialOfServiceTaf: Not processing this transaction: This Transaction is not denied, ms=0.003419
[2019-06-10T13:51:35.772+00:00|INFO|ROOT|qtp1528923159-182] 2019-06-10T13:51:35.772+0000 DEBUG [cadi] X509Taf: No Certificate Info on Transaction, ms=0.010566
[2019-06-10T13:51:35.772+00:00|INFO|ROOT|qtp1528923159-182] 2019-06-10T13:51:35.772+0000 DEBUG [cadi] BasicHttpTaf: demo@... authenticated by cached AAF password, ms=0.041259
[2019-06-10T13:51:35.772+00:00|INFO|ROOT|qtp1528923159-182] 2019-06-10T13:51:35.772+0000 INFO [cadi] Authenticated: demo@... authenticated by cached AAF password from 10.42.216.188:45922
[2019-06-10T13:51:35.772+00:00|INFO|ROOT|qtp1528923159-182] 2019-06-10T13:51:35.772+0000 DEBUG [cadi] isUserInRole: demo@... has org.onap.policy.pdpd.healthcheck|dev-policy-drools-0|get, 0.038811 ms
[2019-06-10T13:51:35.773+00:00|INFO|AuthorizationFilter|qtp1528923159-182] user Basic Authorization for demo@... evaluated on Mon Jun 10 13:51:35 UTC 2019 in role org.onap.policy.pdpd.healthcheck|dev-policy-drools-0|get is authorized to GET
[2019-06-10T13:51:35.788+00:00|INFO|ROOT|qtp1528923159-182] 2019-06-10T13:51:35.788+0000 WARN [cadi] Trans: user=demo@...[BAth],ip=10.42.216.188,ms=15.877652,validate=0.351069,code=15.510055
[2019-06-10T13:51:35.901+00:00|ERROR|InlineBusTopicSink|DMAAP-source-unauthenticated.DCAE_CL_OUTPUT] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=dcae.policy.shared, consumerInstance=dev-policy-drools-0, fetchTimeout=15000, fetchLimit=100, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-unauthenticated.DCAE_CL_OUTPUT,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=, apiSecret=, useHttps=true, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=unauthenticated.DCAE_CL_OUTPUT, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch because of
javax.net.ssl.SSLException: Received fatal alert: internal_error
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
        at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2038)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1135)
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396)
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355)
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373)
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708)
        at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384)
        at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368)
        at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87)
        at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64)
        at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:172)
        at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:224)
        at java.lang.Thread.run(Thread.java:748)
[2019-06-10T13:51:35.902+00:00|INFO|CambriaConsumerImpl|DMAAP-source-unauthenticated.DCAE_CL_OUTPUT] UEB GET /events/unauthenticated.DCAE_CL_OUTPUT/dcae.policy.shared/dev-policy-drools-0?timeout=15000&limit=100
[2019-06-10T13:51:35.902+00:00|WARN|HostSelector|DMAAP-source-unauthenticated.DCAE_CL_OUTPUT] All hosts were blacklisted; reverting to full set of hosts.
[2019-06-10T13:51:35.902+00:00|INFO|HttpClient|DMAAP-source-unauthenticated.DCAE_CL_OUTPUT] GET https://message-router:3905/events/unauthenticated.DCAE_CL_OUTPUT/dcae.policy.shared/dev-policy-drools-0?timeout=15000&limit=100 (anonymous) ...
[2019-06-10T13:51:35.910+00:00|WARN|HttpClient|DMAAP-source-unauthenticated.DCAE_CL_OUTPUT] Error executing HTTP request. Received fatal alert: internal_error; blacklisting for 2 minutes
[2019-06-10T13:51:35.910+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-unauthenticated.DCAE_CL_OUTPUT] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of Received fatal alert: internal_error - backoff for 15000 ms.
[2019-06-10T13:51:36.165+00:00|ERROR|InlineBusTopicSink|DMAAP-source-SDNR-CL-RSP] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=eb1159c8-7e19-4ca0-b49f-366464176e03, consumerInstance=dev-policy-drools-0, fetchTimeout=15000, fetchLimit=100, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-SDNR-CL-RSP,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=, apiSecret=, useHttps=true, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=SDNR-CL-RSP, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch because of
javax.net.ssl.SSLException: Received fatal alert: internal_error
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
        at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2038)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1135)
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396)
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355)
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373)
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708)
        at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384)
        at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368)
        at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87)
        at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64)
        at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:172)
        at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:224)
        at java.lang.Thread.run(Thread.java:748)
[2019-06-10T13:51:36.165+00:00|INFO|CambriaConsumerImpl|DMAAP-source-SDNR-CL-RSP] UEB GET /events/SDNR-CL-RSP/eb1159c8-7e19-4ca0-b49f-366464176e03/dev-policy-drools-0?timeout=15000&limit=100
[2019-06-10T13:51:36.165+00:00|WARN|HostSelector|DMAAP-source-SDNR-CL-RSP] All hosts were blacklisted; reverting to full set of hosts.
[2019-06-10T13:51:36.165+00:00|INFO|HttpClient|DMAAP-source-SDNR-CL-RSP] GET https://message-router:3905/events/SDNR-CL-RSP/eb1159c8-7e19-4ca0-b49f-366464176e03/dev-policy-drools-0?timeout=15000&limit=100 (anonymous) ...
[2019-06-10T13:51:36.167+00:00|ERROR|InlineBusTopicSink|DMAAP-source-APPC-LCM-WRITE] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=e9a291f0-086d-4bca-b84e-0d9d93dbc180, consumerInstance=dev-policy-drools-0, fetchTimeout=15000, fetchLimit=100, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-APPC-LCM-WRITE,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=, apiSecret=, useHttps=true, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=APPC-LCM-WRITE, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch because of
javax.net.ssl.SSLException: Received fatal alert: internal_error
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
        at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2038)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1135)
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396)
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355)
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373)
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708)
        at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384)
        at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368)
        at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87)
        at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64)
        at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:172)
        at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:224)
        at java.lang.Thread.run(Thread.java:748)
[2019-06-10T13:51:36.167+00:00|INFO|CambriaConsumerImpl|DMAAP-source-APPC-LCM-WRITE] UEB GET /events/APPC-LCM-WRITE/e9a291f0-086d-4bca-b84e-0d9d93dbc180/dev-policy-drools-0?timeout=15000&limit=100
[2019-06-10T13:51:36.167+00:00|WARN|HostSelector|DMAAP-source-APPC-LCM-WRITE] All hosts were blacklisted; reverting to full set of hosts.
[2019-06-10T13:51:36.167+00:00|INFO|HttpClient|DMAAP-source-APPC-LCM-WRITE] GET https://message-router:3905/events/APPC-LCM-WRITE/e9a291f0-086d-4bca-b84e-0d9d93dbc180/dev-policy-drools-0?timeout=15000&limit=100 (anonymous) ...
[2019-06-10T13:51:36.172+00:00|WARN|HttpClient|DMAAP-source-APPC-LCM-WRITE] Error executing HTTP request. Received fatal alert: internal_error; blacklisting for 2 minutes
[2019-06-10T13:51:36.172+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-APPC-LCM-WRITE] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of Received fatal alert: internal_error - backoff for 15000 ms.
[2019-06-10T13:51:36.173+00:00|WARN|HttpClient|DMAAP-source-SDNR-CL-RSP] Error executing HTTP request. Received fatal alert: internal_error; blacklisting for 2 minutes
[2019-06-10T13:51:36.173+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-SDNR-CL-RSP] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of Received fatal alert: internal_error - backoff for 15000 ms.
[2019-06-10T13:51:39.929+00:00|ERROR|InlineBusTopicSink|DMAAP-source-PDPD-CONFIGURATION] SingleThreadedDmaapTopicSource [userName=null, password=-, getTopicCommInfrastructure()=DMAAP, toString()=SingleThreadedBusTopicSource [consumerGroup=12e77824-cba7-48dc-8251-b80b76529ead, consumerInstance=dev-policy-drools-0, fetchTimeout=15000, fetchLimit=100, consumer=CambriaConsumerWrapper [fetchTimeout=15000], alive=true, locked=false, uebThread=Thread[DMAAP-source-PDPD-CONFIGURATION,5,main], topicListeners=1, toString()=BusTopicBase [apiKey=, apiSecret=, useHttps=true, allowSelfSignedCerts=false, toString()=TopicBase [servers=[message-router], topic=PDPD-CONFIGURATION, #recentEvents=0, locked=false, #topicListeners=1]]]]: cannot fetch because of
javax.net.ssl.SSLException: Received fatal alert: internal_error
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
        at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2038)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1135)
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396)
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355)
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373)
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:708)
        at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:384)
        at com.att.nsa.apiClient.http.HttpClient.get(HttpClient.java:368)
        at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:87)
        at com.att.nsa.cambria.client.impl.CambriaConsumerImpl.fetch(CambriaConsumerImpl.java:64)
        at org.onap.policy.common.endpoints.event.comm.bus.internal.BusConsumer$CambriaConsumerWrapper.fetch(BusConsumer.java:172)
        at org.onap.policy.common.endpoints.event.comm.bus.internal.SingleThreadedBusTopicSource.run(SingleThreadedBusTopicSource.java:224)
        at java.lang.Thread.run(Thread.java:748)
[2019-06-10T13:51:39.929+00:00|INFO|CambriaConsumerImpl|DMAAP-source-PDPD-CONFIGURATION] UEB GET /events/PDPD-CONFIGURATION/12e77824-cba7-48dc-8251-b80b76529ead/dev-policy-drools-0?timeout=15000&limit=100
[2019-06-10T13:51:39.929+00:00|WARN|HostSelector|DMAAP-source-PDPD-CONFIGURATION] All hosts were blacklisted; reverting to full set of hosts.
[2019-06-10T13:51:39.929+00:00|INFO|HttpClient|DMAAP-source-PDPD-CONFIGURATION] GET https://message-router:3905/events/PDPD-CONFIGURATION/12e77824-cba7-48dc-8251-b80b76529ead/dev-policy-drools-0?timeout=15000&limit=100 (anonymous) ...
[2019-06-10T13:51:39.936+00:00|WARN|HttpClient|DMAAP-source-PDPD-CONFIGURATION] Error executing HTTP request. Received fatal alert: internal_error; blacklisting for 2 minutes
[2019-06-10T13:51:39.936+00:00|ERROR|BusConsumer$CambriaConsumerWrapper|DMAAP-source-PDPD-CONFIGURATION] CambriaConsumerWrapper [fetchTimeout=15000]: cannot fetch because of Received fatal alert: internal_error - backoff for 15000 ms.
[2019-06-10T13:51:44.873+00:00|INFO|HttpClient|DMAAP-source-APPC-CL]    --> HTTP/1.1 200 OK
[2019-06-10T13:51:44.873+00:00|INFO|CambriaConsumerImpl|DMAAP-source-APPC-CL] UEB GET /events/APPC-CL/5d3db7a2-6611-475e-a259-a6c5e5dfc712/dev-policy-drools-0?timeout=15000&limit=100
[2019-06-10T13:51:44.873+00:00|INFO|HttpClient|DMAAP-source-APPC-CL] GET http://message-router:3904/events/APPC-CL/5d3db7a2-6611-475e-a259-a6c5e5dfc712/dev-policy-drools-0?timeout=15000&limit=100 (anonymous)


Are you aware of this issue? I would appreciate so much your kind help on this!

Kind regards,
Xoan



 

Join onap-discuss@lists.onap.org to automatically receive all group messages.