Topics

[dcae] VES unknown host error


William Kurkian <wkurkian3@...>
 

I have an onap cluster with a VES and DMaaP running. I deleted the VES pod, and let kubernetes bring it back up, in order to test another component, and i started getting this error in the VES logs:

[20:03:25,245][WARN ][pool-5-thread-1]Send failed, 28 message to send.
[20:03:25,245][ERROR][pool-5-thread-1]PUB_CHRONIC_FAILURE: Send failure count is 1589, above threshold 5.
[20:03:25,963][INFO ][pool-4-thread-1]sending 59 msgs to /events/unauthenticated.SEC_FAULT_OUTPUT. Oldest: 1725649 ms
[20:03:25,963][WARN ][pool-4-thread-1]All hosts were blacklisted; reverting to full set of hosts.
[20:03:25,964][INFO ][pool-4-thread-1]POST http://onap-dmaap:3904/events/unauthenticated.SEC_FAULT_OUTPUT (anonymous) ...
[20:03:25,965][WARN ][pool-4-thread-1]Unknown host onap-dmaap; blacklisting for 10 minutes
[20:03:25,965][WARN ][pool-4-thread-1]onap-dmaap
java.net.UnknownHostException: onap-dmaap
        at java.net.InetAddress.getAllByName0(InetAddress.java:1280)
        at java.net.InetAddress.getAllByName(InetAddress.java:1192)
        at java.net.InetAddress.getAllByName(InetAddress.java:1126)
        at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45)
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112)
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359)
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:622)
        at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:452)

Does anyone have any insight into this ?

Thanks,
William


Vijay VK
 

William,

Based on the error posted,  the output topic for VESCollector looks to be using default setting (correct MR hostname on the URL should be - message-router.onap.svc.cluster.local); this will be typically overridden when the collector is instantiated with configuration from Consul.  One possible issue is the Consul state might have been lost (possibly any restart?).

 

Could you run the below curl from within pod (vescollector) and share the o/p?

 

curl config-binding-service:10000/service_component/dcae-ves-collector

curl consul-server.onap:8500/v1/catalog/service/config-binding-service

 

Regards,

-Vijay

 

From: onap-discuss@... <onap-discuss@...> On Behalf Of William Kurkian
Sent: Monday, July 02, 2018 4:56 PM
To: onap-discuss@...
Subject: [onap-discuss] [dcae] VES unknown host error

 

I have an onap cluster with a VES and DMaaP running. I deleted the VES pod, and let kubernetes bring it back up, in order to test another component, and i started getting this error in the VES logs:

 

[20:03:25,245][WARN ][pool-5-thread-1]Send failed, 28 message to send.

[20:03:25,245][ERROR][pool-5-thread-1]PUB_CHRONIC_FAILURE: Send failure count is 1589, above threshold 5.

[20:03:25,963][INFO ][pool-4-thread-1]sending 59 msgs to /events/unauthenticated.SEC_FAULT_OUTPUT. Oldest: 1725649 ms

[20:03:25,963][WARN ][pool-4-thread-1]All hosts were blacklisted; reverting to full set of hosts.

[20:03:25,964][INFO ][pool-4-thread-1]POST http://onap-dmaap:3904/events/unauthenticated.SEC_FAULT_OUTPUT (anonymous) ...

[20:03:25,965][WARN ][pool-4-thread-1]Unknown host onap-dmaap; blacklisting for 10 minutes

[20:03:25,965][WARN ][pool-4-thread-1]onap-dmaap

java.net.UnknownHostException: onap-dmaap

        at java.net.InetAddress.getAllByName0(InetAddress.java:1280)

        at java.net.InetAddress.getAllByName(InetAddress.java:1192)

        at java.net.InetAddress.getAllByName(InetAddress.java:1126)

        at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45)

        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112)

        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359)

        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381)

        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)

        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)

        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)

        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)

        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)

        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)

        at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:622)

        at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:452)

 

Does anyone have any insight into this ?

 

Thanks,

William


William Kurkian <wkurkian3@...>
 

Thanks,

It looks like they don't return anything:
root@dcae-ves-collector:~# curl consul-server.onap:8500/v1/catalog/service/config-binding-service
[]root@dcae-ves-collector:~# curl config-binding-service:10000/service_component/dcae-ves-collector
root@dcae-ves-collector:~#

There may have been a restart. Is there a way I can restore the consul server ?

William


On Mon, Jul 2, 2018 at 6:26 PM, VENKATESH KUMAR, VIJAY <vv770d@...> wrote:

William,

Based on the error posted,  the output topic for VESCollector looks to be using default setting (correct MR hostname on the URL should be - message-router.onap.svc.cluster.local); this will be typically overridden when the collector is instantiated with configuration from Consul.  One possible issue is the Consul state might have been lost (possibly any restart?).

 

Could you run the below curl from within pod (vescollector) and share the o/p?

 

curl config-binding-service:10000/service_component/dcae-ves-collector

curl consul-server.onap:8500/v1/catalog/service/config-binding-service

 

Regards,

-Vijay

 

From: onap-discuss@... <onap-discuss@...> On Behalf Of William Kurkian
Sent: Monday, July 02, 2018 4:56 PM
To: onap-discuss@...
Subject: [onap-discuss] [dcae] VES unknown host error

 

I have an onap cluster with a VES and DMaaP running. I deleted the VES pod, and let kubernetes bring it back up, in order to test another component, and i started getting this error in the VES logs:

 

[20:03:25,245][WARN ][pool-5-thread-1]Send failed, 28 message to send.

[20:03:25,245][ERROR][pool-5-thread-1]PUB_CHRONIC_FAILURE: Send failure count is 1589, above threshold 5.

[20:03:25,963][INFO ][pool-4-thread-1]sending 59 msgs to /events/unauthenticated.SEC_FAULT_OUTPUT. Oldest: 1725649 ms

[20:03:25,963][WARN ][pool-4-thread-1]All hosts were blacklisted; reverting to full set of hosts.

[20:03:25,964][INFO ][pool-4-thread-1]POST http://onap-dmaap:3904/events/unauthenticated.SEC_FAULT_OUTPUT (anonymous) ...

[20:03:25,965][WARN ][pool-4-thread-1]Unknown host onap-dmaap; blacklisting for 10 minutes

[20:03:25,965][WARN ][pool-4-thread-1]onap-dmaap

java.net.UnknownHostException: onap-dmaap

        at java.net.InetAddress.getAllByName0(InetAddress.java:1280)

        at java.net.InetAddress.getAllByName(InetAddress.java:1192)

        at java.net.InetAddress.getAllByName(InetAddress.java:1126)

        at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45)

        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112)

        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359)

        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381)

        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)

        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)

        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)

        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)

        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)

        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)

        at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:622)

        at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:452)

 

Does anyone have any insight into this ?

 

Thanks,

William



Vijay VK
 

Hi William,

Unfortunately the steps to restore consul state is not straight forward currently (Consul itself is deployed by OOM, DCAE during bootstrap register few services and KV store when components are deployed; the bootstrap container itself exits after triggering all deploy).

1.     Could you check the consul UI (on the nodeport 30270 exposed) and verify if you see component and its corresponding Key/value information are available? If you do see then, then we can re-register CBS service via below post (from any of the pods) and restart docker ves pod (this will force configuration fetch upon restart and load valid properties).

curl -v -X PUT -H "Content-Type: application/json" --data '{"ID": "dcae-cbs1", "Name": "config-binding-service", "Address": "config-binding-service", "Port": 10000}' consul-server.onap:8500/v1/agent/service/register

2.     If however the Key/value tab is missing DCAE components k/v store in CONSUL – then we may need to redeploy DCAE via bootstrap to restore the components and Consul state correctly. We are looking into some optimization that will simplify consul state restoration in R3 to address such issues; this is being tracked under DCAEGEN2-580.

Thanks,

Vijay

 

From: William Kurkian <wkurkian3@...>
Sent: Tuesday, July 03, 2018 9:43 AM
To: VENKATESH KUMAR, VIJAY <vv770d@...>
Cc: onap-discuss@...
Subject: Re: [onap-discuss] [dcae] VES unknown host error

 

Thanks,

 

It looks like they don't return anything:

root@dcae-ves-collector:~# curl consul-server.onap:8500/v1/catalog/service/config-binding-service

[]root@dcae-ves-collector:~# curl config-binding-service:10000/service_component/dcae-ves-collector

root@dcae-ves-collector:~#

 

There may have been a restart. Is there a way I can restore the consul server ?

 

William

 

 

On Mon, Jul 2, 2018 at 6:26 PM, VENKATESH KUMAR, VIJAY <vv770d@...> wrote:

William,

Based on the error posted,  the output topic for VESCollector looks to be using default setting (correct MR hostname on the URL should be - message-router.onap.svc.cluster.local); this will be typically overridden when the collector is instantiated with configuration from Consul.  One possible issue is the Consul state might have been lost (possibly any restart?).

 

Could you run the below curl from within pod (vescollector) and share the o/p?

 

curl config-binding-service:10000/service_component/dcae-ves-collector

curl consul-server.onap:8500/v1/catalog/service/config-binding-service

 

Regards,

-Vijay

 

From: onap-discuss@... <onap-discuss@...> On Behalf Of William Kurkian
Sent: Monday, July 02, 2018 4:56 PM
To: onap-discuss@...
Subject: [onap-discuss] [dcae] VES unknown host error

 

I have an onap cluster with a VES and DMaaP running. I deleted the VES pod, and let kubernetes bring it back up, in order to test another component, and i started getting this error in the VES logs:

 

[20:03:25,245][WARN ][pool-5-thread-1]Send failed, 28 message to send.

[20:03:25,245][ERROR][pool-5-thread-1]PUB_CHRONIC_FAILURE: Send failure count is 1589, above threshold 5.

[20:03:25,963][INFO ][pool-4-thread-1]sending 59 msgs to /events/unauthenticated.SEC_FAULT_OUTPUT. Oldest: 1725649 ms

[20:03:25,963][WARN ][pool-4-thread-1]All hosts were blacklisted; reverting to full set of hosts.

[20:03:25,964][INFO ][pool-4-thread-1]POST http://onap-dmaap:3904/events/unauthenticated.SEC_FAULT_OUTPUT (anonymous) ...

[20:03:25,965][WARN ][pool-4-thread-1]Unknown host onap-dmaap; blacklisting for 10 minutes

[20:03:25,965][WARN ][pool-4-thread-1]onap-dmaap

java.net.UnknownHostException: onap-dmaap

        at java.net.InetAddress.getAllByName0(InetAddress.java:1280)

        at java.net.InetAddress.getAllByName(InetAddress.java:1192)

        at java.net.InetAddress.getAllByName(InetAddress.java:1126)

        at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45)

        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112)

        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359)

        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381)

        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)

        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)

        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)

        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)

        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)

        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)

        at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:622)

        at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:452)

 

Does anyone have any insight into this ?

 

Thanks,

William

 


William Kurkian <wkurkian3@...>
 

Thanks, I redeployed it in the meantime to meet some needs, but I'll be trying this out later as I think I will reproduce the original situation soon. 

Thanks,
William

On Tue, Jul 3, 2018 at 8:16 PM, VENKATESH KUMAR, VIJAY <vv770d@...> wrote:

Hi William,

Unfortunately the steps to restore consul state is not straight forward currently (Consul itself is deployed by OOM, DCAE during bootstrap register few services and KV store when components are deployed; the bootstrap container itself exits after triggering all deploy).

1.     Could you check the consul UI (on the nodeport 30270 exposed) and verify if you see component and its corresponding Key/value information are available? If you do see then, then we can re-register CBS service via below post (from any of the pods) and restart docker ves pod (this will force configuration fetch upon restart and load valid properties).

curl -v -X PUT -H "Content-Type: application/json" --data '{"ID": "dcae-cbs1", "Name": "config-binding-service", "Address": "config-binding-service", "Port": 10000}' consul-server.onap:8500/v1/agent/service/register

2.     If however the Key/value tab is missing DCAE components k/v store in CONSUL – then we may need to redeploy DCAE via bootstrap to restore the components and Consul state correctly. We are looking into some optimization that will simplify consul state restoration in R3 to address such issues; this is being tracked under DCAEGEN2-580.

Thanks,

Vijay

 

From: William Kurkian <wkurkian3@...>
Sent: Tuesday, July 03, 2018 9:43 AM
To: VENKATESH KUMAR, VIJAY <vv770d@...>
Cc: onap-discuss@...
Subject: Re: [onap-discuss] [dcae] VES unknown host error

 

Thanks,

 

It looks like they don't return anything:

root@dcae-ves-collector:~# curl consul-server.onap:8500/v1/catalog/service/config-binding-service

[]root@dcae-ves-collector:~# curl config-binding-service:10000/service_component/dcae-ves-collector

root@dcae-ves-collector:~#

 

There may have been a restart. Is there a way I can restore the consul server ?

 

William

 

 

On Mon, Jul 2, 2018 at 6:26 PM, VENKATESH KUMAR, VIJAY <vv770d@...> wrote:

William,

Based on the error posted,  the output topic for VESCollector looks to be using default setting (correct MR hostname on the URL should be - message-router.onap.svc.cluster.local); this will be typically overridden when the collector is instantiated with configuration from Consul.  One possible issue is the Consul state might have been lost (possibly any restart?).

 

Could you run the below curl from within pod (vescollector) and share the o/p?

 

curl config-binding-service:10000/service_component/dcae-ves-collector

curl consul-server.onap:8500/v1/catalog/service/config-binding-service

 

Regards,

-Vijay

 

From: onap-discuss@... <onap-discuss@...> On Behalf Of William Kurkian
Sent: Monday, July 02, 2018 4:56 PM
To: onap-discuss@...
Subject: [onap-discuss] [dcae] VES unknown host error

 

I have an onap cluster with a VES and DMaaP running. I deleted the VES pod, and let kubernetes bring it back up, in order to test another component, and i started getting this error in the VES logs:

 

[20:03:25,245][WARN ][pool-5-thread-1]Send failed, 28 message to send.

[20:03:25,245][ERROR][pool-5-thread-1]PUB_CHRONIC_FAILURE: Send failure count is 1589, above threshold 5.

[20:03:25,963][INFO ][pool-4-thread-1]sending 59 msgs to /events/unauthenticated.SEC_FAULT_OUTPUT. Oldest: 1725649 ms

[20:03:25,963][WARN ][pool-4-thread-1]All hosts were blacklisted; reverting to full set of hosts.

[20:03:25,964][INFO ][pool-4-thread-1]POST http://onap-dmaap:3904/events/unauthenticated.SEC_FAULT_OUTPUT (anonymous) ...

[20:03:25,965][WARN ][pool-4-thread-1]Unknown host onap-dmaap; blacklisting for 10 minutes

[20:03:25,965][WARN ][pool-4-thread-1]onap-dmaap

java.net.UnknownHostException: onap-dmaap

        at java.net.InetAddress.getAllByName0(InetAddress.java:1280)

        at java.net.InetAddress.getAllByName(InetAddress.java:1192)

        at java.net.InetAddress.getAllByName(InetAddress.java:1126)

        at org.apache.http.impl.conn.SystemDefaultDnsResolver.resolve(SystemDefaultDnsResolver.java:45)

        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:112)

        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:359)

        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:381)

        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)

        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)

        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)

        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)

        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)

        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)

        at com.att.nsa.apiClient.http.HttpClient.runCall(HttpClient.java:622)

        at com.att.nsa.apiClient.http.HttpClient.post(HttpClient.java:452)

 

Does anyone have any insight into this ?

 

Thanks,

William