Re: dev-dmaap-dmaap-dr-node pod does not come up due to https call to dev-dmaap-dmaap-dr-prov pod #dmaap #casablanca


Conor Ward
 

Hi,


The issue with only dmaap-dr-node not coming up is not related to DMAAP-964. DMAAP-964 is an issue which occurs due to the mariadb not being instantiated correctly so dmaap-dr-prov and dmaap-dr-node do not come up.

The issue we are now seeing with only dmaap-dr-node not coming up is due to AAF certs expiring. We are tracking that issue here DMAAP-1048, we also reached out to the onap-seccom mailing list for any advice on extending or renewing the certs but haven't heard anything back yet. The link to that message is here - https://lists.onap.org/g/onap-seccom/topic/aaf_certificates_expired/29915899?p=,,,20,0,0,0::recentpostdate%2Fsticky,,,20,2,0,29915899.


I'm guessing that this issue has occurred before so any help or feedback would be appreciated.


Regards,

Conor


From: onap-discuss@... <onap-discuss@...> on behalf of Michael O'Brien <frank.obrien@...>
Sent: Wednesday 20 February 2019 03:10:28
To: onap-discuss@...; kranthi.guttikonda@...; alessandro.dalessandro@...
Subject: Re: [onap-discuss] dev-dmaap-dmaap-dr-node pod does not come up due to https call to dev-dmaap-dmaap-dr-prov pod #dmaap #casablanca
 

Guys seeing this again both on casablanca 20190218 and master 20180218 - both running only dmaap (nothing else deployed or deployed with aaf)

LOG-898
master
onap onap-dmaap-dmaap-dr-node-57f77bd54d-q7v78 0/1 CrashLoopBackOff 9 26m 10.42.112.70 obriensystemsu0 <none>

casablanca (3.0.1 essentially)
onap onap-dmaap-dmaap-dr-node-57f77bd54d-kw8n7 0/1 Running 79 5h

 

I am also seeing an issue with SDNC ansible on both the latest Casablanca and master – sdnc required dmaap and sdc to come up - note

onap          onap-sdnc-sdnc-ansible-server-7d595dd8-fvm95       0/1       CrashLoopBackOff   41         3h

 

/michael

From: onap-discuss@... <onap-discuss@...> On Behalf Of kranthi guttikonda
Sent: Tuesday, February 19, 2019 2:07 PM
To: onap-discuss@...; alessandro.dalessandro@...
Subject: Re: [onap-discuss] dev-dmaap-dmaap-dr-node pod does not come up due to https call to dev-dmaap-dmaap-dr-prov pod #dmaap #casablanca

 

Please take a look into https://jira.onap.org/browse/DMAAP-964

 

From: <onap-discuss@...> on behalf of D'Alessandro Alessandro Gerardo <alessandro.dalessandro@...>
Reply-To: "onap-discuss@..." <onap-discuss@...>, "alessandro.dalessandro@..." <alessandro.dalessandro@...>
Date: Tuesday, February 19, 2019 at 12:16 PM
To: "onap-discuss@..." <onap-discuss@...>
Subject: [onap-discuss] dev-dmaap-dmaap-dr-node pod does not come up due to https call to dev-dmaap-dmaap-dr-prov pod #dmaap #casablanca

 

Hi all,

I have deployed Casablanca and I have an issue with the highlighted pod

 

 

dev-dmaap-dbc-pg-0                                                                    1/1      Running            0          30m       10.42.29.89   k8s-13    <none>

dev-dmaap-dbc-pg-1                                                                    1/1       Running            0          27m       10.42.182.91 k8s-7     <none>

dev-dmaap-dbc-pgpool-7b748d5894-88wb9                     1/1       Running            0          30m       10.42.140.152   k8s-11    <none>

dev-dmaap-dbc-pgpool-7b748d5894-pj8zv                        1/1        Running            0          30m       10.42.183.217   k8s-6     <none>

dev-dmaap-dmaap-bus-controller-6757c4c86-rv9zh       1/1       Running            0          30m       10.42.156.206   k8s-10    <none>

dev-dmaap-dmaap-dr-db-bb4c67cfd-84c99                        1/1       Running            0          30m       10.42.138.166   k8s-10    <none>

dev-dmaap-dmaap-dr-node-5655ffbd55-gdfpx                 0/1       Running            6         30m       10.42.30.95     k8s-6     <none>

dev-dmaap-dmaap-dr-prov-66df46884f-sjd7x                  1/1       Running            0          30m       10.42.13.150    k8s-7     <none>

dev-dmaap-message-router-684b499dbc-8c6dm            1/1       Running            0         30m       10.42.191.38    k8s-11    <none>

dev-dmaap-message-router-kafka-8466bf6864-t7gtq    1/1       Running            0          30m      10.42.32.83     k8s-4     <none>

dev-dmaap-message-router-zookeeper-5bd997b466-dfmzl  1/1       Running            0          30m       10.42.25.154    k8s-2     <none>

 

Log from dev-dmaap-dmaap-dr-node-5655ffbd55-gdfpx is:

 

16:03:22.198 ERROR [Node Configuration Timer] org.onap.dmaap.datarouter.node.NodeConfigManager - EELF0004E  Configuration failed.

java.net.UnknownHostException: dmaap-dr-prov - try again later.

provurl:: https://dmaap-dr-prov:8443/internal/prov

16:03:22.200 ERROR [Node Configuration Timer] org.onap.dmaap.datarouter.node.NodeConfigManager - EELF0004E  Configuration failed.

java.net.UnknownHostException: dmaap-dr-prov - try again later.

provurl:: https://dmaap-dr-prov:8443/internal/prov

 

 

what I notice is that if the request comes from http it is satisfied while it is NOT if it comes from https

 

please look at the details here below from the robot container:   

 

curl -v http://dmaap-dr-prov:8080/internal/prov

 

{

"feeds": [

],

"groups": [

],

"subscriptions": [

],

"parameters": {

   "ACTIVE_POD": "dmaap-dr-prov",

   "DELIVERY_INIT_RETRY_INTERVAL": 10,

   "DELIVERY_MAX_AGE": 86400,

   "DELIVERY_MAX_RETRY_INTERVAL": 3600,

   "DELIVERY_RETRY_RATIO": 2,

   "LOGROLL_INTERVAL": 300,

   "NODES": ["dmaap-dr-node"],

   "PROV_ACTIVE_NAME": "dmaap-dr-prov",

   "PROV_AUTH_ADDRESSES":

["dmaap-dr-prov","dmaap-dr-node"],

   "PROV_AUTH_SUBJECTS": [""],

   "PROV_DOMAIN": "onap",

   "PROV_MAXFEED_COUNT": 10000,

   "PROV_MAXSUB_COUNT": 100000,

   "PROV_NAME": "dmaap-dr-prov",

   "PROV_REQUIRE_CERT": "false",

   "PROV_REQUIRE_SECURE": "false",

   "STANDBY_POD": "",

   "_INT_VALUES":

["LOGROLL_INTERVAL","PROV_MAXFEED_COUNT","PROV_MAXSUB_COUNT","DELIVERY_INIT_RETRY_INTERVAL","DELIVERY_MAX_RETRY_INTERVAL","DELIVERY_RETRY_RATIO","DELIVERY_MAX_AGE"]

},

"ingress": [

],

"egress": {

},

"routing": [

]

 

 

curl - https://dmaap-dr-prov:8080/internal/prov     THE SAME REQUEST OF DMAAP-DR-NODE

 

 *   Trying 10.43.74.144...

* TCP_NODELAY set

* Connected to dmaap-dr-prov (10.43.74.144) port 8080 (#0)

* ALPN, offering h2

* ALPN, offering http/1.1

* successfully set certificate verify locations:

*   CAfile: /etc/ssl/certs/ca-certificates.crt

   CApath: /etc/ssl/certs

* TLSv1.2 (OUT), TLS handshake, Client hello (1):

* error:1408F10B:SSL routines:ssl3_get_record:wrong version number

* stopped the pause stream!

* Closing connection 0

curl: (35) error:1408F10B:SSL

routines:ssl3_get_record:wrong version number

 

 

I’m re-installing Casablanca after changing the file: root@rancher:~/oom/kubernetes/dmaap/charts/dmaap-bus-controller/resources/dmaap/onap.json  as below:

from "drProvUrl": "https://dmaap-dr-prov:8443"    to

"drProvUrl": "http://dmaap-dr-prov:8080"

 

I’m wondering if someone else has already experienced the same issue and tested this change to share with me if there will be side effects.

 

Thanks in advance

Best regards,

Alessandro

This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service

Join onap-discuss@lists.onap.org to automatically receive all group messages.