DMAAP-message-router-mirrormaker pods is not running #dmaap


satish kumar
 

Hello Team,

I have deployed ONAP Dublin on bare metal about 2 months ago. Everything was working properly after deployment. But, suddenly from last few days, getting problem in dmaap-message-router-mirrormaker (getting CrashLoopBackOff). The error are as follows:
dev-dmaap-message-router-mirrormaker-677857bbc-gjpns          0/1     CrashLoopBackOff   7988       75d



The log for pod dmaap-message-router-mirrormaker is: 

+ exec start-mirrormaker.sh
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.38/containers/dev-dmaap-message-router-mirrormaker-677857bbc-gjpns/json: dial unix /var/run/docker.sock: connect: permission denied
2019-10-07 10:36:15 INFO  - mmagenthome is set :/opt loading properties at /etc/mmagent.config
2019-10-07 10:36:15 INFO  - kafkahome is set :/opt/kafka
2019-10-07 10:36:15 ERROR -  Exception Occered java.net.SocketException: Unexpected end of file from server
2019-10-07 10:36:15 ERROR - Problem publishing to topic, please verify the config org.onap.dmaap.mr.mirrormakeragent MR URL is:http://message-router:3904 Error is:  ERROR:Unexpected end of file from server
ERROR: mmagent startup unsuccessful, please make sure the mmagenthome /etc/mmagent.config is set and mechid have the rights to the topic


I also did robot health check and got the following error corresponding to dmaap:
Basic DMAAP Data Router Health Check                                  | PASS |
------------------------------------------------------------------------------
Basic DMAAP Message Router Health Check                               [ WARN ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', error(104, 'Connection reset by peer'))': /topics
[ WARN ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', error(104, 'Connection reset by peer'))': /topics
[ WARN ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', error(104, 'Connection reset by peer'))': /topics
| FAIL |
ConnectionError: HTTPConnectionPool(host='message-router.onap', port=3904): Max retries exceeded with url: /topics (Caused by ProtocolError('Connection aborted.', error(104, 'Connection reset by peer')))


And in SDC health check "U-EB cluster is not available"
{
      "healthCheckComponent": "DE",
      "healthCheckStatus": "DOWN",
      "description": "U-EB cluster is not available"
    },


Please help me to understand and resolve this problem. Thank you so much in advance.

Regards,
Satish Kumar
 




 


Kuldeep Singh Negi
 

Hi Satish,

 

Hope the below problem could get solved for you.

Currently, we are also facing same issue for this mirror-maker pod. Any guidance on this will be greatly appreciated.

 

Thanks.

 

Regards,

Kuldeep

 

From: onap-discuss@... <onap-discuss@...> On Behalf Of satish kumar via Lists.Onap.Org
Sent: Monday, October 7, 2019 4:27 PM
To: onap-discuss@...
Subject: [onap-discuss] DMAAP-message-router-mirrormaker pods is not running #dmaap

 

Hello Team,

I have deployed ONAP Dublin on bare metal about 2 months ago. Everything was working properly after deployment. But, suddenly from last few days, getting problem in dmaap-message-router-mirrormaker (getting CrashLoopBackOff). The error are as follows:

dev-dmaap-message-router-mirrormaker-677857bbc-gjpns          0/1     CrashLoopBackOff   7988       75d


The log for pod dmaap-message-router-mirrormaker is: 

+ exec start-mirrormaker.sh

Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.38/containers/dev-dmaap-message-router-mirrormaker-677857bbc-gjpns/json: dial unix /var/run/docker.sock: connect: permission denied

2019-10-07 10:36:15 INFO  - mmagenthome is set :/opt loading properties at /etc/mmagent.config

2019-10-07 10:36:15 INFO  - kafkahome is set :/opt/kafka

2019-10-07 10:36:15 ERROR -  Exception Occered java.net.SocketException: Unexpected end of file from server

2019-10-07 10:36:15 ERROR - Problem publishing to topic, please verify the config org.onap.dmaap.mr.mirrormakeragent MR URL is:http://message-router:3904 Error is:  ERROR:Unexpected end of file from server

ERROR: mmagent startup unsuccessful, please make sure the mmagenthome /etc/mmagent.config is set and mechid have the rights to the topic


I also did robot health check and got the following error corresponding to dmaap:

Basic DMAAP Data Router Health Check                                  | PASS |

------------------------------------------------------------------------------

Basic DMAAP Message Router Health Check                               [ WARN ] Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', error(104, 'Connection reset by peer'))': /topics

[ WARN ] Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', error(104, 'Connection reset by peer'))': /topics

[ WARN ] Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'ProtocolError('Connection aborted.', error(104, 'Connection reset by peer'))': /topics

| FAIL |

ConnectionError: HTTPConnectionPool(host='message-router.onap', port=3904): Max retries exceeded with url: /topics (Caused by ProtocolError('Connection aborted.', error(104, 'Connection reset by peer')))


And in SDC health check "U-EB cluster is not available"

{

      "healthCheckComponent": "DE",

      "healthCheckStatus": "DOWN",

      "description": "U-EB cluster is not available"

    },



Please help me to understand and resolve this problem. Thank you so much in advance.

Regards,
Satish Kumar

 




 

::DISCLAIMER::

The contents of this e-mail and any attachment(s) are confidential and intended for the named recipient(s) only. E-mail transmission is not guaranteed to be secure or error-free as information could be intercepted, corrupted, lost, destroyed, arrive late or incomplete, or may contain viruses in transmission. The e mail and its contents (with or without referred errors) shall therefore not attach any liability on the originator or HCL or its affiliates. Views or opinions, if any, presented in this email are solely those of the author and may not necessarily reflect the views or opinions of HCL or its affiliates. Any form of reproduction, dissemination, copying, disclosure, modification, distribution and / or publication of this message without the prior written consent of authorized representative of HCL is strictly prohibited. If you have received this email in error please delete it and notify the sender immediately. Before opening any email and/or attachments, please check them for viruses and other defects.


Michael Pruss
 

Hi Kuldeep and Satish,

Where did you locate the helm charts for mirror maker? There are no explicit charts for mirror maker within upstream OOM and therefore I am unable to deploy it. 

Any information on deployment steps and chart locations for mirror maker will be greatly appreciated.

Thanks,

Michael.


Dominic Lunanuova
 

+Sunil and Mandar,  DMaaP SME’s who know history of Mirror Maker.

 

From: onap-discuss@... <onap-discuss@...> On Behalf Of Michael Pruss
Sent: Monday, November 16, 2020 5:05 PM
To: Kuldeep Singh Negi <kuldeepsinghn@...>; onap-discuss@...
Subject: Re: [onap-discuss] DMAAP-message-router-mirrormaker pods is not running #dmaap

 

Hi Kuldeep and Satish,

Where did you locate the helm charts for mirror maker? There are no explicit charts for mirror maker within upstream OOM and therefore I am unable to deploy it. 

Any information on deployment steps and chart locations for mirror maker will be greatly appreciated.

Thanks,

Michael.


Mandar Sawant <ms5838@...>
 

Mirror Maker was deprecated from El Alto as there was no use case for it. Is it required now for implementing some use case?

 

Thanks,
Mandar

 

From: LUNANUOVA, DOMINIC (DOMINIC) <dgl@...>
Sent: Monday, November 16, 2020 4:18 PM
To: onap-discuss@...; michprus@...; Kuldeep Singh Negi <kuldeepsinghn@...>
Cc: UNNAVA, SUNIL <su622b@...>; SAWANT, MANDAR <ms5838@...>
Subject: RE: [onap-discuss] DMAAP-message-router-mirrormaker pods is not running #dmaap

 

+Sunil and Mandar,  DMaaP SME’s who know history of Mirror Maker.

 

From: onap-discuss@... <onap-discuss@...> On Behalf Of Michael Pruss
Sent: Monday, November 16, 2020 5:05 PM
To: Kuldeep Singh Negi <kuldeepsinghn@...>; onap-discuss@...
Subject: Re: [onap-discuss] DMAAP-message-router-mirrormaker pods is not running #dmaap

 

Hi Kuldeep and Satish,

Where did you locate the helm charts for mirror maker? There are no explicit charts for mirror maker within upstream OOM and therefore I am unable to deploy it. 

Any information on deployment steps and chart locations for mirror maker will be greatly appreciated.

Thanks,

Michael.


Michael Pruss
 

Hi Mandar,

I am working on a hybrid ONAP deployment using Frankfurt release of ONAP.  The idea is to have several components on a cloud platform including a central dmaap hub, then components such as CDS can be deployed on onprem cluster with its own dmaap deployment. Using mirror maker is required to replicate the dmaap topics/messages between the main dmaap hub and the onprem dmaap deployment.

Please let me know if you have any alternatives that do not require mirror maker. Per my understanding, to achieve the above use case mirror maker is required.

Thanks,
Michael


Dominic Lunanuova
 

While you are all discussing this, just want to point out a related Jira we created a long time ago: https://jira.onap.org/browse/OOM-1579

At that time, DMaaP design anticipated there might be a central “hub” and one or more edge deployments because the AT&T ECOMP architecture has long had central/edge deployments.   (e.g. use case: edge data collector/central analysis function)

The blocking issue in ONAP has been some consensus on the technique and naming conventions for inter-K8S services, and as this Jira mentions, some deployment indicator for where you are deploying (and what the central deployment is named).

 

The vision, as it relates to this thread might look like:

  • A “central” k8s deployment with the full set of DMaaP components, AAF, and the CDS central component
  • An “edge” k8s deployment with a streamlined set of DMaaP components (MR, MM, DR Node) and the CDS edge component
  • Provisioning an MR topic in “central” using DMaaP Bus Controller with attributes which indicate the location of publisher (e.g. central) and subscriber (e.g. edge) and a flag for the type of message replication (e.g. “central-to-edge”), which results in the proper provisioning for the message path:
    • Publisher Identity is authorized in AAF to publish on the topic
    • Subscriber Identity is authorized in AAF to subscribe to the topic
    • Edge MM  is provisioned  to replicate Topic from central MR to edge MR
  • End result: publisher produces message to central MR, edge MM replicates message to edge MR, edge subscriber consumes message

 

Note: the direction of the message replication is important – you can’t have bidirectional replication on the same topic – but either direction is supported.

 

As Mandar, Sunil and I are getting pulled from ONAP soon, wish we had the Pruss CDS use case 6 months ago!

While we still have some limited cycles to consult, might somebody want to transition into DMaaP to complete the work using the newer MM?

 

-Dom

 

From: onap-discuss@... <onap-discuss@...> On Behalf Of Michael Pruss
Sent: Tuesday, November 17, 2020 3:15 PM
To: SAWANT, MANDAR <ms5838@...>; onap-discuss@...
Subject: Re: [onap-discuss] DMAAP-message-router-mirrormaker pods is not running #dmaap

 

Hi Mandar,

I am working on a hybrid ONAP deployment using Frankfurt release of ONAP.  The idea is to have several components on a cloud platform including a central dmaap hub, then components such as CDS can be deployed on onprem cluster with its own dmaap deployment. Using mirror maker is required to replicate the dmaap topics/messages between the main dmaap hub and the onprem dmaap deployment.

Please let me know if you have any alternatives that do not require mirror maker. Per my understanding, to achieve the above use case mirror maker is required.

Thanks,
Michael