Re: [El-Alto][vCPE]- APPC not reading DMAAP messages "closed loop" event in vCPE use case


Taka Cho
 

Try non-encrypted openstack password first.

 

Here is a sample to override the helm value:

 

https://gerrit.onap.org/r/gitweb?p=oom.git;a=blob;f=kubernetes/onap/resources/overrides/openstack.yaml;h=5e09d7cfe9ee3bcc98c1ce9efedad70d1016d721;hb=935623ebd592070fc55fe5dc1ebc3bad07fcc714

 

In Frankfurt, there is a fix for iaas encrypted method: https://gerrit.onap.org/r/c/appc/+/98825

 

Taka

 

From: Devangam Manjunatha <Devangam.Manjunatha@...>
Sent: Wednesday, May 13, 2020 6:02 AM
To: FREEMAN, BRIAN D <bf1936@...>; onap-discuss@...; CHO, TAKAMUNE <tc012c@...>
Cc: JOMY JOSE <Jomy.Jose@...>
Subject: Re: [onap-discuss] [El-Alto][vCPE]- APPC not reading DMAAP messages "closed loop" event in vCPE use case

 

Hi Brian, Hi Taka,

 

You are right, I enabled logs of 3 APPC pods, i can see messages from Policy are picked by "rnd-appc-appc-2" pod. 

I hope we are close to vGMuX restart as part of vCPE Closed loop testing. 

 

In APPC log i can see following two errors.

 

Error-1: Request from APPC to Keystone failing with "404 Not Found" error:

 

2020-05-13T05:45:18,840 | INFO  | appc-dispatcher-4 | Keystone                         | 482 - appc-iaas-adapter - 1.6.4 | 1 * Sending client request on thread appc-dispatcher-4

1 > Accept: application/json

1 > Content-Type: application/json

{

  "auth" : {

    "tenantName" : "admin",

    "passwordCredentials" : {

      "username" : "admin",

      "password" : "********"

    }

  }

}

 

2020-05-13T05:45:18,866 | INFO  | appc-dispatcher-4 | Keystone                         | 482 - appc-iaas-adapter - 1.6.4 | 1 * Client response received on thread appc-dispatcher-4

1 < 404

1 < Connection: Keep-Alive

1 < Content-Length: 233

1 < Content-Type: text/html

1 < Date: Wed, 13 May 2020 05:45:18 GMT

1 < Keep-Alive: timeout=5, max=100

1 < Server: Apache/2.4.29 (Ubuntu)

1 < Vary: X-Auth-Token

1 < x-openstack-request-id: req-f8b8830c-b091-4a48-9aaa-5fa6f7451abd

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">

<title>404 Not Found</title>

<h1>Not Found</h1>

<p>The requested URL was not found on the server.  If you entered the URL manually please check your spelling and try again.</p>

 

 

 

 

Followed by above error i can see below error.

 

Error-2: "The allocator failed to allocate a new context to extend the pool":

 

2020-05-13T05:45:18,867 | ERROR | appc-dispatcher-4 | ProviderOperation                | 482 - appc-iaas-adapter - 1.6.4 | APPC0054E Connection to provider http://192.168.9.75:5000/v3 at identity http://192.168.9.75:5000/v3 using tenant name admin (id 70968f3b213c4c8d81d92c06a9dddf7c) failed, reason=The allocator failed to allocate a new context to extend the pool., retrying in 30 seconds, attempt 1 of 10.

org.onap.appc.pool.PoolExtensionException: The allocator failed to allocate a new context to extend the pool.

at org.onap.appc.pool.Pool.extend(Pool.java:336) ~[?:?]

at org.onap.appc.pool.Pool.reserve(Pool.java:257) ~[?:?]

at org.onap.appc.adapter.iaas.provider.operation.impl.base.ProviderOperation.getContext(ProviderOperation.java:373) ~[?:?]

at org.onap.appc.adapter.iaas.provider.operation.impl.RestartServer.restartServer(RestartServer.java:206) ~[?:?]

at org.onap.appc.adapter.iaas.provider.operation.impl.RestartServer.executeProviderOperation(RestartServer.java:247) ~[?:?]

at org.onap.appc.adapter.iaas.provider.operation.impl.base.ProviderOperation.doOperation(ProviderOperation.java:430) ~[?:?]

at org.onap.appc.adapter.iaas.impl.ProviderAdapterImpl.restartServer(ProviderAdapterImpl.java:115) ~[?:?]

at sun.reflect.GeneratedMethodAccessor131.invoke(Unknown Source) ~[?:?]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]

at java.lang.reflect.Method.invoke(Method.java:498) ~[?:?]

 

 

For more details please find the attached logs(i.e. rnd-appc-appc-2_karaf).

 

I checked "appc.properties", Here "provider1.identity" & "provider1.tenant1.password" are fine. 

 

provider1.type=OpenStackProvider

provider1.name=OpenStack

provider1.identity=http://192.168.9.75:5000/v3

provider1.tenant1.name=admin

provider1.tenant1.domain=Default

provider1.tenant1.userid=admin

provider1.tenant1.password=root123

 

 

For reference i attached "appc.properties" and "RegionOne" response.

 

Request to let us know how to fix above 2 issues.

 

Regards,

Manjunath.

 

 


From: FREEMAN, BRIAN D <bf1936@...>
Sent: Tuesday, May 12, 2020 11:09 PM
To: Devangam Manjunatha <Devangam.Manjunatha@...>; onap-discuss@... <onap-discuss@...>; CHO, TAKAMUNE <tc012c@...>; morgan.richomme@... <morgan.richomme@...>; FORSYTH, JAMES <jf2512@...>
Cc: JOMY JOSE <Jomy.Jose@...>; OLLIVIER Cédric TGI/OLN <cedric.ollivier@...>; DESBUREAUX Sylvain TGI/OLN <sylvain.desbureaux@...>
Subject: RE: [onap-discuss] [El-Alto][vCPE]- APPC not reading DMAAP messages "closed loop" event in vCPE use case

 

If you have a replica of 3 you need to look at all 3 appc logs to see which one picked up the message from Policy.

 

If you are not seeing it pick up topics – I would use curl or POSTMAN and see if you see the message on the APPC-LCM-READ topic.

 

Brian

 

 

From: Devangam Manjunatha <Devangam.Manjunatha@...>
Sent: Tuesday, May 12, 2020 1:27 PM
To: FREEMAN, BRIAN D <bf1936@...>; onap-discuss@...; CHO, TAKAMUNE <tc012c@...>; morgan.richomme@...; FORSYTH, JAMES <jf2512@...>
Cc: JOMY JOSE <Jomy.Jose@...>; OLLIVIER Cédric TGI/OLN <cedric.ollivier@...>; DESBUREAUX Sylvain TGI/OLN <sylvain.desbureaux@...>
Subject: Re: [onap-discuss] [El-Alto][vCPE]- APPC not reading DMAAP messages "closed loop" event in vCPE use case

 

Hi Brain,

 

Sorry by mistake older logs also attached in my last mail. Please find the attached latest logs.

 

We are running APPC replica of 3, Sending logs only from  pod "rnd-appc-appc-0".

 

root@rndelato-nfs:~/oom/kubernetes# kubectl get pods|grep appc

rnd-appc-appc-0                                               2/2     Running      0          3d5h

rnd-appc-appc-1                                               2/2     Running      0          3d5h

rnd-appc-appc-2                                               2/2     Running      0          3d5h

rnd-appc-appc-ansible-server-0                                1/1     Running      0          3d5h

rnd-appc-appc-cdt-b4d8f5c95-kvkvm                             1/1     Running      0          3d5h

rnd-appc-appc-db-0                                            1/1     Running      0          3d5h

rnd-appc-appc-db-1                                            1/1     Running      0          3d5h

rnd-appc-appc-db-2                                            1/1     Running      0          3d5h

rnd-appc-appc-dgbuilder-759d9968f9-s6m6x                      1/1     Running      0          3d5h

root@rndelato-nfs:~/oom/kubernetes#

 

 

Dmaap robot health check is passing. I am not sure how to test dmaap pubsub healthcheck. Request to let me know  how to test it.

 

 

root@rndelato-nfs:~/oom/kubernetes# ./robot/ete-k8s.sh onap health

++ export NAMESPACE=onap

++ NAMESPACE=onap

+++ kubectl --namespace onap get pods

+++ sed 's/ .*//'

+++ grep robot

++ POD=rnd-robot-robot-c74bc6964-j4t8s

++ TAGS='-i health'

+++ dirname ./robot/ete-k8s.sh

++ DIR=./robot

++ SCRIPTDIR=scripts/etescript

++ ETEHOME=/var/opt/ONAP

++ [[ health == \e\x\e\c\s\c\r\i\p\t ]]

+++ kubectl --namespace onap exec rnd-robot-robot-c74bc6964-j4t8s -- bash -c 'ls -1q /share/logs/ | wc -l'

++ export GLOBAL_BUILD_NUMBER=13

++ GLOBAL_BUILD_NUMBER=13

+++ printf %04d 13

++ OUTPUT_FOLDER=0013_ete_health

++ DISPLAY_NUM=103

++ VARIABLEFILES='-V /share/config/robot_properties.py'

++ VARIABLES='-v GLOBAL_BUILD_NUMBER:23822'

++ kubectl --namespace onap exec rnd-robot-robot-c74bc6964-j4t8s -- /var/opt/ONAP/runTags.sh -V /share/config/robot_properties.py -v GLOBAL_BUILD_NUMBER:23822 -d /share/logs/0013_ete_health -i health --display 103

Starting Xvfb on display :103 with res 1280x1024x24

Executing robot tests at log level TRACE

==============================================================================

Testsuites                                                                    

==============================================================================

Testsuites.Health-Check :: Test that ONAP components are available via basi...

==============================================================================

Basic A&AI Health Check                                               | PASS |

------------------------------------------------------------------------------

Basic AAF Health Check                                                | PASS |

------------------------------------------------------------------------------

Basic AAF SMS Health Check                                            | PASS |

------------------------------------------------------------------------------

Basic APPC Health Check                                               | PASS |

------------------------------------------------------------------------------

Basic CLI Health Check                                                | PASS |

------------------------------------------------------------------------------

Basic CLAMP Health Check                                              | PASS |

------------------------------------------------------------------------------

Basic DCAE Health Check                                               | PASS |

------------------------------------------------------------------------------

Basic DMAAP Data Router Health Check                                  | PASS |

------------------------------------------------------------------------------

Basic DMAAP Message Router Health Check                               | PASS |

------------------------------------------------------------------------------

Basic DMAAP Bus Controller Health Check With Basic Auth               | PASS |

------------------------------------------------------------------------------

Basic External API NBI Health Check                                   | PASS |

------------------------------------------------------------------------------

Basic Log Elasticsearch Health Check                                  | PASS |

------------------------------------------------------------------------------

Basic Log Kibana Health Check                                         | PASS |

------------------------------------------------------------------------------

Basic Log Logstash Health Check                                       | PASS |

------------------------------------------------------------------------------

Basic Microservice Bus Health Check                                   | PASS |

------------------------------------------------------------------------------

Basic Multicloud API Health Check                                     | PASS |

------------------------------------------------------------------------------

Basic Multicloud-pike API Health Check                                | PASS |

------------------------------------------------------------------------------

Basic Multicloud-starlingx API Health Check                           | PASS |

------------------------------------------------------------------------------

Basic Multicloud-titanium_cloud API Health Check                      | PASS |

------------------------------------------------------------------------------

Basic Multicloud-vio API Health Check                                 | PASS |

------------------------------------------------------------------------------

Basic Multicloud-k8s API Health Check                                 | PASS |

------------------------------------------------------------------------------

Basic OOF-Homing Health Check                                         | PASS |

------------------------------------------------------------------------------

Basic OOF-SNIRO Health Check                                          | PASS |

------------------------------------------------------------------------------

Basic OOF-CMSO Health Check                                           | FAIL |

400 != 200

------------------------------------------------------------------------------

Basic Policy Health Check                                             | PASS |

------------------------------------------------------------------------------

Basic Pomba AAI-context-builder Health Check                          | PASS |

------------------------------------------------------------------------------

Basic Pomba SDC-context-builder Health Check                          | PASS |

------------------------------------------------------------------------------

Basic Pomba Network-discovery-context-builder Health Check            | PASS |

------------------------------------------------------------------------------

Basic Pomba Service-Decomposition Health Check                        | PASS |

------------------------------------------------------------------------------

Basic Pomba Network-Discovery-MicroService Health Check               | PASS |

------------------------------------------------------------------------------

Basic Pomba Pomba-Kibana Health Check                                 | PASS |

------------------------------------------------------------------------------

Basic Pomba Elastic-Search Health Check                               | PASS |

------------------------------------------------------------------------------

Basic Pomba Sdnc-Context-Builder Health Check                         | PASS |

------------------------------------------------------------------------------

Basic Pomba Context-Aggregator Health Check                           | PASS |

------------------------------------------------------------------------------

Basic Portal Health Check                                             | PASS |

------------------------------------------------------------------------------

Basic SDC Health Check                                                (DMaaP:UP)

| PASS |

------------------------------------------------------------------------------

Basic SDNC Health Check                                               | PASS |

------------------------------------------------------------------------------

Basic SO Health Check                                                 | PASS |

------------------------------------------------------------------------------

Basic UseCaseUI API Health Check                                      | PASS |

------------------------------------------------------------------------------

Basic VFC catalog API Health Check                                    | PASS |

------------------------------------------------------------------------------

Basic VFC emsdriver API Health Check                                  | PASS |

------------------------------------------------------------------------------

Basic VFC gvnfmdriver API Health Check                                | PASS |

------------------------------------------------------------------------------

Basic VFC huaweivnfmdriver API Health Check                           | PASS |

------------------------------------------------------------------------------

Basic VFC jujuvnfmdriver API Health Check                             | PASS |

------------------------------------------------------------------------------

Basic VFC multivimproxy API Health Check                              | PASS |

------------------------------------------------------------------------------

Basic VFC nokiav2driver API Health Check                              | PASS |

------------------------------------------------------------------------------

Basic VFC nslcm API Health Check                                      | PASS |

------------------------------------------------------------------------------

Basic VFC resmgr API Health Check                                     | PASS |

------------------------------------------------------------------------------

Basic VFC vnflcm API Health Check                                     | PASS |

------------------------------------------------------------------------------

Basic VFC vnfmgr API Health Check                                     | PASS |

------------------------------------------------------------------------------

Basic VFC vnfres API Health Check                                     | PASS |

------------------------------------------------------------------------------

Basic VFC workflow API Health Check                                   | PASS |

------------------------------------------------------------------------------

Basic VFC ztesdncdriver API Health Check                              | PASS |

------------------------------------------------------------------------------

Basic VFC ztevnfmdriver API Health Check                              | PASS |

------------------------------------------------------------------------------

Basic VID Health Check                                                | PASS |

------------------------------------------------------------------------------

Basic VNFSDK Health Check                                             | PASS |

------------------------------------------------------------------------------

Basic Holmes Rule Management API Health Check                         | PASS |

------------------------------------------------------------------------------

Basic Holmes Engine Management API Health Check                       | PASS |

------------------------------------------------------------------------------

Basic Multicloud-fcaps API Health Check                               | PASS |

------------------------------------------------------------------------------

Basic Modeling genericparser API Health Check                         | FAIL |

502 != 200

------------------------------------------------------------------------------

Basic CDS Health Check                                                | PASS |

------------------------------------------------------------------------------

Testsuites.Health-Check :: Test that ONAP components are available... | FAIL |

61 critical tests, 59 passed, 2 failed

61 tests total, 59 passed, 2 failed

==============================================================================

Testsuites                                                            | FAIL |

61 critical tests, 59 passed, 2 failed

61 tests total, 59 passed, 2 failed

==============================================================================

Output:  /share/logs/0013_ete_health/output.xml

Log:     /share/logs/0013_ete_health/log.html

Report:  /share/logs/0013_ete_health/report.html

command terminated with exit code 2

root@rndelato-nfs:~/oom/kubernetes#

 

 

Regards,

Manjunath.

 


From: FREEMAN, BRIAN D <bf1936@...>
Sent: Tuesday, May 12, 2020 10:10 PM
To: onap-discuss@... <onap-discuss@...>; Devangam Manjunatha <Devangam.Manjunatha@...>; CHO, TAKAMUNE <tc012c@...>; morgan.richomme@... <morgan.richomme@...>; FORSYTH, JAMES <jf2512@...>
Cc: JOMY JOSE <Jomy.Jose@...>; OLLIVIER Cédric TGI/OLN <cedric.ollivier@...>; DESBUREAUX Sylvain TGI/OLN <sylvain.desbureaux@...>
Subject: RE: [onap-discuss] [El-Alto][vCPE]- APPC not reading DMAAP messages "closed loop" event in vCPE use case

 

I only saw two AppC logs – are you running a replica of 1 or 3 ?

 

Did dmaap pass the pubsub healthcheck ?

 

If policy writes and APPC-LCM-READ doesn’t see it then its usually a dmaap write error problem where dmaap looses its connection to kafka (sometimes you have to cleanout dockerdata-nfs if kafka gets confused)

 

brian

 

From: onap-discuss@... <onap-discuss@...> On Behalf Of Devangam Manjunatha
Sent: Tuesday, May 12, 2020 12:30 PM
To: FREEMAN, BRIAN D <bf1936@...>; CHO, TAKAMUNE <tc012c@...>; morgan.richomme@...; onap-discuss@...; FORSYTH, JAMES <jf2512@...>
Cc: JOMY JOSE <Jomy.Jose@...>; OLLIVIER Cédric TGI/OLN <cedric.ollivier@...>; DESBUREAUX Sylvain TGI/OLN <sylvain.desbureaux@...>
Subject: Re: [onap-discuss] [El-Alto][vCPE]- APPC not reading DMAAP messages "closed loop" event in vCPE use case

 

Hi Brain,

 

We are testing vCPE closed loop functionality on ONAP Elalto setup.

When i trigger closed loop event, Till policy i can see the event processing. From Policy to APPC there is communication break.

Please find the attached latest log from different components.

 

In Policy network log i can see below restart request.

 

[2020-05-12T16:00:57.538+00:00|Session org.onap.policy.drools-applications.controlloop.common:controller-usecases:1.5.3:usecases][OUT|DMAAP|APPC-LCM-READ]

{

  "body": {

    "input": {

      "common-header": {

        "timestamp": "2020-05-12T16:00:57.536Z",

        "api-ver": "2.00",

        "originator-id": "53785422-d9d5-4a42-9e0a-50509b0ead5d",

        "request-id": "53785422-d9d5-4a42-9e0a-50509b0ead5d",

        "sub-request-id": "1",

        "flags": {}

      },

      "action": "Restart",

      "action-identifiers": {

        "vnf-id": "9c04c768-9284-43a2-9e2b-1a22966bc8ad"

      }

    }

  },

  "version": "2.0",

  "rpc-name": "restart",

  "correlation-id": "53785422-d9d5-4a42-9e0a-50509b0ead5d-1",

  "type": "request"

}

 

Regards,

Manjunath.

 

 

 


From: FREEMAN, BRIAN D <bf1936@...>
Sent: Tuesday, May 12, 2020 8:52 PM
To: Devangam Manjunatha <Devangam.Manjunatha@...>; CHO, TAKAMUNE <tc012c@...>; morgan.richomme@... <morgan.richomme@...>; onap-discuss@... <onap-discuss@...>; FORSYTH, JAMES <jf2512@...>
Cc: JOMY JOSE <Jomy.Jose@...>; OLLIVIER Cédric TGI/OLN <cedric.ollivier@...>; DESBUREAUX Sylvain TGI/OLN <sylvain.desbureaux@...>
Subject: RE: [onap-discuss] [El-Alto][vCPE]- APPC not reading DMAAP messages "closed loop" event in vCPE use case

 

The topic is not created until the first message from policy is written to the topic.

 

Have you triggered the closed loop through ves/tca/policy yet ?

 

You will see that topic not found error until policy does the first  write  so not sure what you were looking for ?

 

Brian

 

 

From: Devangam Manjunatha <Devangam.Manjunatha@...>
Sent: Tuesday, May 12, 2020 10:45 AM
To: CHO, TAKAMUNE <tc012c@...>; morgan.richomme@...; onap-discuss@...; FREEMAN, BRIAN D <bf1936@...>; FORSYTH, JAMES <jf2512@...>
Cc: JOMY JOSE <Jomy.Jose@...>; OLLIVIER Cédric TGI/OLN <cedric.ollivier@...>; DESBUREAUX Sylvain TGI/OLN <sylvain.desbureaux@...>
Subject: Re: [onap-discuss] [El-Alto][vCPE]- APPC not reading DMAAP messages "closed loop" event in vCPE use case

 

Hi Taka,

 

Our ONAP setup is fresh one, We installed from stack level.

 

Please find the output of the"APPC-CL" curl command below in "appc-appc-0" pod also follwed by Karaf.log from the same pod.

 

ubuntu@rndelato-nfs:~$ kubectl -it exec rnd-appc-appc-0 bash

Defaulting container name to appc.

Use 'kubectl describe pod/rnd-appc-appc-0 -n onap' to see all of the containers in this pod.

bash-4.4$

bash-4.4$

[1] 4643

bash-4.4$ []

 

 

Karaf.log which is enabled during above curl command 

 

 

2020-05-12T14:40:03,670 | INFO  | Appc-Listener-1  | HttpDmaapConsumerImpl            | 444 - appc-common-bundle - 1.6.4 | Got 0 messages from DMaaP

2020-05-12T14:40:03,670 | INFO  | Appc-Listener-1  | EventHandlerImpl                 | 444 - appc-common-bundle - 1.6.4 | Read 0 messages from APPC-LCM-READ as APPC-EVENT-LISTENER-TEST/593.

2020-05-12T14:40:03,670 | INFO  | Appc-Listener-1  | EventHandlerImpl                 | 444 - appc-common-bundle - 1.6.4 | Getting up to 10 incoming events

2020-05-12T14:40:03,671 | INFO  | Appc-Listener-1  | HttpDmaapConsumerImpl            | 444 - appc-common-bundle - 1.6.4 | GET http://message-router.onap:3904/events/APPC-LCM-READ/APPC-EVENT-LISTENER-TEST/593?timeout=60000&limit=10

2020-05-12T14:40:05,882 | INFO  | Appc-Listener-2  | HttpDmaapConsumerImpl            | 444 - appc-common-bundle - 1.6.4 | Got 0 messages from DMaaP

2020-05-12T14:40:05,883 | INFO  | Appc-Listener-2  | EventHandlerImpl                 | 444 - appc-common-bundle - 1.6.4 | Read 0 messages from APPC-CL as appcDemoEventListener/185.

2020-05-12T14:40:05,883 | INFO  | Appc-Listener-2  | EventHandlerImpl                 | 444 - appc-common-bundle - 1.6.4 | Getting up to 1000 incoming events

2020-05-12T14:40:05,883 | INFO  | Appc-Listener-2  | HttpDmaapConsumerImpl            | 444 - appc-common-bundle - 1.6.4 | GET http://message-router.onap:3904/events/APPC-CL/appcDemoEventListener/185?timeout=60000&limit=1000

2020-05-12T14:40:11,535 | INFO  | sshd-SshServer[46dcd05c]-nio2-thread-2 | ServerUserAuthService            | 143 - org.apache.sshd.core - 1.6.0 | Session karaf@/127.0.0.1:47266 authenticated

2020-05-12T14:40:13,097 | INFO  | sshd-SshServer[46dcd05c]-nio2-thread-1 | ServerUserAuthService            | 143 - org.apache.sshd.core - 1.6.0 | Session karaf@/127.0.0.1:47286 authenticated

^C

bash-4.4$

 

 

Regards,

Manjunath.

 

 


From: CHO, TAKAMUNE <tc012c@...>
Sent: Tuesday, May 12, 2020 7:12 PM
To: Devangam Manjunatha <Devangam.Manjunatha@...>; morgan.richomme@... <morgan.richomme@...>; onap-discuss@... <onap-discuss@...>; FREEMAN, BRIAN D <bf1936@...>; FORSYTH, JAMES <jf2512@...>
Cc: JOMY JOSE <Jomy.Jose@...>; OLLIVIER Cédric TGI/OLN <cedric.ollivier@...>; DESBUREAUX Sylvain TGI/OLN <sylvain.desbureaux@...>
Subject: RE: [onap-discuss] [El-Alto][vCPE]- APPC not reading DMAAP messages "closed loop" event in vCPE use case

 

The first you highlighted is related to opendaylight cluster. That could be questionable. Did you clean up pv, pvc and nfs, before you re-deployed etc?

 

You would try to turn on karaf debug log. And send a get APPC-CL curl command from appc container to see what karaf debug log says.

 

Taka

 

From: Devangam Manjunatha <Devangam.Manjunatha@...>
Sent: Tuesday, May 12, 2020 7:49 AM
To: morgan.richomme@...; CHO, TAKAMUNE <tc012c@...>; onap-discuss@...; FREEMAN, BRIAN D <bf1936@...>; FORSYTH, JAMES <jf2512@...>
Cc: JOMY JOSE <Jomy.Jose@...>; OLLIVIER Cédric TGI/OLN <cedric.ollivier@...>; DESBUREAUX Sylvain TGI/OLN <sylvain.desbureaux@...>
Subject: Re: [onap-discuss] [El-Alto][vCPE]- APPC not reading DMAAP messages "closed loop" event in vCPE use case

 

Hi Morgan,

 

Thanks for the update.

 

Hi Taka,

 

I created the "APPC-CL" topic by publishing sample message as shown below.

 

ubuntu@rndelato-nfs:~$ curl -H "Content-Type:text/plain" -X POST -d @sample.txt http://20.0.0.11:30227/events/APPC-CL

{

    "serverTimeMs": 183,

    "count": 1

}ubuntu@rndelato-nfs:~$

 

 

Now i can see "APPC-CL" in the list topics as shown below.

 

ubuntu@rndelato-nfs:~$ curl http://20.0.0.11:30227/topics

{"topics": [

    "VCPE-DHCP-EVENT",

    "org.onap.dmaap.mr.PNF_REGISTRATION",

    "unauthenticated.SEC_HEARTBEAT_OUTPUT",

    "unauthenticated.DCAE_CL_OUTPUT",

    "SDC-DISTR-STATUS-TOPIC-AUTO",

    "AAI-EVENT",

    "SDC-DISTR-NOTIF-TOPIC-AUTO",

    "org.onap.dmaap.mr.PNF_READY",

    "APPC-CL",

    "POLICY-CL-MGT",

    "POLICY-PDP-PAP",

    "POA-AUDIT-RESULT",

    "POA-AUDIT-INIT",

    "APPC-LCM-READ",

    "unauthenticated.VES_MEASUREMENT_OUTPUT",

    "org.onap.dmaap.mr.mirrormakeragent",

    "__consumer_offsets",

    "APPC-LCM-WRITE",

    "POA-RULE-VALIDATION"

]}ubuntu@rndelato-nfs:~$

 

After doing above steps, I tried to push Closed loop event.

Now i am seeing some "unreachable" messages & "SocketTimeoutException" , as shown below. Request to let us know how to overcome below issues.

Please find the attached log for more details.

 

2020-05-12T11:23:32,648 | INFO  | opendaylight-cluster-data-shard-dispatcher-109 | ShardManager                     | 273 - org.opendaylight.controller.sal-distributed-datastore - 1.8.2 | Received UnreachableMember: memberName MemberName{name=member-3}, address: akka.tcp://opendaylight-cluster-data@...:2550

2020-05-12T11:23:32,648 | INFO  | opendaylight-cluster-data-shard-dispatcher-132 | ShardManager                     | 273 - org.opendaylight.controller.sal-distributed-datastore - 1.8.2 | Received UnreachableMember: memberName MemberName{name=member-3}, address: akka.tcp://opendaylight-cluster-data@...:2550

2020-05-12T11:23:32,650 | INFO  | opendaylight-cluster-data-shard-dispatcher-97 | EntityOwnershipShard             | 265 - org.opendaylight.controller.sal-clustering-commons - 1.8.2 | member-1-shard-entity-ownership-operational: onPeerDown: PeerDown [memberName=member-3, peerId=member-3-shard-entity-ownership-operational]

2020-05-12T11:23:37,822 | INFO  | opendaylight-cluster-data-shard-dispatcher-138 | Shard                            | 265 - org.opendaylight.controller.sal-clustering-commons - 1.8.2 | member-1-shard-toaster-config (Follower): Leader akka.tcp://opendaylight-cluster-data@...:2550 is unreachable

2020-05-12T11:23:37,824 | INFO  | opendaylight-cluster-data-shard-dispatcher-138 | Shard                            | 265 - org.opendaylight.controller.sal-clustering-commons - 1.8.2 | member-1-shard-toaster-config (Candidate): Starting new election term 262

2020-05-12T11:23:37,824 | INFO  | opendaylight-cluster-data-shard-dispatcher-138 | Shard                            | 265 - org.opendaylight.controller.sal-clustering-commons - 1.8.2 | member-1-shard-toaster-config (Follower) :- Switching from behavior Follower to Candidate, election term: 262

 

 

 

2020-05-12T11:23:52,375 | INFO  | Appc-Listener-1  | HttpDmaapConsumerImpl            | 444 - appc-common-bundle - 1.6.4 | GET http://message-router.onap:3904/events/APPC-LCM-READ/APPC-EVENT-LISTENER-TEST/593?timeout=60000&limit=10

2020-05-12T11:24:11,484 | INFO  | sshd-SshServer[46dcd05c]-nio2-thread-1 | ServerUserAuthService            | 143 - org.apache.sshd.core - 1.6.0 | Session karaf@/127.0.0.1:47314 authenticated

2020-05-12T11:24:12,923 | INFO  | sshd-SshServer[46dcd05c]-nio2-thread-1 | ServerUserAuthService            | 143 - org.apache.sshd.core - 1.6.0 | Session karaf@/127.0.0.1:47328 authenticated

2020-05-12T11:24:47,212 | ERROR | Appc-Listener-2  | HttpDmaapConsumerImpl            | 444 - appc-common-bundle - 1.6.4 | Got exception while querying DMaaP. Message: Read timed out

java.net.SocketTimeoutException: Read timed out

at java.net.SocketInputStream.socketRead0(Native Method) ~[?:?]

at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) [?:?]

at java.net.SocketInputStream.read(SocketInputStream.java:171) [?:?]

at java.net.SocketInputStream.read(SocketInputStream.java:141) [?:?]

 

 

Regards,

Manjunath.

 


From: morgan.richomme@... <morgan.richomme@...>
Sent: Tuesday, May 12, 2020 1:46 PM
To: Devangam Manjunatha <Devangam.Manjunatha@...>; CHO, TAKAMUNE <tc012c@...>; onap-discuss@... <onap-discuss@...>; FREEMAN, BRIAN D <bf1936@...>; FORSYTH, JAMES <jf2512@...>
Cc: JOMY JOSE <Jomy.Jose@...>; OLLIVIER Cédric TGI/OLN <cedric.ollivier@...>; DESBUREAUX Sylvain TGI/OLN <sylvain.desbureaux@...>
Subject: RE:[onap-discuss] [El-Alto][vCPE]- APPC not reading DMAAP messages "closed loop" event in vCPE use case

 

Hi Manjunath

 

in our CI chains, when we deploy a k8s to host an ONAP we run the OPNFV functest-kubernetes suite to check that the cluster is healthy.

 

It is not a 100% guarantee but it is a good first step to see if the problem is in the infrastructure or in ONAP.

It corresponds to the k8s upstream tests repackaged by OPNFV (same way to test OpenStack, k8s and .. ONAP)

The troubleshooting infra/ONAP/VNF is not trivial. For instance ONAP is using lots of Database, including Cassandra Database that are not Ceph friendly and requires performant disk for instance. In this case ONAP and the infrastrucure are coupled.

 

Hope it helps

 

/Morgan

 


De : Devangam Manjunatha [Devangam.Manjunatha@...]
Envoyé : lundi 11 mai 2020 20:30
À : CHO, TAKAMUNE; onap-discuss@...; FREEMAN, BRIAN D; FORSYTH, JAMES; RICHOMME Morgan TGI/OLN
Cc : JOMY JOSE
Objet : Re: [onap-discuss] [El-Alto][vCPE]- APPC not reading DMAAP messages "closed loop" event in vCPE use case

Hi Morgan,

 

W.r.t below Taka's reply, our K8S setup is not stable. We heard that there is some test framework to test the stability of K8S cluster.

Will you please provide steps to test/validate our K8S cluster stability.

 

attaching logs for reference.

 

Regards,

Manjunath.

 

 


From: Devangam Manjunatha <Devangam.Manjunatha@...>
Sent: Monday, May 11, 2020 8:51 PM
To: CHO, TAKAMUNE <tc012c@...>; onap-discuss@... <onap-discuss@...>; FREEMAN, BRIAN D <bf1936@...>; FORSYTH, JAMES <jf2512@...>
Cc: JOMY JOSE <Jomy.Jose@...>
Subject: Re: [onap-discuss] [El-Alto][vCPE]- APPC not reading DMAAP messages "closed loop" event in vCPE use case

 

Looping Jimmy,

 

In "aai-aai-data-router" again we are seeing "SSLException". Our OOM is up to date with the AAI patch release till Apr-2020 (i.e. https://gerrit.onap.org/r/c/aai/oom/+/104419).

 

Note sure why we are facing again this issue.

 

2020-05-11T13:40:56.224Z||pool-3-thread-2|data-router||org.onap.aai.event.EventBusConsumer||ERROR|AC0501E|AC0501E Could not process request from event bus due to the following exception: javax.net.ssl.SSLException: SSLException invoking https://message-router.onap:3905/events/AAI-EVENT/datarouter/datarouter?timeout=1000&limit=100: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty|

javax.ws.rs.ProcessingException: javax.net.ssl.SSLException: SSLException invoking https://message-router.onap:3905/events/AAI-EVENT/datarouter/datarouter?timeout=1000&limit=100: java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty

at org.apache.cxf.jaxrs.client.AbstractClient.checkClientException(AbstractClient.java:619)

at org.apache.cxf.jaxrs.client.AbstractClient.preProcessResult(AbstractClient.java:595)

 

 

root@controller:/home/user/Naveen/oom_elalto_0904/oom/kubernetes# cd aai/components/aai-resources/resources/config/aaf

root@controller:/home/user/Naveen/oom_elalto_0904/oom/kubernetes/aai/components/aai-resources/resources/config/aaf# ls -lrt

total 32

-rw-r--r-- 1 user user   60 Apr 22 22:39 permissions.properties

-rw-r--r-- 1 user user  545 Apr 22 22:39 org.osaaf.location.props

-rw-r--r-- 1 user user  933 Apr 22 22:39 org.onap.aai.props

-rw-r--r-- 1 user user 4347 Apr 22 22:39 org.onap.aai.p12

-rw-r--r-- 1 user user 2074 Apr 22 22:39 org.onap.aai.keyfile

-rw-r--r-- 1 user user  338 Apr 22 22:39 cadi.properties

-rw-r--r-- 1 user user 1208 Apr 22 22:39 bath_config.csv

root@controller:/home/user/Naveen/oom_elalto_0904/oom/kubernetes/aai/components/aai-resources/resources/config/aaf# md5sum org.onap.aai.p12

67ae173d38ba718ea41d69a8264458c6  org.onap.aai.p12

root@controller:/home/user/Naveen/oom_elalto_0904/oom/kubernetes/aai/components/aai-resources/resources/config/aaf#

 

Please find the attached AAI log.

 

Hi Taka,

 

The patch(i.e. 

 

 

 

 

 

L&T Technology Services Ltd

www.LTTS.com

L&T Technology Services Limited (LTTS) is committed to safeguard your data privacy. For more information to view our commitment towards data privacy under GDPR, please visit the privacy policy on our website www.Ltts.com. This Email may contain confidential or privileged information for the intended recipient (s). If you are not the intended recipient, please do not use or disseminate the information, notify the sender and delete it from your system.

Join onap-discuss@lists.onap.org to automatically receive all group messages.