Topics

#dublin #aaf #aai #vfw #aai #aaf #dublin #vfw

Himesh Shukla
 

Hi all,

I have done the OOM setup in ONAP Dublin release. And working on the vFW use-case by following the steps mentioned in the link below

https://wiki.onap.org/pages/viewpage.action?pageId=68538653#DesignTime-738939159

Status:

Now, I am at the end of the Design part where in (SDC Modelling & Distribution) part of the above link, I am getting an error in aai component while Distribution of service.

After Clicking on "Monitor" to check the progress of the distribution, Where all the ONAP components are shown as deployed OK. Except aai component stuck in an error and show deployed Error as 1

Issue:

On troubleshooting on this, I could see the one of the aaf pod is under 'CrashloopBackOff' state. as shown below:

AAF-vault is in CrashLoopBackOff mode in Dublin release which is installed using rke, oom and openstack.

Can someone suggest /direct on how to get this resolved ?

 

ubuntu@onap-dub-controller-1:~/oom/kubernetes$ kubectl get pods -n onap | grep aaf

dev-aaf-aaf-cass-76c8c584f-kmr6d                        1/1     Running            0          84m

dev-aaf-aaf-cm-6d7959c65-prbng                          1/1     Running            0          84m

dev-aaf-aaf-fs-9898d569-cmqtx                           1/1     Running            0          84m

dev-aaf-aaf-gui-586484467-cc8fh                         1/1     Running            0          84m

dev-aaf-aaf-locate-6867c7fb9d-hvqbp                     1/1     Running            0          84m

dev-aaf-aaf-oauth-59cb84d56d-r574d                      1/1     Running            0          84m

dev-aaf-aaf-service-5c598d6fb-hp8rf                     1/1     Running            0          84m

dev-aaf-aaf-sms-7c4474c74b-m4bw8                        0/1     Init:0/1           8          84m

dev-aaf-aaf-sms-preload-bvtts                           0/1     Init:0/1           8          84m

dev-aaf-aaf-sms-quorumclient-0                          1/1     Running            0          84m

dev-aaf-aaf-sms-quorumclient-1                          1/1     Running            0          84m

dev-aaf-aaf-sms-quorumclient-2                          1/1     Running            0          84m

dev-aaf-aaf-sms-vault-0                                 0/2     CrashLoopBackOff   42         84m

dev-aaf-aaf-sshsm-distcenter-pxw8p                      0/1     Completed          0          84m

dev-aaf-aaf-sshsm-testca-bhkjw                          0/1     Completed          0          84m

 

ubuntu@onap-dub-controller-1:~/oom/kubernetes$ kubectl describe pod dev-aaf-aaf-sms-vault-0 -n onap

Name:               dev-aaf-aaf-sms-vault-0

Namespace:          onap

Priority:           0

PriorityClassName:  <none>

Node:               onap-dub-worker-1/192.168.1.11

Start Time:         Fri, 29 Nov 2019 11:48:39 +0000

Labels:             app=aaf-sms-vault

                    controller-revision-hash=dev-aaf-aaf-sms-vault-568cb984

                    release=dev-aaf

                    statefulset.kubernetes.io/pod-name=dev-aaf-aaf-sms-vault-0

Annotations:        cni.projectcalico.org/podIP: 192.166.2.140/32

Status:             Running

IP:                 192.166.2.140

Controlled By:      StatefulSet/dev-aaf-aaf-sms-vault

Containers:

  aaf-sms-vault:

    Container ID:  docker://c81435b46fb32f0c2c31c783b9dd1aa6fef5565c884d336209b125cb742f4f9d

    Image:         nexus3.onap.org:10001/library/vault:0.10.0

    Image ID:      docker-pullable://nexus3.onap.org:10001/library/vault@sha256:2b8fcff0340e927bf621a518fa268093365c3e7e0f7c2f3c9895c75f1c3bc233

    Port:          8200/TCP

    Host Port:     0/TCP

    Command:

      vault

      server

      -config

      /vault/config/config.json

    State:          Waiting

      Reason:       CrashLoopBackOff

    Last State:     Terminated

      Reason:       Error

      Exit Code:    1

      Started:      Fri, 29 Nov 2019 13:11:47 +0000

      Finished:     Fri, 29 Nov 2019 13:11:47 +0000

    Ready:          False

    Restart Count:  21

    Limits:

      cpu:     200m

      memory:  500Mi

    Requests:

      cpu:        10m

      memory:     10Mi

    Environment:  <none>

    Mounts:

      /etc/localtime from localtime (ro)

      /var/run/secrets/kubernetes.io/serviceaccount from default-token-55bjh (ro)

      /vault/config/config.json from dev-aaf-aaf-sms-vault-vault (rw)

  aaf-sms-vault-backend:

    Container ID:  docker://8307dee9adebdb55cfb8a5cb53f2695bc63c6c772c79e352d128aec429d195c9

    Image:         nexus3.onap.org:10001/library/consul:1.0.6

    Image ID:      docker-pullable://nexus3.onap.org:10001/library/consul@sha256:21e67b49c92a2d2d62b46bb7c054cbe05470c7dc05263384ede11506a39914f2

    Port:          8500/TCP

    Host Port:     0/TCP

    Command:

      consul

      agent

      -server

      -client

      0.0.0.0

      -bootstrap-expect=1

      -config-file

      /consul/config/config.json

    State:          Waiting

      Reason:       CrashLoopBackOff

    Last State:     Terminated

      Reason:       Error

      Exit Code:    1

      Started:      Fri, 29 Nov 2019 13:11:48 +0000

      Finished:     Fri, 29 Nov 2019 13:11:48 +0000

    Ready:          False

    Restart Count:  21

    Limits:

      cpu:     200m

      memory:  500Mi

    Requests:

      cpu:        10m

      memory:     10Mi

    Environment:  <none>

    Mounts:

      /consul/config/config.json from dev-aaf-aaf-sms-vault-consulconfiguration (rw)

      /consul/data from dev-aaf-aaf-sms-vault-consuldata (rw)

      /etc/localtime from localtime (ro)

      /var/run/secrets/kubernetes.io/serviceaccount from default-token-55bjh (ro)

Conditions:

  Type              Status

  Initialized       True

  Ready             False

  ContainersReady   False

  PodScheduled      True

Volumes:

  dev-aaf-aaf-sms-vault-consuldata:

    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)

    ClaimName:  dev-aaf-aaf-sms-vault

    ReadOnly:   false

  dev-aaf-aaf-sms-vault-consulconfiguration:

    Type:      ConfigMap (a volume populated by a ConfigMap)

    Name:      dev-aaf-aaf-sms-vault-consul

    Optional:  false

  dev-aaf-aaf-sms-vault-vault:

    Type:      ConfigMap (a volume populated by a ConfigMap)

    Name:      dev-aaf-aaf-sms-vault-vault

    Optional:  false

  localtime:

    Type:          HostPath (bare host directory volume)

    Path:          /etc/localtime

    HostPathType:

  default-token-55bjh:

    Type:        Secret (a volume populated by a Secret)

    SecretName:  default-token-55bjh

    Optional:    false

QoS Class:       Burstable

Node-Selectors:  <none>

Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s

                 node.kubernetes.io/unreachable:NoExecute for 300s

Events:

  Type     Reason   Age                    From                        Message

  ----     ------   ----                   ----                        -------

  Warning  BackOff  5m39s (x392 over 85m)  kubelet, onap-dub-worker-1  Back-off restarting failed container

  Warning  BackOff  46s (x419 over 85m)    kubelet, onap-dub-worker-1  Back-off restarting failed container

 

 

ubuntu@onap-dub-controller-1:~/oom/kubernetes$ kubectl logs -n onap dev-aaf-aaf-sms-vault-0 -c aaf-sms-vault

Error detecting api address: Get http://localhost:8500/v1/agent/self: dial tcp 127.0.0.1:8500: connect: connection refused

Error initializing core: missing API address, please set in configuration or via environment

ubuntu@onap-dub-controller-1:~/oom/kubernetes$ kubectl logs -n onap dev-aaf-aaf-sms-vault-0 -c aaf-sms-vault-backend

==> No private IPv4 address found

ubuntu@onap-dub-controller-1:~/oom/kubernetes$

 

I have Tried deleting and re-deploying this pod multiple times didn't solve the issue. I have tried deleting whole aaf component and re-deploying it, But it didn't solved it either.

I have come across few similar issues in the onap-discuss forum, but didn't see any solution to this issue. Hence raising this issue in forum

Any help with this is much appreciated. Thanks in advance

 

Regards,

Himesh Shukla