Re: [oom][sdnc] Just deploy sdnc?


Michael O'Brien <frank.obrien@...>
 

Curtis,

   Hi, I was meaning on validating all the dependencies that are enforced in the charts anyway during the cd.sh changes to repeatedly bring up the system in sequential-dependent order for LOG-898 and TSC-25.

 

   There are 105 dependencies in the charts in Mandeep’s init container –container-name references off spec:initContainers: (a generated spec will take some work to navigate some indirect references).   These dependencies force a deploy order – not all are specified and we may want to make them all conditional to aide dev deployments – however without the dependencies REST functionality will be limited.

 

   Anyway you will need dmaap, consul and sdc (specifically message-router, consul and sdc-be) to be up in order for the 2 ueb and dmaap listeners to go past init state in sdnc – there is no wider tree in this particular branch.  For example if you deploy only dmaap and consul and then sdnc – the ueb container will wait for sdc but everything else will start in 3 min after a first deploy of 3 min for dmaap and consul.  One of the later pods to come up will be dgbuilder but it has no dependencies except to the internal db container.

 

   Details at

https://wiki.onap.org/display/DW/Log+Streaming+Compliance+and+API#LogStreamingComplianceandAPI-DeploymentDependencyTree

 

   Testing

   Notice the 2 phase deployment, dmaap, sdc, ueb first then sdnc second.

   Note: adding forced dependencies has the drawback of the pods not coming up without a large memory footprint – requiring a 32g vm – some rest/api dependencies like AAI are not specified – ideally we follow the AAI approach and add conditional dependencies keyed on the –set enabled=true/false flags for particular sub-pods.

 

sudo nohup ./cd.sh -b 3.0.0-ONAP -e onap -p false -n nexus3.onap.org:10001 -f true -s 1200 -c false -d true -w false -r false &

 

 

amdocs@obriensystemsu1:~$ kubectl get pods --all-namespaces

NAMESPACE     NAME                                                   READY     STATUS       RESTARTS   AGE

kube-system   heapster-7b48b696fc-qg85t                              1/1       Running      0          7h

kube-system   kube-dns-6655f78c68-96pgc                              3/3       Running      0          7h

kube-system   kubernetes-dashboard-6f54f7c4b-p42cw                   1/1       Running      0          7h

kube-system   monitoring-grafana-7877679464-v5hjs                    1/1       Running      0          7h

kube-system   monitoring-influxdb-64664c6cf5-zmxwb                   1/1       Running      0          7h

kube-system   tiller-deploy-6f4745cbcf-bqmcf                         1/1       Running      0          7h

onap          onap-consul-consul-849447d678-lmcs7                    1/1       Running      0          1h

onap          onap-consul-consul-server-0                            1/1       Running      0          1h

onap          onap-consul-consul-server-1                            1/1       Running      0          59m

onap          onap-consul-consul-server-2                            1/1       Running      0          59m

onap          onap-dmaap-dbc-pg-0                                    1/1       Running      0          1h

onap          onap-dmaap-dbc-pg-1                                    1/1       Running      0          59m

onap          onap-dmaap-dbc-pgpool-c5f8498-dmh69                    1/1       Running      0          1h

onap          onap-dmaap-dbc-pgpool-c5f8498-mmd9z                    1/1       Running      0          1h

onap          onap-dmaap-dmaap-bus-controller-557dc8c59c-5rzpm       1/1       Running      0          1h

onap          onap-dmaap-dmaap-dr-db-576f7968b8-r77gv                1/1       Running      0          1h

onap          onap-dmaap-dmaap-dr-node-7647f9d6d8-qmkj2              1/1       Running      0          1h

onap          onap-dmaap-dmaap-dr-prov-f4d84869f-k78pg               1/1       Running      0          1h

onap          onap-dmaap-message-router-76f4799d-c9cjq               1/1       Running      0          1h

onap          onap-dmaap-message-router-kafka-56c7c546c-tb6br        1/1       Running      0          1h

onap          onap-dmaap-message-router-zookeeper-7d69b496bc-5fgkf   1/1       Running      0          1h

onap          onap-sdc-sdc-be-66cb559689-fkpc7                       2/2       Running      0          1h

onap          onap-sdc-sdc-be-config-backend-gqfh8                   0/1       Completed    0          1h

onap          onap-sdc-sdc-cs-759f5d9d79-p27n6                       1/1       Running      0          1h

onap          onap-sdc-sdc-cs-config-cassandra-7w2wf                 0/1       Completed    0          1h

onap          onap-sdc-sdc-dcae-be-5b55b7dc9c-5nbdw                  2/2       Running      0          1h

onap          onap-sdc-sdc-dcae-be-tools-c6dkf                       0/1       Completed    0          50m

onap          onap-sdc-sdc-dcae-be-tools-frwhs                       0/1       Init:Error   0          1h

onap          onap-sdc-sdc-dcae-dt-5cd48f7598-24w4g                  2/2       Running      0          1h

onap          onap-sdc-sdc-dcae-fe-6c6f66664b-lbnsh                  2/2       Running      0          1h

onap          onap-sdc-sdc-dcae-tosca-lab-7576c79d74-hv58w           2/2       Running      0          1h

onap          onap-sdc-sdc-es-8d55bd56-bxlr2                         1/1       Running      0          1h

onap          onap-sdc-sdc-es-config-elasticsearch-88hdt             0/1       Completed    0          1h

onap          onap-sdc-sdc-fe-68c87b558-s887r                        2/2       Running      0          1h

onap          onap-sdc-sdc-kb-6cb4d7d57d-dcfx6                       1/1       Running      0          1h

onap          onap-sdc-sdc-onboarding-be-7876b44956-plbxv            2/2       Running      0          1h

onap          onap-sdc-sdc-onboarding-be-cassandra-init-dskg5        0/1       Completed    0          1h

onap          onap-sdc-sdc-wfd-be-85758b765-q26pb                    1/1       Running      0          1h

onap          onap-sdc-sdc-wfd-be-workflow-init-sxq5b                0/1       Completed    0          1h

onap          onap-sdc-sdc-wfd-fe-7947c855d8-7dt22                   2/2       Running      0          1h

onap          onap-sdnc-controller-blueprints-bc66576c5-kk68m        1/1       Running      1          40m

onap          onap-sdnc-controller-blueprints-db-0                   1/1       Running      0          40m

onap          onap-sdnc-nengdb-0                                     1/1       Running      0          40m

onap          onap-sdnc-network-name-gen-7f95f5bfc8-l86ns            1/1       Running      0          40m

onap          onap-sdnc-sdnc-0                                       2/2       Running      0          40m

onap          onap-sdnc-sdnc-ansible-server-7d595dd8-lx2hz           1/1       Running      0          40m

onap          onap-sdnc-sdnc-db-0                                    2/2       Running      0          40m

onap          onap-sdnc-sdnc-dgbuilder-7c8fcbff77-hk7j2              1/1       Running      0          40m

onap          onap-sdnc-sdnc-dmaap-listener-fd8bf55cf-hjns6          1/1       Running      0          40m

onap          onap-sdnc-sdnc-portal-55976f4bd-vf62t                  1/1       Running      2          40m

onap          onap-sdnc-sdnc-ueb-listener-7bf689f8b9-dr9xh           1/1       Running      0          40m

amdocs@obriensystemsu1:~$ kubectl top nodes

NAME              CPU(cores)   CPU%      MEMORY(bytes)   MEMORY%

obriensystemsu1   2228m        18%       26002Mi         48%

amdocs@obriensystemsu1:~$ free

              total        used        free      shared  buff/cache   available

Mem:       55339300    18195324     1908292      100980    35235684    36047660

Swap:       9289724        4236     9285488

amdocs@obriensystemsu1:~$ df

Filesystem                           1K-blocks     Used Available Use% Mounted on

udev                                  27647576        0  27647576   0% /dev

tmpfs                                  5533932    32532   5501400   1% /run

/dev/mapper/obriensystemsu0--vg-root 155367356 93401612  54050432  64% /

tmpfs                                 27669648    23600  27646048   1% /dev/shm

tmpfs                                     5120        4      5116   1% /run/lock

tmpfs                                 27669648        0  27669648   0% /sys/fs/cgroup

/dev/sda1                               482922    62963    395025  14% /boot

tmpfs                                  5533932       64   5533868   1% /run/user/1000

amdocs@obriensystemsu1:~$ sudo helm list

[sudo] password for amdocs:

NAME            REVISION        UPDATED                         STATUS          CHART           NAMESPACE

onap            3               Sun Jan 13 19:25:55 2019        DEPLOYED        onap-3.0.0      onap

onap-consul     2               Sun Jan 13 19:25:57 2019        DEPLOYED        consul-3.0.0    onap

onap-dmaap      2               Sun Jan 13 19:25:58 2019        DEPLOYED        dmaap-3.0.0     onap

onap-sdc        2               Sun Jan 13 19:26:03 2019        DEPLOYED        sdc-3.0.0       onap

onap-sdnc       1               Sun Jan 13 19:26:04 2019        DEPLOYED        sdnc-3.0.0      onap

 

Use my script and break up the deployment into dmaap, sdc and consul in dev0.yaml and sdnc in dev1.yaml – or use the -f option and put the 4 in sequence in the list

https://gerrit.onap.org/r/#/c/75422/

 

Note: not all charts make their dependencies explicit – and not all REST/API and compile/runtime-reflection (pom.xml) dependencies are reflected in the current list.

These are specified so far.

 

We will discuss this at the next PTL meet

https://wiki.onap.org/display/DW/PTL+2019-01-14

 

Thank you

/michae

 

From: onap-discuss@... <onap-discuss@...> On Behalf Of Curtis
Sent: Friday, January 11, 2019 7:43 AM
To: onap-discuss@...; Michael O'Brien <Frank.Obrien@...>
Subject: Re: [onap-discuss] [oom][sdnc] Just deploy sdnc?

 

 

 

On Thu, Jan 10, 2019 at 4:09 PM Michael O'Brien <frank.obrien@...> wrote:

Curtis,

   Hello fellow onap deployment enthusiast.

 

Hi :)

 

   Your question is very relevant for developer environments and limited memory VM’s like a dev kubernetes cluster running on a laptop.

   I am trying to keep track of issues I also run into here.

   https://wiki.onap.org/display/DW/ONAP+Development#ONAPDevelopment-DeveloperDeployment

 

   Ideally – each onap pod in the values.yaml can be deployed independently – in general I have seen this – there are the odd non-atomic check between pods like sdc to dmaap for their hc.

   In general for runtime at least you will see issues with dependent microservices – streaming logs from the filebeat sidecar will fail for example if log-logstash is not up – however 2/2 and 3/3 pods including a filebeat container are not blocked by runtime – and will start fine.

    In your case the sdnc-0 filebeat is not working because the emptydir pv is likely not up yet between the 2 pods.

   You can consult one possible hierarchy of deployment in the review https://gerrit.onap.org/r/#/c/75422/

 

   For your init failures like dmaap – I would expect they should not be blocked by no dmaap pods – but I usually run with at least the base pods aaf,dmaap and aai if I run any other pod like sdnc, appd, sdc, so etc…

    Is there a reason you are not running with some minimal subset – in one of my development cases I run with just robot and aai or robot and log/pomba/dmaap – however I did not exercise all the functionality to verify this scenario was fully functional.

 

There's no specific reason, to be honest I'm just not sure what is a reasonable subset. :) Mostly I am exploring how to deploy pieces of ONAP using OOM instead of the entirety. sdnc seemed a good place to start, but it was a rather arbitrary decision on my part.

 

Thanks very much for the above information! I may have a few questions after I go through the what you have discussed, but this is extremely helpful and exactly the kind of information I need. :)

 

Thanks,

Curtis

 

    /michael

 

3.0.0-ONAP Casablanca tag from yesterday

onap onap-sdnc-nengdb-0 1/1 Running 0 5h

onap onap-sdnc-network-name-gen-7f95f5bfc8-92jdx 1/1 Running 0 5h

onap onap-sdnc-sdnc-0 2/2 Running 0 5h

onap onap-sdnc-sdnc-ansible-server-7d595dd8-rp5rc 1/1 Running 0 5h

onap onap-sdnc-sdnc-db-0 2/2 Running 0 5h

onap onap-sdnc-sdnc-dgbuilder-7c8fcbff77-9drhc 1/1 Running 0 5h

onap onap-sdnc-sdnc-dmaap-listener-fd8bf55cf-qqqwz 1/1 Running 0 5h

onap onap-sdnc-sdnc-portal-55976f4bd-qpd6x 1/1 Running 0 5h

onap onap-sdnc-sdnc-ueb-listener-7bf689f8b9-q2p4x 1/1 Running 0 5h

 

 

A rule of thumb – if you don’t see the pod up within an hour – it will likely not come up  - this is a timeout limitation of kubernetes jobs – that is being fixed by moving to helm hooks in Dublin – in that case restart your pod – flip the enabled flag – however an important point – delete the /dockerdata-nfs/onap-<pod> dir – as some pods will not rerun their config job if the directory is populated.

 

Also note the deletion cleanup steps – also for rogue pods

https://wiki.onap.org/display/DW/ONAP+Development#ONAPDevelopment-RemoveaDeployment

 

do a describe on your failed pod to get info – it looks like sdnc-0 is waiting on the dmaap and ueb listeners – check requirements.yaml hierarchy to be sure.

casablanca-sdnc-0                                  1/2     CrashLoopBackOff   46         3h31m

 

From: onap-discuss@

lists.onap.org <onap-discuss@...> On Behalf Of Curtis
Sent: Wednesday, January 9, 2019 5:55 PM
To: onap-discuss@...
Subject: [onap-discuss] [oom][sdnc] Just deploy sdnc?

 

Hi,

 

I'd just like to deploy sdnc if possible. I have turned off everything in the onap/values.yml file except sdnc. It seems to have deployed, I can access the portal, but there are a few pods which seem unnecessary, though I'm not completely sure--am new to sdnc.  (For sdnc-0 filebeat is not working.) dmaap listening and ueb listening are waiting for the check script to find message-router and sdc-be.

 

If anyone has any recommendations or concerns around just deploying sdn-c, let me know. :) Otherwise will continue working through it. I just thought I'd mention this.

 

Current pods:

 

$ k get pods -n onap
NAME                                               READY   STATUS             RESTARTS   AGE
casablanca-controller-blueprints-84dfd855b-phplq   1/1     Running            0          4h19m
casablanca-controller-blueprints-db-0              1/1     Running            0          4h19m
casablanca-nengdb-0                                1/1     Running            0          4h19m
casablanca-network-name-gen-5d4847f747-th2t2       1/1     Running            0          4h19m
casablanca-sdnc-0                                  1/2     CrashLoopBackOff   46         3h31m
casablanca-sdnc-ansible-server-5c9bb57797-tvf7x    1/1     Running            0          4h19m
casablanca-sdnc-db-0                               2/2     Running            0          4h19m
casablanca-sdnc-dgbuilder-75b88dd7cd-xfjcm         1/1     Running            0          4h19m
casablanca-sdnc-dmaap-listener-9d4f55c64-s94sc     0/1     Init:0/1           1          4h19m
casablanca-sdnc-portal-597966f975-bnvrd            1/1     Running            0          4h19m
casablanca-sdnc-ueb-listener-79c9cc6fb6-hnvnj      0/1     Init:0/1           0          4h19m

 

Thanks kindly,

Curtis

This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service



--

This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service

Join onap-discuss@lists.onap.org to automatically receive all group messages.