[oom][sdnc] Just deploy sdnc?
|
Curtis <serverascode@...>
Hi, I'd just like to deploy sdnc if possible. I have turned off everything in the onap/values.yml file except sdnc. It seems to have deployed, I can access the portal, but there are a few pods which seem unnecessary, though I'm not completely sure--am new to sdnc. (For sdnc-0 filebeat is not working.) dmaap listening and ueb listening are waiting for the check script to find message-router and sdc-be. If anyone has any recommendations or concerns around just deploying sdn-c, let me know. :) Otherwise will continue working through it. I just thought I'd mention this. Current pods: $ k get pods -n onap NAME READY STATUS RESTARTS AGE casablanca-controller-blueprints-84dfd855b-phplq 1/1 Running 0 4h19m casablanca-controller-blueprints-db-0 1/1 Running 0 4h19m casablanca-nengdb-0 1/1 Running 0 4h19m casablanca-network-name-gen-5d4847f747-th2t2 1/1 Running 0 4h19m casablanca-sdnc-0 1/2 CrashLoopBackOff 46 3h31m casablanca-sdnc-ansible-server-5c9bb57797-tvf7x 1/1 Running 0 4h19m casablanca-sdnc-db-0 2/2 Running 0 4h19m casablanca-sdnc-dgbuilder-75b88dd7cd-xfjcm 1/1 Running 0 4h19m casablanca-sdnc-dmaap-listener-9d4f55c64-s94sc 0/1 Init:0/1 1 4h19m casablanca-sdnc-portal-597966f975-bnvrd 1/1 Running 0 4h19m casablanca-sdnc-ueb-listener-79c9cc6fb6-hnvnj 0/1 Init:0/1 0 4h19m Thanks kindly, Curtis |
|
|
|
Michael O'Brien <frank.obrien@...>
Curtis, Hello fellow onap deployment enthusiast. Your question is very relevant for developer environments and limited memory VM’s like a dev kubernetes cluster running on a laptop. I am trying to keep track of issues I also run into here. https://wiki.onap.org/display/DW/ONAP+Development#ONAPDevelopment-DeveloperDeployment
Ideally – each onap pod in the values.yaml can be deployed independently – in general I have seen this – there are the odd non-atomic check between pods like sdc to dmaap for their hc. In general for runtime at least you will see issues with dependent microservices – streaming logs from the filebeat sidecar will fail for example if log-logstash is not up – however 2/2 and 3/3 pods including a filebeat container are not blocked by runtime – and will start fine. In your case the sdnc-0 filebeat is not working because the emptydir pv is likely not up yet between the 2 pods. You can consult one possible hierarchy of deployment in the review https://gerrit.onap.org/r/#/c/75422/
For your init failures like dmaap – I would expect they should not be blocked by no dmaap pods – but I usually run with at least the base pods aaf,dmaap and aai if I run any other pod like sdnc, appd, sdc, so etc… Is there a reason you are not running with some minimal subset – in one of my development cases I run with just robot and aai or robot and log/pomba/dmaap – however I did not exercise all the functionality to verify this scenario was fully functional. /michael
3.0.0-ONAP Casablanca tag from yesterday onap onap-sdnc-nengdb-0 1/1 Running 0 5h onap onap-sdnc-network-name-gen-7f95f5bfc8-92jdx 1/1 Running 0 5h onap onap-sdnc-sdnc-0 2/2 Running 0 5h onap onap-sdnc-sdnc-ansible-server-7d595dd8-rp5rc 1/1 Running 0 5h onap onap-sdnc-sdnc-db-0 2/2 Running 0 5h onap onap-sdnc-sdnc-dgbuilder-7c8fcbff77-9drhc 1/1 Running 0 5h onap onap-sdnc-sdnc-dmaap-listener-fd8bf55cf-qqqwz 1/1 Running 0 5h onap onap-sdnc-sdnc-portal-55976f4bd-qpd6x 1/1 Running 0 5h onap onap-sdnc-sdnc-ueb-listener-7bf689f8b9-q2p4x 1/1 Running 0 5h
A rule of thumb – if you don’t see the pod up within an hour – it will likely not come up - this is a timeout limitation of kubernetes jobs – that is being fixed by moving to helm hooks in Dublin – in that case restart your pod – flip the enabled flag – however an important point – delete the /dockerdata-nfs/onap-<pod> dir – as some pods will not rerun their config job if the directory is populated.
Also note the deletion cleanup steps – also for rogue pods https://wiki.onap.org/display/DW/ONAP+Development#ONAPDevelopment-RemoveaDeployment
do a describe on your failed pod to get info – it looks like sdnc-0 is waiting on the dmaap and ueb listeners – check requirements.yaml hierarchy to be sure. casablanca-sdnc-0 1/2 CrashLoopBackOff 46 3h31m
From: onap-discuss@ lists.onap.org <onap-discuss@...> On Behalf Of
Curtis
Hi,
I'd just like to deploy sdnc if possible. I have turned off everything in the onap/values.yml file except sdnc. It seems to have deployed, I can access the portal, but there are a few pods which seem unnecessary, though I'm not completely sure--am new to sdnc. (For sdnc-0 filebeat is not working.) dmaap listening and ueb listening are waiting for the check script to find message-router and sdc-be.
If anyone has any recommendations or concerns around just deploying sdn-c, let me know. :) Otherwise will continue working through it. I just thought I'd mention this.
Current pods:
$ k get pods -n onap
Thanks kindly, Curtis This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service |
|
|
|
Michael O'Brien <frank.obrien@...>
Forgot a point about the onap namespace that applies to everyone.
Namespace=onap
If you are not deploying more than one deployment of onap (2 sdnc’s for example) – name the namespace onap – as I understand there are still a couple instances of expecting a hardcoded onap namespace. I understand there is a jira on allowing for more than 1 deployment of onap per kubernetes cluster that will fix this – I just cannot find it.
Use “onap” and you will reduce one more wildcard when triaging any failed pods
From: onap-discuss@... <onap-discuss@...>
On Behalf Of Michael O'Brien
Sent: Thursday, January 10, 2019 4:09 PM To: onap-discuss@...; serverascode@... Subject: Re: [onap-discuss] [oom][sdnc] Just deploy sdnc?
Curtis, Hello fellow onap deployment enthusiast. Your question is very relevant for developer environments and limited memory VM’s like a dev kubernetes cluster running on a laptop. I am trying to keep track of issues I also run into here. https://wiki.onap.org/display/DW/ONAP+Development#ONAPDevelopment-DeveloperDeployment
Ideally – each onap pod in the values.yaml can be deployed independently – in general I have seen this – there are the odd non-atomic check between pods like sdc to dmaap for their hc. In general for runtime at least you will see issues with dependent microservices – streaming logs from the filebeat sidecar will fail for example if log-logstash is not up – however 2/2 and 3/3 pods including a filebeat container are not blocked by runtime – and will start fine. In your case the sdnc-0 filebeat is not working because the emptydir pv is likely not up yet between the 2 pods. You can consult one possible hierarchy of deployment in the review https://gerrit.onap.org/r/#/c/75422/
For your init failures like dmaap – I would expect they should not be blocked by no dmaap pods – but I usually run with at least the base pods aaf,dmaap and aai if I run any other pod like sdnc, appd, sdc, so etc… Is there a reason you are not running with some minimal subset – in one of my development cases I run with just robot and aai or robot and log/pomba/dmaap – however I did not exercise all the functionality to verify this scenario was fully functional. /michael
3.0.0-ONAP Casablanca tag from yesterday onap onap-sdnc-nengdb-0 1/1 Running 0 5h onap onap-sdnc-network-name-gen-7f95f5bfc8-92jdx 1/1 Running 0 5h onap onap-sdnc-sdnc-0 2/2 Running 0 5h onap onap-sdnc-sdnc-ansible-server-7d595dd8-rp5rc 1/1 Running 0 5h onap onap-sdnc-sdnc-db-0 2/2 Running 0 5h onap onap-sdnc-sdnc-dgbuilder-7c8fcbff77-9drhc 1/1 Running 0 5h onap onap-sdnc-sdnc-dmaap-listener-fd8bf55cf-qqqwz 1/1 Running 0 5h onap onap-sdnc-sdnc-portal-55976f4bd-qpd6x 1/1 Running 0 5h onap onap-sdnc-sdnc-ueb-listener-7bf689f8b9-q2p4x 1/1 Running 0 5h
A rule of thumb – if you don’t see the pod up within an hour – it will likely not come up - this is a timeout limitation of kubernetes jobs – that is being fixed by moving to helm hooks in Dublin – in that case restart your pod – flip the enabled flag – however an important point – delete the /dockerdata-nfs/onap-<pod> dir – as some pods will not rerun their config job if the directory is populated.
Also note the deletion cleanup steps – also for rogue pods https://wiki.onap.org/display/DW/ONAP+Development#ONAPDevelopment-RemoveaDeployment
do a describe on your failed pod to get info – it looks like sdnc-0 is waiting on the dmaap and ueb listeners – check requirements.yaml hierarchy to be sure. casablanca-sdnc-0 1/2 CrashLoopBackOff 46 3h31m
From: onap-discuss@ lists.onap.org <onap-discuss@...>
On Behalf Of Curtis
Hi,
I'd just like to deploy sdnc if possible. I have turned off everything in the onap/values.yml file except sdnc. It seems to have deployed, I can access the portal, but there are a few pods which seem unnecessary, though I'm not completely sure--am new to sdnc. (For sdnc-0 filebeat is not working.) dmaap listening and ueb listening are waiting for the check script to find message-router and sdc-be.
If anyone has any recommendations or concerns around just deploying sdn-c, let me know. :) Otherwise will continue working through it. I just thought I'd mention this.
Current pods:
$ k get pods -n onap
Thanks kindly, Curtis This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service |
|
|
|
Alexis de Talhouet
Hi
These 4 aren’t needed if you’re not trying to play with CDS (tool use to model the resources needed to instantiate a service, along with how to resolve them). Unfortunately we didn’t add knob to disable this, but you could still edit the sdnc/requirements.yaml file to remove to following charts: - controller-blueprints - network-name-gen Thanks, Alexis |
|
|
|
Curtis <serverascode@...>
On Thu, Jan 10, 2019 at 4:09 PM Michael O'Brien <frank.obrien@...> wrote:
Hi :)
There's no specific reason, to be honest I'm just not sure what is a reasonable subset. :) Mostly I am exploring how to deploy pieces of ONAP using OOM instead of the entirety. sdnc seemed a good place to start, but it was a rather arbitrary decision on my part. Thanks very much for the above information! I may have a few questions after I go through the what you have discussed, but this is extremely helpful and exactly the kind of information I need. :) Thanks, Curtis
-- Blog: serverascode.com |
|
|
|
Curtis <serverascode@...>
On Thu, Jan 10, 2019 at 5:38 PM Alexis de Talhouët <adetalhouet89@...> wrote:
Thanks kindly, I was wondering if some of the components could be removed. Much appreciated!
-- Blog: serverascode.com |
|
|
|
Michael O'Brien <frank.obrien@...>
Curtis, Hi, I was meaning on validating all the dependencies that are enforced in the charts anyway during the cd.sh changes to repeatedly bring up the system in sequential-dependent order for LOG-898 and TSC-25.
There are 105 dependencies in the charts in Mandeep’s init container –container-name references off spec:initContainers: (a generated spec will take some work to navigate some indirect references). These dependencies force a deploy order – not all are specified and we may want to make them all conditional to aide dev deployments – however without the dependencies REST functionality will be limited.
Anyway you will need dmaap, consul and sdc (specifically message-router, consul and sdc-be) to be up in order for the 2 ueb and dmaap listeners to go past init state in sdnc – there is no wider tree in this particular branch. For example if you deploy only dmaap and consul and then sdnc – the ueb container will wait for sdc but everything else will start in 3 min after a first deploy of 3 min for dmaap and consul. One of the later pods to come up will be dgbuilder but it has no dependencies except to the internal db container.
Details at
Testing Notice the 2 phase deployment, dmaap, sdc, ueb first then sdnc second. Note: adding forced dependencies has the drawback of the pods not coming up without a large memory footprint – requiring a 32g vm – some rest/api dependencies like AAI are not specified – ideally we follow the AAI approach and add conditional dependencies keyed on the –set enabled=true/false flags for particular sub-pods.
sudo nohup ./cd.sh -b 3.0.0-ONAP -e onap -p false -n nexus3.onap.org:10001 -f true -s 1200 -c false -d true -w false -r false &
amdocs@obriensystemsu1:~$ kubectl get pods --all-namespaces NAMESPACE NAME READY STATUS RESTARTS AGE kube-system heapster-7b48b696fc-qg85t 1/1 Running 0 7h kube-system kube-dns-6655f78c68-96pgc 3/3 Running 0 7h kube-system kubernetes-dashboard-6f54f7c4b-p42cw 1/1 Running 0 7h kube-system monitoring-grafana-7877679464-v5hjs 1/1 Running 0 7h kube-system monitoring-influxdb-64664c6cf5-zmxwb 1/1 Running 0 7h kube-system tiller-deploy-6f4745cbcf-bqmcf 1/1 Running 0 7h onap onap-consul-consul-849447d678-lmcs7 1/1 Running 0 1h onap onap-consul-consul-server-0 1/1 Running 0 1h onap onap-consul-consul-server-1 1/1 Running 0 59m onap onap-consul-consul-server-2 1/1 Running 0 59m onap onap-dmaap-dbc-pg-0 1/1 Running 0 1h onap onap-dmaap-dbc-pg-1 1/1 Running 0 59m onap onap-dmaap-dbc-pgpool-c5f8498-dmh69 1/1 Running 0 1h onap onap-dmaap-dbc-pgpool-c5f8498-mmd9z 1/1 Running 0 1h onap onap-dmaap-dmaap-bus-controller-557dc8c59c-5rzpm 1/1 Running 0 1h onap onap-dmaap-dmaap-dr-db-576f7968b8-r77gv 1/1 Running 0 1h onap onap-dmaap-dmaap-dr-node-7647f9d6d8-qmkj2 1/1 Running 0 1h onap onap-dmaap-dmaap-dr-prov-f4d84869f-k78pg 1/1 Running 0 1h onap onap-dmaap-message-router-76f4799d-c9cjq 1/1 Running 0 1h onap onap-dmaap-message-router-kafka-56c7c546c-tb6br 1/1 Running 0 1h onap onap-dmaap-message-router-zookeeper-7d69b496bc-5fgkf 1/1 Running 0 1h onap onap-sdc-sdc-be-66cb559689-fkpc7 2/2 Running 0 1h onap onap-sdc-sdc-be-config-backend-gqfh8 0/1 Completed 0 1h onap onap-sdc-sdc-cs-759f5d9d79-p27n6 1/1 Running 0 1h onap onap-sdc-sdc-cs-config-cassandra-7w2wf 0/1 Completed 0 1h onap onap-sdc-sdc-dcae-be-5b55b7dc9c-5nbdw 2/2 Running 0 1h onap onap-sdc-sdc-dcae-be-tools-c6dkf 0/1 Completed 0 50m onap onap-sdc-sdc-dcae-be-tools-frwhs 0/1 Init:Error 0 1h onap onap-sdc-sdc-dcae-dt-5cd48f7598-24w4g 2/2 Running 0 1h onap onap-sdc-sdc-dcae-fe-6c6f66664b-lbnsh 2/2 Running 0 1h onap onap-sdc-sdc-dcae-tosca-lab-7576c79d74-hv58w 2/2 Running 0 1h onap onap-sdc-sdc-es-8d55bd56-bxlr2 1/1 Running 0 1h onap onap-sdc-sdc-es-config-elasticsearch-88hdt 0/1 Completed 0 1h onap onap-sdc-sdc-fe-68c87b558-s887r 2/2 Running 0 1h onap onap-sdc-sdc-kb-6cb4d7d57d-dcfx6 1/1 Running 0 1h onap onap-sdc-sdc-onboarding-be-7876b44956-plbxv 2/2 Running 0 1h onap onap-sdc-sdc-onboarding-be-cassandra-init-dskg5 0/1 Completed 0 1h onap onap-sdc-sdc-wfd-be-85758b765-q26pb 1/1 Running 0 1h onap onap-sdc-sdc-wfd-be-workflow-init-sxq5b 0/1 Completed 0 1h onap onap-sdc-sdc-wfd-fe-7947c855d8-7dt22 2/2 Running 0 1h onap onap-sdnc-controller-blueprints-bc66576c5-kk68m 1/1 Running 1 40m onap onap-sdnc-controller-blueprints-db-0 1/1 Running 0 40m onap onap-sdnc-nengdb-0 1/1 Running 0 40m onap onap-sdnc-network-name-gen-7f95f5bfc8-l86ns 1/1 Running 0 40m onap onap-sdnc-sdnc-0 2/2 Running 0 40m onap onap-sdnc-sdnc-ansible-server-7d595dd8-lx2hz 1/1 Running 0 40m onap onap-sdnc-sdnc-db-0 2/2 Running 0 40m onap onap-sdnc-sdnc-dgbuilder-7c8fcbff77-hk7j2 1/1 Running 0 40m onap onap-sdnc-sdnc-dmaap-listener-fd8bf55cf-hjns6 1/1 Running 0 40m onap onap-sdnc-sdnc-portal-55976f4bd-vf62t 1/1 Running 2 40m onap onap-sdnc-sdnc-ueb-listener-7bf689f8b9-dr9xh 1/1 Running 0 40m amdocs@obriensystemsu1:~$ kubectl top nodes NAME CPU(cores) CPU% MEMORY(bytes) MEMORY% obriensystemsu1 2228m 18% 26002Mi 48% amdocs@obriensystemsu1:~$ free total used free shared buff/cache available Mem: 55339300 18195324 1908292 100980 35235684 36047660 Swap: 9289724 4236 9285488 amdocs@obriensystemsu1:~$ df Filesystem 1K-blocks Used Available Use% Mounted on udev 27647576 0 27647576 0% /dev tmpfs 5533932 32532 5501400 1% /run /dev/mapper/obriensystemsu0--vg-root 155367356 93401612 54050432 64% / tmpfs 27669648 23600 27646048 1% /dev/shm tmpfs 5120 4 5116 1% /run/lock tmpfs 27669648 0 27669648 0% /sys/fs/cgroup /dev/sda1 482922 62963 395025 14% /boot tmpfs 5533932 64 5533868 1% /run/user/1000 amdocs@obriensystemsu1:~$ sudo helm list [sudo] password for amdocs: NAME REVISION UPDATED STATUS CHART NAMESPACE onap 3 Sun Jan 13 19:25:55 2019 DEPLOYED onap-3.0.0 onap onap-consul 2 Sun Jan 13 19:25:57 2019 DEPLOYED consul-3.0.0 onap onap-dmaap 2 Sun Jan 13 19:25:58 2019 DEPLOYED dmaap-3.0.0 onap onap-sdc 2 Sun Jan 13 19:26:03 2019 DEPLOYED sdc-3.0.0 onap onap-sdnc 1 Sun Jan 13 19:26:04 2019 DEPLOYED sdnc-3.0.0 onap
Use my script and break up the deployment into dmaap, sdc and consul in dev0.yaml and sdnc in dev1.yaml – or use the -f option and put the 4 in sequence in the list https://gerrit.onap.org/r/#/c/75422/
Note: not all charts make their dependencies explicit – and not all REST/API and compile/runtime-reflection (pom.xml) dependencies are reflected in the current list. These are specified so far.
We will discuss this at the next PTL meet https://wiki.onap.org/display/DW/PTL+2019-01-14
Thank you /michae
From: onap-discuss@... <onap-discuss@...>
On Behalf Of Curtis
Sent: Friday, January 11, 2019 7:43 AM To: onap-discuss@...; Michael O'Brien <Frank.Obrien@...> Subject: Re: [onap-discuss] [oom][sdnc] Just deploy sdnc?
On Thu, Jan 10, 2019 at 4:09 PM Michael O'Brien <frank.obrien@...> wrote:
Hi :)
There's no specific reason, to be honest I'm just not sure what is a reasonable subset. :) Mostly I am exploring how to deploy pieces of ONAP using OOM instead of the entirety. sdnc seemed a good place to start, but it was a rather arbitrary decision on my part.
Thanks very much for the above information! I may have a few questions after I go through the what you have discussed, but this is extremely helpful and exactly the kind of information I need. :)
Thanks, Curtis
Blog: serverascode.com This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service 
|
|
|
|
Curtis <serverascode@...>
On Sun, Jan 13, 2019 at 10:27 PM Michael O'Brien <Frank.Obrien@...> wrote:
Thanks Michael, I will take a look in the next few days at what you have discussed here. Thanks!
-- Blog: serverascode.com |
|
|