Topics

[oom] 110 kubernetes pod limit increase for single node dev deployment testing - procedure

Michael O'Brien <frank.obrien@...>
 

Team,

              There is a workaround for the default (rancher template) –max-pods=110 limit per node – I would set it to over 200 – I left this on the side since June because I have AWS clusters – but I think it would be good to have a single node deployment for dev testing – with the workaround you can bring up all of ONAP within 115G – with a set of dev profiles – keyed to use cases like the vFW we should be able to get back to 64g versions – without having an NFS enabled cluster – something that can fit on a Thinkpad P52.

 

Procedure and video here

https://wiki.onap.org/display/DW/Cloud+Native+Deployment#CloudNativeDeployment-Changemax-podsfromdefault110podlimit

              I put the procedure and a 1m video above – the rest call to do this in line 111 of https://git.onap.org/logging-analytics/tree/deploy/rancher/oom_rancher_setup.sh#n111 is being tested – but for now you can edit the kubernetes template during the wait time before the environment instances is created from it

Manual procedure: change the kubernetes template (1pt2) before using it to create an environment (1a7)

add --max-pods=500 to the "Additional Kubelet Flags" box on the v1.10.13 version of the kubernetes template from the "Manage Environments" dropdown on the left of the 8880 rancher console.

 

Results: using a WIP single node procedure

https://wiki.onap.org/display/DW/Cloud+Native+Deployment#CloudNativeDeployment-Scriptedundercloud(Helm/Kubernetes/Docker)andONAPinstall-SingleVM

We run around 164 persistent pods currently – it takes about an hour to get a clean system up (including 20+ min docker pulls directly nexus3)

Single AWS 244G 32vCore VM with 110 pod limit workaround - 164 pods (including both secondary DCAEGEN2 orchestrations at 30 and 55 min) - most of the remaining 8 container failures are known/in-progress issues.

 

ubuntu@ip-172-31-20-218:~$ free

              total        used        free      shared  buff/cache   available

Mem:      251754696   111586672    45000724      193628    95167300   137158588

ubuntu@ip-172-31-20-218:~$ kubectl get pods --all-namespaces | grep onap | wc -l

164

ubuntu@ip-172-31-20-218:~$ kubectl get pods --all-namespaces | grep onap | grep -E '1/1|2/2' | wc -l

155

ubuntu@ip-172-31-20-218:~$ kubectl get pods --all-namespaces | grep -E '0/|1/2' | wc -l

8

ubuntu@ip-172-31-20-218:~$ kubectl get pods --all-namespaces | grep -E '0/|1/2'

onap          dep-dcae-ves-collector-59d4ff58f7-94rpq                 1/2       Running                 0          4m

onap          onap-aai-champ-68ff644d85-rv7tr                         0/1       Running                 0          59m

onap          onap-aai-gizmo-856f86d664-q5pvg                         1/2       CrashLoopBackOff        10         59m

onap          onap-oof-85864d6586-zcsz5                               0/1       ImagePullBackOff        0          59m

onap          onap-pomba-kibana-d76b6dd4c-sfbl6                       0/1       Init:CrashLoopBackOff   8          59m

onap          onap-pomba-networkdiscovery-85d76975b7-mfk92            1/2       CrashLoopBackOff        11         59m

onap          onap-pomba-networkdiscoveryctxbuilder-c89786dfc-qnlx9   1/2       CrashLoopBackOff        10         59m

onap          onap-vid-84c88db589-8cpgr   

 

note: the dcae ves collector comes up 2/2 around 65 min

/michael

This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement,

Eric Debeau
 

Hello

We need to work on a smaller footprint for testing purposes.

I agree that using a single VM can facilitate the work, but we also need to reduce the memory footprint.
Most of pods are consuming a lot of memory not necessary for simple tests.

I am currently working on a solution to reduce the global footprint with a target to lower the memory footprint for the vFW use-case as low as possible.

Best Regards

Eric

Mike Elliott
 

Hi Eric,

 

Are you familiar with the resource limits that are being added to the helm charts (for all projects) under OOM-1145 ?

One of the benefits of this work is to set the minimum and maximum limits for memory used by each onap component (pod).

Are you working on something similar or do you have any insight that you can share with the teams that might help us get the best results out of this effort?

 

As a side note, one of the largest reductions in memory usage will most likely come as part of Dublin, in the form of a shared database cluster. This would be instead of each project spinning up their own cluster instances in order to meet S3P requirements.

 

Thanks,

Mike.

 

From: <onap-discuss@...> on behalf of Eric Debeau <eric.debeau@...>
Reply-To: "onap-discuss@..." <onap-discuss@...>, "eric.debeau@..." <eric.debeau@...>
Date: Wednesday, September 5, 2018 at 4:13 PM
To: "onap-discuss@..." <onap-discuss@...>
Subject: Re: [onap-discuss] [oom] 110 kubernetes pod limit increase for single node dev deployment testing - procedure

 

Hello

We need to work on a smaller footprint for testing purposes.

I agree that using a single VM can facilitate the work, but we also need to reduce the memory footprint.
Most of pods are consuming a lot of memory not necessary for simple tests.

I am currently working on a solution to reduce the global footprint with a target to lower the memory footprint for the vFW use-case as low as possible.

Best Regards

Eric

This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement,

Eric Debeau
 

Hi Mike

 

Yes, I am aware about this work ;-)

 

But, it would not reduce the memory to be used as long as most of Docker are launched with JVM using ‘big’ heap size.

I am working to reduce the heap size for various components. I hope that I can share valuable results in the next weeks.

 

I agree that we would get good results by optimizing data bases (using various Cassandra databases is not optimized).

 

Best Regards

 

Eric

 

De : Mike Elliott [mailto:Mike.Elliott@...]
Envoyé : jeudi 6 septembre 2018 19:25
À : onap-discuss@...; DEBEAU Eric IMT/OLN
Objet : Re: [onap-discuss] [oom] 110 kubernetes pod limit increase for single node dev deployment testing - procedure

 

Hi Eric,

 

Are you familiar with the resource limits that are being added to the helm charts (for all projects) under OOM-1145 ?

One of the benefits of this work is to set the minimum and maximum limits for memory used by each onap component (pod).

Are you working on something similar or do you have any insight that you can share with the teams that might help us get the best results out of this effort?

 

As a side note, one of the largest reductions in memory usage will most likely come as part of Dublin, in the form of a shared database cluster. This would be instead of each project spinning up their own cluster instances in order to meet S3P requirements.

 

Thanks,

Mike.

 

From: <onap-discuss@...> on behalf of Eric Debeau <eric.debeau@...>
Reply-To: "onap-discuss@..." <onap-discuss@...>, "eric.debeau@..." <eric.debeau@...>
Date: Wednesday, September 5, 2018 at 4:13 PM
To: "onap-discuss@..." <onap-discuss@...>
Subject: Re: [onap-discuss] [oom] 110 kubernetes pod limit increase for single node dev deployment testing - procedure

 

Hello

We need to work on a smaller footprint for testing purposes.

I agree that using a single VM can facilitate the work, but we also need to reduce the memory footprint.
Most of pods are consuming a lot of memory not necessary for simple tests.

I am currently working on a solution to reduce the global footprint with a target to lower the memory footprint for the vFW use-case as low as possible.

Best Regards

Eric

This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement,

_________________________________________________________________________________________________________________________

Ce message et ses pieces jointes peuvent contenir des informations confidentielles ou privilegiees et ne doivent donc
pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu ce message par erreur, veuillez le signaler
a l'expediteur et le detruire ainsi que les pieces jointes. Les messages electroniques etant susceptibles d'alteration,
Orange decline toute responsabilite si ce message a ete altere, deforme ou falsifie. Merci.

This message and its attachments may contain confidential or privileged information that may be protected by law;
they should not be distributed, used or copied without authorisation.
If you have received this email in error, please notify the sender and delete this message and its attachments.
As emails may be altered, Orange is not liable for messages that have been modified, changed or falsified.
Thank you.