LF,

   Hi, we have been seeing burst behavior for the last week. This morning in the integration lab session we see this on multiple machines.

   Downloads are fast when they are running – but completely stall every min or so for 30-60 sec.

   For example pulling the dmaap container manually – a single pull took 3 min on one system – 18 on antoher and had sections where the job ran about 1 to 5 with bursts to 25 Mb/sec (max is usually 10Mb/sec per stream – up to 5 in parallel – I have seen 25) – but there were times 30-60 sec where the image download completely paused at 0Mb/sec.

   This behavior is different or is in combination with the traditional slowdown we used to see  - very slow pulls – this issue is new in that it works fine during the burst – but has sections where all pulls pause.  To be frank – I did see this pause behavior briefly for a couple seconds back even in Dec.

   I propose we meet again to test downloads again and get some volumetrics.

   Some details on this and previous issues in

https://jira.onap.org/browse/OOM-933

Time 1130 EST 20180430

see slowdown again after our last meeting - a burst behaviour (burst/hold in 30-60 sec intervals)

This in an azure system (clean) running a manual docker download - see attachment on OOM-716

root@obrienondemand20180430:/var/lib/waagent/custom-script/download/0# history

    1  cd /var/lib/waagent/custom-script/download/0/

    2  ls

    3  tail -f stdout

    4  wget https://jira.onap.org/secure/attachment/11444/prepull_docker_master.sh

    6  sudo chmod 777 prepull_docker_master.sh 

    7  vi prepull_docker_master.sh 

    8  docker login -u docker -p docker nexus3.onap.org:10001

    9  docker pull nexus3.onap.org:10001/onap/dmaap/dmaap-mr:1.1.4

   10  vi prepull_docker_master.sh 

   11  docker pull nexus3.onap.org:10001/onap/vid:1.2.1

   12  docker pull nexus3.onap.org:10001/onap/dmaap/dmaap-mr:1.1.4

root@obrienondemand20180430:/var/lib/waagent/custom-script/download/0# docker pull nexus3.onap.org:10001/onap/dmaap/dmaap-mr:1.1.4

1.1.4: Pulling from onap/dmaap/dmaap-mr

0c5a750448f5: Pull complete 

e2356a247318: Pull complete 

12f105c650c3: Pull complete 

46d4cec7ab72: Pull complete 

f5b0369aa7fc: Pull complete 

a93519a09c8b: Pull complete 

73ffcbe01622: Waiting 

071f37eca411: Waiting 

dc33933d1e9b: Waiting 

cc3a1fce1c35: Waiting 

ab292f5d0d16: Waiting 

9cd00aac030f: Waiting 

3 min

root@obrienondemand20180430:/var/lib/waagent/custom-script/download/0# docker pull nexus3.onap.org:10001/onap/dmaap/dmaap-mr:1.1.4

1.1.4: Pulling from onap/dmaap/dmaap-mr

0c5a750448f5: Pull complete 

e2356a247318: Pull complete 

12f105c650c3: Pull complete 

46d4cec7ab72: Pull complete 

f5b0369aa7fc: Pull complete 

a93519a09c8b: Pull complete 

73ffcbe01622: Pull complete 

071f37eca411: Pull complete 

dc33933d1e9b: Pull complete 

cc3a1fce1c35: Pull complete 

ab292f5d0d16: Pull complete 

9cd00aac030f: Pull complete 

Digest: sha256:8dff567ab978965f545fc1cadc8060cd34a2599ba586e786795469b32fc670ff

Status: Downloaded newer image for nexus3.onap.org:10001/onap/dmaap/dmaap-mr:1.1.4

testing full predeploy - started 1148

1149

ubuntu@obrienondemand20180430:~$ sudo ps -ef | grep docker | grep pull | wc -l

77

10 sec below

32eda82f66ec: Downloading [================>                                  ] 96.86 MB/298.4 MB

32eda82f66ec: Downloading [====================>                              ]   124 MB/298.4 MB

1153

ubuntu@obrienondemand20180430:~$ sudo ps -ef | grep docker | grep pull | wc -l

75