LF,
Hi, we have been seeing burst behavior for the last week. This
morning in the integration lab session we see this on multiple
machines.
Downloads are fast when they are running - but completely stall
every min or so for 30-60 sec.

For example pulling the dmaap container manually - a single pull took
3 min on one system - 18 on antoher and had sections where the job ran
about 1 to 5 with bursts to 25 Mb/sec (max is usually 10Mb/sec per
stream - up to 5 in parallel - I have seen 25) - but there were times
30-60 sec where the image download completely paused at 0Mb/sec.

This behavior is different or is in combination with the traditional
slowdown we used to see - very slow pulls - this issue is new in that
it works fine during the burst - but has sections where all pulls
pause. To be frank - I did see this pause behavior briefly for a
couple seconds back even in Dec.


I propose we meet again to test downloads again and get some
volumetrics.
Some details on this and previous issues in

https://jira.onap.org/browse/OOM-933

Time 1130 EST 20180430

see slowdown again after our last meeting - a burst behaviour
(burst/hold in 30-60 sec intervals)

This in an azure system (clean) running a manual docker download - see
attachment on OOM-716<https://jira.onap.org/browse/OOM-716>

root@obrienondemand20180430:/var/lib/waagent/custom-script/download/0#
history

1 cd /var/lib/waagent/custom-script/download/0/

2 ls

3 tail -f stdout

4 wget
https://jira.onap.org/secure/attachment/11444/prepull_docker_master.sh

6 sudo chmod 777 prepull_docker_master.sh

7 vi prepull_docker_master.sh

8 docker login -u docker -p docker nexus3.onap.org:10001

9 docker pull nexus3.onap.org:10001/onap/dmaap/dmaap-mr:1.1.4

10 vi prepull_docker_master.sh

11 docker pull nexus3.onap.org:10001/onap/vid:1.2.1

12 docker pull nexus3.onap.org:10001/onap/dmaap/dmaap-mr:1.1.4





root@obrienondemand20180430:/var/lib/waagent/custom-script/download/0#
docker pull nexus3.onap.org:10001/onap/dmaap/dmaap-mr:1.1.4

1.1.4: Pulling from onap/dmaap/dmaap-mr

0c5a750448f5: Pull complete

e2356a247318: Pull complete

12f105c650c3: Pull complete

46d4cec7ab72: Pull complete

f5b0369aa7fc: Pull complete

a93519a09c8b: Pull complete

73ffcbe01622: Waiting

071f37eca411: Waiting

dc33933d1e9b: Waiting

cc3a1fce1c35: Waiting

ab292f5d0d16: Waiting

9cd00aac030f: Waiting





3 min

root@obrienondemand20180430:/var/lib/waagent/custom-script/download/0#
docker pull nexus3.onap.org:10001/onap/dmaap/dmaap-mr:1.1.4

1.1.4: Pulling from onap/dmaap/dmaap-mr

0c5a750448f5: Pull complete

e2356a247318: Pull complete

12f105c650c3: Pull complete

46d4cec7ab72: Pull complete

f5b0369aa7fc: Pull complete

a93519a09c8b: Pull complete

73ffcbe01622: Pull complete

071f37eca411: Pull complete

dc33933d1e9b: Pull complete

cc3a1fce1c35: Pull complete

ab292f5d0d16: Pull complete

9cd00aac030f: Pull complete

Digest:
sha256:8dff567ab978965f545fc1cadc8060cd34a2599ba586e786795469b32fc670ff

Status: Downloaded newer image for
nexus3.onap.org:10001/onap/dmaap/dmaap-mr:1.1.4

testing full predeploy - started 1148

1149

ubuntu@obrienondemand20180430:~$ sudo ps -ef | grep docker | grep pull
| wc -l

77

10 sec below

32eda82f66ec: Downloading [================>
] 96.86 MB/298.4 MB

32eda82f66ec: Downloading [====================>
] 124 MB/298.4 MB

1153

ubuntu@obrienondemand20180430:~$ sudo ps -ef | grep docker | grep pull
| wc -l

75

This message and the information contained herein is proprietary and
confidential and subject to the Amdocs policy statement,

you may review at https://www.amdocs.com/about/email-disclaimer
<https://www.amdocs.com/about/email-disclaimer>