Team,

   Passed to LF.

Posted my latest test to the end of OOM-933

https://jira.onap.org/browse/OOM-933

Quoting here

Fast overall but issues with pauses still.

NExus3 jvm retrofit of heap from 1.2 to 6g testing on 128g 32vCore azure vm docker 17.03 10G network

Dont think it is the JVM unless I see the heap stats - and the cpu spike on GC collection
check the thread pool size and the open file setting (Gary) to start

25 min download of 90 containers is normal - but with pausing behavior (single vm not 2 like last time)
still seeing slowdown 5 min in - with bursts - but no full pause for 30-60 like before
around 12 min seeing 5 second thread blocks ever 15 sec when 50 of 90 left

    1  sudo curl https://releases.rancher.com/install-docker/17.03.sh | sh

    2  sudo usermod -aG docker ubuntu

./prepull_docker_master.sh

times are 2 min apart - numbers are remaining docker image pulls

Last login: Fri May  4 19:22:04 2018 from 173.32.179.153

ubuntu@a-onap-auto-20180503-0:~$ sudo ps -ef | grep docker | grep pull | wc -l

79

ubuntu@a-onap-auto-20180503-0:~$ 1522

ubuntu@a-onap-auto-20180503-0:~$ sudo ps -ef | grep docker | grep pull | wc -l

77

ubuntu@a-onap-auto-20180503-0:~$ 1524

ubuntu@a-onap-auto-20180503-0:~$ sudo ps -ef | grep docker | grep pull | wc -l

73

ubuntu@a-onap-auto-20180503-0:~$ 1526

ubuntu@a-onap-auto-20180503-0:~$ sudo ps -ef | grep docker | grep pull | wc -l

71

ubuntu@a-onap-auto-20180503-0:~$ 1528

ubuntu@a-onap-auto-20180503-0:~$ sudo ps -ef | grep docker | grep pull | wc -l

66

ubuntu@a-onap-auto-20180503-0:~$ 1530

ubuntu@a-onap-auto-20180503-0:~$ sudo ps -ef | grep docker | grep pull | wc -l

59

ubuntu@a-onap-auto-20180503-0:~$ 1532

ee5c7445b360: Downloading [========================>                          ] 161.7 MB/329.7 MB

ee5c7445b360: Downloading [========================>                          ] 164.4 MB/329.7 MB

ee5c7445b360: Downloading [=========================>                         ] 168.1 MB/329.7 MB

ee5c7445b360: Downloading [=============================>                     ] 197.4 MB/329.7 MB

above 10 sec apart 

ubuntu@a-onap-auto-20180503-0:~$ sudo ps -ef | grep docker | grep pull | wc -l

50

ubuntu@a-onap-auto-20180503-0:~$ 1534

ubuntu@a-onap-auto-20180503-0:~$ sudo ps -ef | grep docker | grep pull | wc -l

40

ubuntu@a-onap-auto-20180503-0:~$ 1536

15 sec thread blocks - every 20 sec

speed back up now

ubuntu@a-onap-auto-20180503-0:~$ sudo ps -ef | grep docker | grep pull | wc -l

37

ubuntu@a-onap-auto-20180503-0:~$ 1538

ubuntu@a-onap-auto-20180503-0:~$ sudo ps -ef | grep docker | grep pull | wc -l

32

ubuntu@a-onap-auto-20180503-0:~$ 1540

ubuntu@a-onap-auto-20180503-0:~$ sudo ps -ef | grep docker | grep pull | wc -l

23

ubuntu@a-onap-auto-20180503-0:~$ 1542

ubuntu@a-onap-auto-20180503-0:~$ sudo ps -ef | grep docker | grep pull | wc -l

17

ubuntu@a-onap-auto-20180503-0:~$ 1544

current speed

d652b8d9672e: Downloading [=================>                                 ] 204.9 MB/577 MB

d652b8d9672e: Downloading [=========================>                         ] 293.9 MB/577 MB

d652b8d9672e: Downloading [=================================>                 ] 389.2 MB/577 MB

d652b8d9672e: Downloading [=====================================>             ] 433.9 MB/577 MB

d652b8d9672e: Downloading [===========================================>       ] 502.4 MB/577 MB

above 10 sec apart

ubuntu@a-onap-auto-20180503-0:~$ 1544

ubuntu@a-onap-auto-20180503-0:~$ sudo ps -ef | grep docker | grep pull | wc -l

7

ubuntu@a-onap-auto-20180503-0:~$ 1546

ubuntu@a-onap-auto-20180503-0:~$ sudo ps -ef | grep docker | grep pull | wc -l

2

ubuntu@a-onap-auto-20180503-0:~$ 1547

ubuntu@a-onap-auto-20180503-0:~$ sudo ps -ef | grep docker | grep pull | wc -l

1

ubuntu@a-onap-auto-20180503-0:~$ 1548

ubuntu@a-onap-auto-20180503-0:~$ sudo ps -ef | grep docker | grep pull | wc -l

0

/michael