Topics

Nexus3 proxy verified 80-100x faster downloads since 20181217 - I will provide a nexus3.onap.cloud Azure based proxy until Jan as workaround #lfn


Michael O'Brien <frank.obrien@...>
 

Team,

   The issue with nexus3.onap.info was likely a routing change Sunday.  For those requiring Casablanca 3.0.0-ONAP tagged images for new deployments I will have a 2nd proxy for public consumption up later today.  This will free up the LF to address the RC in the new year.

   Bottom line is a pull that takes 45 sec on the proxy takes 80-160 min from nexus3.onap.org only for this week – increasing pull times from 25 min to 35h – I retested an hour ago at 1730EST.

 

   Linux Foundation - I have your back - the quiet period started on the 17th for yourselves, for the rest of us on the 24th.  I prototyped a proxy on AWS nexus3.onap.info  - 80-100X faster image downloads.  I will put up a public one on azure that is on a larger VM for the rest of the community to use (will need a cert - I will add to the jira) - if you use nexus3.onap.cloud please leave a note on the jira

 

See details on the testing on clean VMs in and out of windriver against both nexus3.onap.org and nexus3.onap.info.

I will post details as soon as I get it up about the larger azure proxy - it will take at least 35 hours to saturate the nexus3 proxy with 40G of Casablanca images via the docker_preload.sh script  - I will do master later and advise when the proxy is ready.

 

Details on prepping you host VMs to use the nexus3 proxy temporarily.

https://wiki.onap.org/display/DW/Cloud+Native+Deployment#CloudNativeDeployment-NexusProxy

and

https://jira.onap.org/browse/TSC-79

 

# on each host

obrienbiometrics:nexus michaelobrien$ scp ubuntu@...:~/certs/domain.crt .

domain.crt                                                                                                                                                                             

obrienbiometrics:nexus michaelobrien$ scp domain.crt ubuntu@...:~/

domain.crt                                                                                                                                                                              100% 2114    15.7KB/s   00:00  

# to avoid

ubuntu@ip-172-31-33-46:~$ sudo docker login -u docker -p docker nexus3.onap.info:5000

Error response from daemon: Get https://nexus3.onap.info:5000/v1/users/: x509: certificate signed by unknown authority

# cp cert

ubuntu@ip-172-31-33-46:~$ sudo mkdir /etc/docker/certs.d

ubuntu@ip-172-31-33-46:~$ sudo mkdir /etc/docker/certs.d/nexus3.onap.info:5000

ubuntu@ip-172-31-33-46:~$ sudo cp domain.crt /etc/docker/certs.d/nexus3.onap.info:5000/ca.crt

ubuntu@ip-172-31-33-46:~$ sudo systemctl restart docker

ubuntu@ip-172-31-33-46:~$ sudo docker login -u docker -p docker nexus3.onap.info:5000

Login Succeeded

# testing

# vm with the image existing - 2 sec

ubuntu@ip-172-31-33-46:~$ sudo docker pull nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8

2.1.8: Pulling from onap/aaf/aaf_agent

Digest: sha256:71781f3cfa51066abb1a4a35267af37beec01b6bb75817fdfae056582839290c

Status: Downloaded newer image for nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8

 

# vm with layers existing except for last 5 - 5 sec

ubuntu@a-cd-master:~$ sudo docker pull nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8

2.1.8: Pulling from onap/aaf/aaf_agent

18d680d61657: Already exists

.. 20

49e90af50c7d: Already exists

396607166153: Pull complete

8023c56d0234: Pull complete

440441b9ca5e: Pull complete

4376cc11267c: Pull complete

acb05d09ff6e: Pull complete

Digest: sha256:71781f3cfa51066abb1a4a35267af37beec01b6bb75817fdfae056582839290c

Status: Downloaded newer image for nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8

 

# clean AWS VM (clean install of docker) - no pulls yet - 45 sec for everything

ubuntu@ip-172-31-14-34:~$ sudo docker pull nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8

2.1.8: Pulling from onap/aaf/aaf_agent

18d680d61657: Pulling fs layer

0addb6fece63: Pulling fs layer

78e58219b215: Pulling fs layer

eb6959a66df2: Pulling fs layer

321bd3fd2d0e: Pull complete

819d6de9e493: Pull complete

9c5cbae584e0: Pull complete

6b8abae643fc: Pull complete

595cdf95b083: Pull complete

d5f35f05364b: Pull complete

33422549438c: Pull complete

507242d06459: Pull complete

797c42cd7990: Pull complete

55cb180bd7a7: Pull complete

8f791662f04f: Pull complete

b21eb3946af1: Pull complete

36d0f286a7ec: Pull complete

2150f5ecf4e4: Pull complete

5b1a5109ab06: Pull complete

880ff0160341: Pull complete

14bbbdc46f89: Pull complete

03b5e4771470: Pull complete

fdd6eab46e6c: Pull complete

80ff62223f5f: Pull complete

e0633204c627: Pull complete

1b3a93e91ad9: Pull complete

29b72f79b735: Pull complete

a2cb53787c38: Pull complete

49e90af50c7d: Pull complete

396607166153: Pull complete

8023c56d0234: Pull complete

440441b9ca5e: Pull complete

4376cc11267c: Pull complete

acb05d09ff6e: Pull complete

Digest: sha256:71781f3cfa51066abb1a4a35267af37beec01b6bb75817fdfae056582839290c

Status: Downloaded newer image for nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8

ubuntu@ip-172-31-14-34:~$ sudo docker images

REPOSITORY                                 TAG                 IMAGE ID            CREATED             SIZE

nexus3.onap.info:5000/onap/aaf/aaf_agent   2.1.8               090b326a7f11        5 weeks ago         1.14 GB

 

# going to test a same size image directly from the LF - with minimal common layers

nexus3.onap.org:10001/onap/testsuite                    1.3.2                c4b58baa95e8        3 weeks ago         1.13 GB

# 5 min in we are still at 3% - numbers below are a min old

ubuntu@ip-172-31-14-34:~$ sudo docker pull nexus3.onap.org:10001/onap/testsuite:1.3.2

1.3.2: Pulling from onap/testsuite

32802c0cfa4d: Downloading [=============>                                     ] 8.416 MB/32.1 MB

da1315cffa03: Download complete

fa83472a3562: Download complete

f85999a86bef: Download complete

3eca7452fe93: Downloading [=======================>                           ] 8.517 MB/17.79 MB

9f002f13a564: Downloading [=========================================>         ] 8.528 MB/10.24 MB

02682cf43e5c: Waiting

dfa9878b26c8: Waiting

fdb3a2e7127c: Waiting

adab76b27695: Waiting

3e1db7307fd3: Waiting

d18a8cc3d4cd: Waiting

973f5d70b3c4: Waiting

eb19d05de017: Waiting

ac1f3947b000: Waiting

01ae31c1279e: Waiting

24ac662c0884: Waiting

268c36718aa7: Waiting

98559383536b: Waiting

856a1ffe236e: Waiting

80cbab20328c: Waiting

caed9c28b75f: Waiting

e3c00d7d86d4: Waiting

3109ae4e798e: Waiting

2dc71030fc1d: Waiting

f1d763192c3d: Waiting

adceeebb9b2b: Waiting

419a6a0c4acd: Waiting

048d8fd47ec5: Waiting

63fc3f31cfdc: Waiting

5441f94df8b4: Waiting

754645df4601: Waiting

# in 5 min we get 3% 35/1130Mb - which comes out to 162 min for 1.13G for .org as opposed to 45 sec for .info - which is a 200X slowdown - some of this is due to the fact my nexus3.onap.info is on the same VPC as my test VM - testing on openlab

 

# openlab - 2 min 40 sec which is 3.6 times slower - expected than in AWS - (25 min pulls vs 90min in openlab) - this makes nexus.onap.org 60 times slower in openlab than a proxy running from AWS (2 vCore/16G/ssd VM)

ubuntu@onap-oom-obrien-rancher-e4:~$ sudo docker pull nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8

2.1.8: Pulling from onap/aaf/aaf_agent

18d680d61657: Pull complete

...

acb05d09ff6e: Pull complete

Digest: sha256:71781f3cfa51066abb1a4a35267af37beec01b6bb75817fdfae056582839290c

Status: Downloaded newer image for nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8

 

 

#pulling smaller from nexus3.onap.info 2 min 20 - for 36Mb = 0.23Mb/sec - extrapolated to 1.13Gb for above is 5022 sec or 83 min - half the rough calculation above

ubuntu@onap-oom-obrien-rancher-e4:~$ sudo docker pull nexus3.onap.org:10001/onap/aaf/sms:3.0.1

3.0.1: Pulling from onap/aaf/sms

c67f3896b22c: Pull complete

134fac3103be: Pull complete

c7fdcaf7f439: Pull complete

590a963725eb: Pull complete

4df2c296b645: Pull complete

15dd1c6ebaf1: Pull complete

76eeb922b789: Pull complete

Digest: sha256:d5b64947edb93848acacaa9820234aa29e58217db9f878886b7bafae00fdb436

Status: Downloaded newer image for nexus3.onap.org:10001/onap/aaf/sms:3.0.1

 

# conclusion - nexus3.onap.org is experiencing a routing issue from their DC outbound causing a 80-100x slowdown over a proxy nexus3 - since 20181217 - as local jenkins.onap.org builds complete faster

# workaround is to use a nexus3 proxy above

 

 

and adding to values.yaml

 

global:

  #repository: nexus3.onap.org:10001

  repository: nexus3.onap.info:5000

  repositoryCred:

    user: docker

    password: docker

 

This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service


Keong Lim
 

So does this mean that the docker_prepull.sh script and the https://wiki.onap.org/display/DW/ONAP+OOM+Beijing+-+Hosting+docker+images+locally instructions should become part of the default setup for OOM?
e.g. in https://onap.readthedocs.io/en/latest/submodules/oom.git/docs/oom_quickstart_guide.html

It seems reasonable for each OOM deployment to also host its own proxy cache for scalability.


Lianhao Lu
 

Hi Brien,

 

Thanks for setting up the proxy. One question is that how to get the certificate of nexus3.onap.info? I tried “scp ubuntu@...:~/certs/domain.crt .”, but it failed with ssh public key issues. Without that certificate, I can not docker login.

 

-Lianhao

 

From: onap-tsc@... [mailto:onap-tsc@...] On Behalf Of Michael O'Brien
Sent: Wednesday, December 19, 2018 8:10 AM
To: onap-discuss@...; onap-tsc@...; helpdesk@...
Subject: [onap-tsc] Nexus3 proxy verified 80-100x faster downloads since 20181217 - I will provide a nexus3.onap.cloud Azure based proxy until Jan as workaround #nexus3

 

Team,

   The issue with nexus3.onap.info was likely a routing change Sunday.  For those requiring Casablanca 3.0.0-ONAP tagged images for new deployments I will have a 2nd proxy for public consumption up later today.  This will free up the LF to address the RC in the new year.

   Bottom line is a pull that takes 45 sec on the proxy takes 80-160 min from nexus3.onap.org only for this week – increasing pull times from 25 min to 35h – I retested an hour ago at 1730EST.

 

   Linux Foundation - I have your back - the quiet period started on the 17th for yourselves, for the rest of us on the 24th.  I prototyped a proxy on AWS nexus3.onap.info  - 80-100X faster image downloads.  I will put up a public one on azure that is on a larger VM for the rest of the community to use (will need a cert - I will add to the jira) - if you use nexus3.onap.cloud please leave a note on the jira

 

See details on the testing on clean VMs in and out of windriver against both nexus3.onap.org and nexus3.onap.info.

I will post details as soon as I get it up about the larger azure proxy - it will take at least 35 hours to saturate the nexus3 proxy with 40G of Casablanca images via the docker_preload.sh script  - I will do master later and advise when the proxy is ready.

 

Details on prepping you host VMs to use the nexus3 proxy temporarily.

https://wiki.onap.org/display/DW/Cloud+Native+Deployment#CloudNativeDeployment-NexusProxy

and

https://jira.onap.org/browse/TSC-79

 

# on each host

obrienbiometrics:nexus michaelobrien$ scp ubuntu@...:~/certs/domain.crt .

domain.crt                                                                                                                                                                             

obrienbiometrics:nexus michaelobrien$ scp domain.crt ubuntu@...:~/

domain.crt                                                                                                                                                                              100% 2114    15.7KB/s   00:00  

# to avoid

ubuntu@ip-172-31-33-46:~$ sudo docker login -u docker -p docker nexus3.onap.info:5000

Error response from daemon: Get https://nexus3.onap.info:5000/v1/users/: x509: certificate signed by unknown authority

# cp cert

ubuntu@ip-172-31-33-46:~$ sudo mkdir /etc/docker/certs.d

ubuntu@ip-172-31-33-46:~$ sudo mkdir /etc/docker/certs.d/nexus3.onap.info:5000

ubuntu@ip-172-31-33-46:~$ sudo cp domain.crt /etc/docker/certs.d/nexus3.onap.info:5000/ca.crt

ubuntu@ip-172-31-33-46:~$ sudo systemctl restart docker

ubuntu@ip-172-31-33-46:~$ sudo docker login -u docker -p docker nexus3.onap.info:5000

Login Succeeded

 

# testing

# vm with the image existing - 2 sec

ubuntu@ip-172-31-33-46:~$ sudo docker pull nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8

2.1.8: Pulling from onap/aaf/aaf_agent

Digest: sha256:71781f3cfa51066abb1a4a35267af37beec01b6bb75817fdfae056582839290c

Status: Downloaded newer image for nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8

 

 

# vm with layers existing except for last 5 - 5 sec

ubuntu@a-cd-master:~$ sudo docker pull nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8

2.1.8: Pulling from onap/aaf/aaf_agent

18d680d61657: Already exists

.. 20

49e90af50c7d: Already exists

396607166153: Pull complete

8023c56d0234: Pull complete

440441b9ca5e: Pull complete

4376cc11267c: Pull complete

acb05d09ff6e: Pull complete

Digest: sha256:71781f3cfa51066abb1a4a35267af37beec01b6bb75817fdfae056582839290c

Status: Downloaded newer image for nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8

 

 

# clean AWS VM (clean install of docker) - no pulls yet - 45 sec for everything

ubuntu@ip-172-31-14-34:~$ sudo docker pull nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8

2.1.8: Pulling from onap/aaf/aaf_agent

18d680d61657: Pulling fs layer

0addb6fece63: Pulling fs layer

78e58219b215: Pulling fs layer

eb6959a66df2: Pulling fs layer

321bd3fd2d0e: Pull complete

819d6de9e493: Pull complete

9c5cbae584e0: Pull complete

6b8abae643fc: Pull complete

595cdf95b083: Pull complete

d5f35f05364b: Pull complete

33422549438c: Pull complete

507242d06459: Pull complete

797c42cd7990: Pull complete

55cb180bd7a7: Pull complete

8f791662f04f: Pull complete

b21eb3946af1: Pull complete

36d0f286a7ec: Pull complete

2150f5ecf4e4: Pull complete

5b1a5109ab06: Pull complete

880ff0160341: Pull complete

14bbbdc46f89: Pull complete

03b5e4771470: Pull complete

fdd6eab46e6c: Pull complete

80ff62223f5f: Pull complete

e0633204c627: Pull complete

1b3a93e91ad9: Pull complete

29b72f79b735: Pull complete

a2cb53787c38: Pull complete

49e90af50c7d: Pull complete

396607166153: Pull complete

8023c56d0234: Pull complete

440441b9ca5e: Pull complete

4376cc11267c: Pull complete

acb05d09ff6e: Pull complete

Digest: sha256:71781f3cfa51066abb1a4a35267af37beec01b6bb75817fdfae056582839290c

Status: Downloaded newer image for nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8

ubuntu@ip-172-31-14-34:~$ sudo docker images

REPOSITORY                                 TAG                 IMAGE ID            CREATED             SIZE

nexus3.onap.info:5000/onap/aaf/aaf_agent   2.1.8               090b326a7f11        5 weeks ago         1.14 GB

 

 

# going to test a same size image directly from the LF - with minimal common layers

nexus3.onap.org:10001/onap/testsuite                    1.3.2                c4b58baa95e8        3 weeks ago         1.13 GB

# 5 min in we are still at 3% - numbers below are a min old

ubuntu@ip-172-31-14-34:~$ sudo docker pull nexus3.onap.org:10001/onap/testsuite:1.3.2

1.3.2: Pulling from onap/testsuite

32802c0cfa4d: Downloading [=============>                                     ] 8.416 MB/32.1 MB

da1315cffa03: Download complete

fa83472a3562: Download complete

f85999a86bef: Download complete

3eca7452fe93: Downloading [=======================>                           ] 8.517 MB/17.79 MB

9f002f13a564: Downloading [=========================================>         ] 8.528 MB/10.24 MB

02682cf43e5c: Waiting

dfa9878b26c8: Waiting

fdb3a2e7127c: Waiting

adab76b27695: Waiting

3e1db7307fd3: Waiting

d18a8cc3d4cd: Waiting

973f5d70b3c4: Waiting

eb19d05de017: Waiting

ac1f3947b000: Waiting

01ae31c1279e: Waiting

24ac662c0884: Waiting

268c36718aa7: Waiting

98559383536b: Waiting

856a1ffe236e: Waiting

80cbab20328c: Waiting

caed9c28b75f: Waiting

e3c00d7d86d4: Waiting

3109ae4e798e: Waiting

2dc71030fc1d: Waiting

f1d763192c3d: Waiting

adceeebb9b2b: Waiting

419a6a0c4acd: Waiting

048d8fd47ec5: Waiting

63fc3f31cfdc: Waiting

5441f94df8b4: Waiting

754645df4601: Waiting

 

# in 5 min we get 3% 35/1130Mb - which comes out to 162 min for 1.13G for .org as opposed to 45 sec for .info - which is a 200X slowdown - some of this is due to the fact my nexus3.onap.info is on the same VPC as my test VM - testing on openlab

 

 

# openlab - 2 min 40 sec which is 3.6 times slower - expected than in AWS - (25 min pulls vs 90min in openlab) - this makes nexus.onap.org 60 times slower in openlab than a proxy running from AWS (2 vCore/16G/ssd VM)

ubuntu@onap-oom-obrien-rancher-e4:~$ sudo docker pull nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8

2.1.8: Pulling from onap/aaf/aaf_agent

 

18d680d61657: Pull complete

...

acb05d09ff6e: Pull complete

Digest: sha256:71781f3cfa51066abb1a4a35267af37beec01b6bb75817fdfae056582839290c

Status: Downloaded newer image for nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8

 

 

 

#pulling smaller from nexus3.onap.info 2 min 20 - for 36Mb = 0.23Mb/sec - extrapolated to 1.13Gb for above is 5022 sec or 83 min - half the rough calculation above

ubuntu@onap-oom-obrien-rancher-e4:~$ sudo docker pull nexus3.onap.org:10001/onap/aaf/sms:3.0.1

3.0.1: Pulling from onap/aaf/sms

c67f3896b22c: Pull complete

134fac3103be: Pull complete

c7fdcaf7f439: Pull complete

590a963725eb: Pull complete

4df2c296b645: Pull complete

15dd1c6ebaf1: Pull complete

76eeb922b789: Pull complete

Digest: sha256:d5b64947edb93848acacaa9820234aa29e58217db9f878886b7bafae00fdb436

Status: Downloaded newer image for nexus3.onap.org:10001/onap/aaf/sms:3.0.1

 

 

# conclusion - nexus3.onap.org is experiencing a routing issue from their DC outbound causing a 80-100x slowdown over a proxy nexus3 - since 20181217 - as local jenkins.onap.org builds complete faster

# workaround is to use a nexus3 proxy above

 

 

and adding to values.yaml

 

global:

  #repository: nexus3.onap.org:10001

  repository: nexus3.onap.info:5000

  repositoryCred:

    user: docker

    password: docker

 

This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service