Re: [ONAP Helpdesk #65866] [linuxfoundation.org #65866] RE: Nexus3 proxy verified 80-100x faster downloads since 20181217 - I will provide a nexus3.onap.cloud Azure based proxy until Jan as workaround #lfn


Michael O'Brien <frank.obrien@...>
 

Team,
Nexus3.onap.info is now at the ccsdk's
Nexus3.onap.cloud is at the end of the aai’s – forgot to post (was asked) where the cert is – added to
https://jira.onap.org/browse/TSC-79
https://jira.onap.org/secure/attachment/13025/domain_nexus3_onap_cloud.crt
or
https://jira.onap.org/browse/LOG-905
https://jira.onap.org/secure/attachment/13020/domain_nexus3_onap_cloud.crt
Note: docker pulls for images not pulled from nexus3.onap.org will still be slow until they are in the cache – eta is mid Friday – I will advise when fully cached.
Remember to rename it via
https://wiki.onap.org/display/DW/Cloud+Native+Deployment#CloudNativeDeployment-NexusProxy
Thank you
/michael

-----Original Message-----
From: lianhao.lu@intel.com via RT <onap-helpdesk@rt.linuxfoundation.org>
Sent: Wednesday, December 19, 2018 4:20 AM
To: Michael O'Brien <Frank.Obrien@amdocs.com>
Cc: onap-discuss@lists.onap.org; onap-tsc@lists.onap.org
Subject: [ONAP Helpdesk #65866] [linuxfoundation.org #65866] RE: Nexus3 proxy verified 80-100x faster downloads since 20181217 - I will provide a nexus3.onap.cloud Azure based proxy until Jan as workaround #nexus3

Hi Brien,

Thanks for setting up the proxy. One question is that how to get the certificate of nexus3.onap.info? I tried "scp ubuntu@nexus3.onap.info:~/certs/domain.crt<mailto:ubuntu@nexus3.onap.info:~/certs/domain.crt> .", but it failed with ssh public key issues. Without that certificate, I can not docker login.

-Lianhao

From: onap-tsc@lists.onap.org [mailto:onap-tsc@lists.onap.org] On Behalf Of Michael O'Brien
Sent: Wednesday, December 19, 2018 8:10 AM
To: onap-discuss@lists.onap.org; onap-tsc@lists.onap.org; helpdesk@onap.org
Subject: [onap-tsc] Nexus3 proxy verified 80-100x faster downloads since 20181217 - I will provide a nexus3.onap.cloud Azure based proxy until Jan as workaround #nexus3


Team,

The issue with nexus3.onap.info was likely a routing change Sunday. For those requiring Casablanca 3.0.0-ONAP tagged images for new deployments I will have a 2nd proxy for public consumption up later today. This will free up the LF to address the RC in the new year.

Bottom line is a pull that takes 45 sec on the proxy takes 80-160 min from nexus3.onap.org only for this week - increasing pull times from 25 min to 35h - I retested an hour ago at 1730EST.



Linux Foundation - I have your back - the quiet period started on the 17th for yourselves, for the rest of us on the 24th. I prototyped a proxy on AWS nexus3.onap.info - 80-100X faster image downloads. I will put up a public one on azure that is on a larger VM for the rest of the community to use (will need a cert - I will add to the jira) - if you use nexus3.onap.cloud please leave a note on the jira



See details on the testing on clean VMs in and out of windriver against both nexus3.onap.org and nexus3.onap.info.

I will post details as soon as I get it up about the larger azure proxy - it will take at least 35 hours to saturate the nexus3 proxy with 40G of Casablanca images via the docker_preload.sh script - I will do master later and advise when the proxy is ready.



Details on prepping you host VMs to use the nexus3 proxy temporarily.

https://wiki.onap.org/display/DW/Cloud+Native+Deployment#CloudNativeDeployment-NexusProxy

and

https://jira.onap.org/browse/TSC-79



# on each host

obrienbiometrics:nexus michaelobrien$ scp ubuntu@nexus3.onap.info:~/certs/domain.crt<mailto:ubuntu@nexus3.onap.info:~/certs/domain.crt> .

domain.crt

obrienbiometrics:nexus michaelobrien$ scp domain.crt ubuntu@ld3.onap.info:~/<mailto:ubuntu@ld3.onap.info:~/>

domain.crt 100% 2114 15.7KB/s 00:00

# to avoid

ubuntu@ip-172-31-33-46:~$ sudo docker login -u docker -p docker nexus3.onap.info:5000

Error response from daemon: Get https://nexus3.onap.info:5000/v1/users/: x509: certificate signed by unknown authority

# cp cert

ubuntu@ip-172-31-33-46:~$ sudo mkdir /etc/docker/certs.d

ubuntu@ip-172-31-33-46:~$ sudo mkdir /etc/docker/certs.d/nexus3.onap.info:5000

ubuntu@ip-172-31-33-46:~$ sudo cp domain.crt /etc/docker/certs.d/nexus3.onap.info:5000/ca.crt

ubuntu@ip-172-31-33-46:~$ sudo systemctl restart docker

ubuntu@ip-172-31-33-46:~$ sudo docker login -u docker -p docker nexus3.onap.info:5000

Login Succeeded



# testing

# vm with the image existing - 2 sec

ubuntu@ip-172-31-33-46:~$ sudo docker pull nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8

2.1.8: Pulling from onap/aaf/aaf_agent

Digest: sha256:71781f3cfa51066abb1a4a35267af37beec01b6bb75817fdfae056582839290c

Status: Downloaded newer image for nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8





# vm with layers existing except for last 5 - 5 sec

ubuntu@a-cd-master:~$ sudo docker pull nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8

2.1.8: Pulling from onap/aaf/aaf_agent

18d680d61657: Already exists

.. 20

49e90af50c7d: Already exists

396607166153: Pull complete

8023c56d0234: Pull complete

440441b9ca5e: Pull complete

4376cc11267c: Pull complete

acb05d09ff6e: Pull complete

Digest: sha256:71781f3cfa51066abb1a4a35267af37beec01b6bb75817fdfae056582839290c

Status: Downloaded newer image for nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8





# clean AWS VM (clean install of docker) - no pulls yet - 45 sec for everything

ubuntu@ip-172-31-14-34:~$ sudo docker pull nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8

2.1.8: Pulling from onap/aaf/aaf_agent

18d680d61657: Pulling fs layer

0addb6fece63: Pulling fs layer

78e58219b215: Pulling fs layer

eb6959a66df2: Pulling fs layer

321bd3fd2d0e: Pull complete

819d6de9e493: Pull complete

9c5cbae584e0: Pull complete

6b8abae643fc: Pull complete

595cdf95b083: Pull complete

d5f35f05364b: Pull complete

33422549438c: Pull complete

507242d06459: Pull complete

797c42cd7990: Pull complete

55cb180bd7a7: Pull complete

8f791662f04f: Pull complete

b21eb3946af1: Pull complete

36d0f286a7ec: Pull complete

2150f5ecf4e4: Pull complete

5b1a5109ab06: Pull complete

880ff0160341: Pull complete

14bbbdc46f89: Pull complete

03b5e4771470: Pull complete

fdd6eab46e6c: Pull complete

80ff62223f5f: Pull complete

e0633204c627: Pull complete

1b3a93e91ad9: Pull complete

29b72f79b735: Pull complete

a2cb53787c38: Pull complete

49e90af50c7d: Pull complete

396607166153: Pull complete

8023c56d0234: Pull complete

440441b9ca5e: Pull complete

4376cc11267c: Pull complete

acb05d09ff6e: Pull complete

Digest: sha256:71781f3cfa51066abb1a4a35267af37beec01b6bb75817fdfae056582839290c

Status: Downloaded newer image for nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8

ubuntu@ip-172-31-14-34:~$ sudo docker images

REPOSITORY TAG IMAGE ID CREATED SIZE

nexus3.onap.info:5000/onap/aaf/aaf_agent 2.1.8 090b326a7f11 5 weeks ago 1.14 GB





# going to test a same size image directly from the LF - with minimal common layers

nexus3.onap.org:10001/onap/testsuite 1.3.2 c4b58baa95e8 3 weeks ago 1.13 GB

# 5 min in we are still at 3% - numbers below are a min old

ubuntu@ip-172-31-14-34:~$ sudo docker pull nexus3.onap.org:10001/onap/testsuite:1.3.2

1.3.2: Pulling from onap/testsuite

32802c0cfa4d: Downloading [=============> ] 8.416 MB/32.1 MB

da1315cffa03: Download complete

fa83472a3562: Download complete

f85999a86bef: Download complete

3eca7452fe93: Downloading [=======================> ] 8.517 MB/17.79 MB

9f002f13a564: Downloading [=========================================> ] 8.528 MB/10.24 MB

02682cf43e5c: Waiting

dfa9878b26c8: Waiting

fdb3a2e7127c: Waiting

adab76b27695: Waiting

3e1db7307fd3: Waiting

d18a8cc3d4cd: Waiting

973f5d70b3c4: Waiting

eb19d05de017: Waiting

ac1f3947b000: Waiting

01ae31c1279e: Waiting

24ac662c0884: Waiting

268c36718aa7: Waiting

98559383536b: Waiting

856a1ffe236e: Waiting

80cbab20328c: Waiting

caed9c28b75f: Waiting

e3c00d7d86d4: Waiting

3109ae4e798e: Waiting

2dc71030fc1d: Waiting

f1d763192c3d: Waiting

adceeebb9b2b: Waiting

419a6a0c4acd: Waiting

048d8fd47ec5: Waiting

63fc3f31cfdc: Waiting

5441f94df8b4: Waiting

754645df4601: Waiting



# in 5 min we get 3% 35/1130Mb - which comes out to 162 min for 1.13G for .org as opposed to 45 sec for .info - which is a 200X slowdown - some of this is due to the fact my nexus3.onap.info is on the same VPC as my test VM - testing on openlab





# openlab - 2 min 40 sec which is 3.6 times slower - expected than in AWS - (25 min pulls vs 90min in openlab) - this makes nexus.onap.org 60 times slower in openlab than a proxy running from AWS (2 vCore/16G/ssd VM)

ubuntu@onap-oom-obrien-rancher-e4:~$ sudo docker pull nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8

2.1.8: Pulling from onap/aaf/aaf_agent



18d680d61657: Pull complete

...

acb05d09ff6e: Pull complete

Digest: sha256:71781f3cfa51066abb1a4a35267af37beec01b6bb75817fdfae056582839290c

Status: Downloaded newer image for nexus3.onap.info:5000/onap/aaf/aaf_agent:2.1.8







#pulling smaller from nexus3.onap.info 2 min 20 - for 36Mb = 0.23Mb/sec - extrapolated to 1.13Gb for above is 5022 sec or 83 min - half the rough calculation above

ubuntu@onap-oom-obrien-rancher-e4:~$ sudo docker pull nexus3.onap.org:10001/onap/aaf/sms:3.0.1

3.0.1: Pulling from onap/aaf/sms

c67f3896b22c: Pull complete

134fac3103be: Pull complete

c7fdcaf7f439: Pull complete

590a963725eb: Pull complete

4df2c296b645: Pull complete

15dd1c6ebaf1: Pull complete

76eeb922b789: Pull complete

Digest: sha256:d5b64947edb93848acacaa9820234aa29e58217db9f878886b7bafae00fdb436

Status: Downloaded newer image for nexus3.onap.org:10001/onap/aaf/sms:3.0.1





# conclusion - nexus3.onap.org is experiencing a routing issue from their DC outbound causing a 80-100x slowdown over a proxy nexus3 - since 20181217 - as local jenkins.onap.org builds complete faster

# workaround is to use a nexus3 proxy above





and adding to values.yaml



global:

#repository: nexus3.onap.org:10001

repository: nexus3.onap.info:5000

repositoryCred:

user: docker

password: docker

This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service


This email and the information contained herein is proprietary and confidential and subject to the Amdocs Email Terms of Service, which you may review at https://www.amdocs.com/about/email-terms-of-service <https://www.amdocs.com/about/email-terms-of-service>

Join onap-discuss@lists.onap.org to automatically receive all group messages.