New Cassablanca install - issues with VFC


Koyle, John
 

Unfortunately this environment is internal.  I've deployed everything as described using helm - a version list follows below.  I did try upgrading the following vfc components (db/vnflcm/nslcm/generic-vnfm-driver/catalog) from v1.2.1 -> v1.2.2 (2018-11-30 release), but this made no difference.

# helm list
NAME                    REVISION        UPDATED                         STATUS          CHART               NAMESPACE
onap                    77              Wed Jun 26 14:05:23 2019        DEPLOYED        onap-3.0.0              onap
...
onap-vfc                1               Wed Jun 26 14:05:59 2019        DEPLOYED        vfc-3.0.0               onap

I believe this is further narrowed down to something IPv6 related (likely env related, as I've not intentionally setup ipv6).  If I use curl -4 http://msb-iag/api/microservices/v1/services the request never fails.  Removing the -4 param and it fails about 30-50% of the time.

John


From: Yan Yang <yangyanyj@...>
Sent: Sunday, July 7, 2019 8:27 PM
To: Koyle, John; 'Catherine LEFEVRE'; onap-users@...
Subject: 答复: [Onap-users] New Cassablanca install - issues with VFC
 

Message received from external source. Exercise caution when opening attachments, clicking links, or exchanging information.

 

Hi John,

 

From your analysis for the problems, it seems DNS issue .

 

In order to better locate the problem, can you tell me the version of vfc and oom you are using ? We will try to reproduce the problem, if your env is public, you can also send me the env information I can login and check the reason.

 

 

BR,

Yan

发件人: Koyle, John [mailto:john.koyle@...]
发送时间: 201976 0:50
收件人: Catherine LEFEVRE; onap-users@...; Yan Yang
主题: Re: [Onap-users] New Cassablanca install - issues with VFC

 

I spent a bit more time with this, and the issue seems container specific and DNS related.

 

Connecting to one of the failing VFC containers and trying to curl http://msb-iag/api/microservices/v1/services fails about 30-40% of the time with the following error:

 

# curl http://msb-iag/api/microservices/v1/services

curl: (6) Could not resolve host: msb-iag

 

Doing a tcpdump on the dns requests show only checks to the host - without the domain search suffix .onap.svc.cluster.local.  Presumably the failing for the the MySQL connections is the same issue.

 

 

Interestingly enough, I spun up a simple busybox container in the same cluster, and using wget to the same URL above there are never any name resolution failures.

 

TIA!

John

 

 


From: onap-users@... <onap-users@...> on behalf of Catherine LEFEVRE <catherine.lefevre@...>
Sent: Friday, July 5, 2019 10:26 AM
To: Koyle, John; onap-users@...; Yan Yang
Subject: Re: [Onap-users] New Cassablanca install - issues with VFC

 

Message received from external source. Exercise caution when opening attachments, clicking links, or exchanging information.

 

Thank you John for reaching us.

Yan is our ONAP VFC PTL.

 

Yan,

Can you please look at this issue?

 

Many thanks & regards

Catherine

 

From: onap-users@... [mailto:onap-users@...] On Behalf Of Koyle, John
Sent: Tuesday, June 25, 2019 12:40 AM
To: onap-users@...
Subject: [Onap-users] New Cassablanca install - issues with VFC

 

I'm new to Onap setup/config but I've nearly got everything is up and running within a virtualized k8s cluster.  A number of the VFC pods are throwing the same errors when the startup.

 

vfc-catalog/vfc-nslcm/vnflcm/vnfmgr all log the following (trimmed errors)

 

2019-06-24 22:27:09|||||lcm.pub.utils.restcall||140440895162112||call_req||ERROR||Traceback (most recent call last):

  File "/service/vfc/nfvo/lcm/lcm/pub/utils/restcall.py", line 70, in call_req

    raise ex

ServerNotFoundError: Unable to find the server at msb-iag

 

....

  File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 963, in connect

    raise exc

django.db.utils.OperationalError: (2003, "Can't connect to MySQL server on 'vfc-db' ([Errno -2] Name or service not known)")

 

 

the vfc-db is up/running - in fact, if I exec into the container I can curl msb-iag and use the mysql client to connect to the db using the credentials (and also see the schema has been deployed).

 

The error seems to indicate a dns/resolution issue - but I have no issues after execing into the container.  In fact, if I kill the processes and restart manually the services appear to start (at least Django comes up and the service binds to the port.)

 

Any thoughts/ideas would be appreciated.

 

Thanks,

John