New Cassablanca install - issues with VFC
Unfortunately this environment is internal. I've deployed everything as described using helm - a version list follows below. I did try upgrading the following vfc components (db/vnflcm/nslcm/generic-vnfm-driver/catalog) from v1.2.1 -> v1.2.2 (2018-11-30 release), but this made no difference.
# helm list
NAME REVISION UPDATED STATUS CHART NAMESPACE
onap 77 Wed Jun 26 14:05:23 2019 DEPLOYED onap-3.0.0 onap...
onap-vfc 1 Wed Jun 26 14:05:59 2019 DEPLOYED vfc-3.0.0 onap
I believe this is further narrowed down to something IPv6 related (likely env related, as I've not intentionally setup ipv6). If I use curl -4 http://msb-iag/api/microservices/v1/services the request never fails. Removing the -4 param and it fails about 30-50% of the time.
From: Yan Yang <yangyanyj@...>
Sent: Sunday, July 7, 2019 8:27 PM
To: Koyle, John; 'Catherine LEFEVRE'; onap-users@...
Subject: 答复: [Onap-users] New Cassablanca install - issues with VFC
From your analysis for the problems, it seems DNS issue .
In order to better locate the problem, can you tell me the version of vfc and oom you are using ? We will try to reproduce the problem, if your env is public, you can also send me the env information I can login and check the reason.
发件人: Koyle, John [mailto:john.koyle@...]
I spent a bit more time with this, and the issue seems container specific and DNS related.
Connecting to one of the failing VFC containers and trying to curl http://msb-iag/api/microservices/v1/services fails about 30-40% of the time with the following error:
curl: (6) Could not resolve host: msb-iag
Doing a tcpdump on the dns requests show only checks to the host - without the domain search suffix .onap.svc.cluster.local. Presumably the failing for the the MySQL connections is the same issue.
Interestingly enough, I spun up a simple busybox container in the same cluster, and using wget to the same URL above there are never any name resolution failures.
onap-users@... <onap-users@...> on behalf of Catherine LEFEVRE <catherine.lefevre@...>
Thank you John for reaching us.
Yan is our ONAP VFC PTL.
Can you please look at this issue?
Many thanks & regards
I'm new to Onap setup/config but I've nearly got everything is up and running within a virtualized k8s cluster. A number of the VFC pods are throwing the same errors when the startup.
vfc-catalog/vfc-nslcm/vnflcm/vnfmgr all log the following (trimmed errors)
2019-06-24 22:27:09|||||lcm.pub.utils.restcall||140440895162112||call_req||ERROR||Traceback (most recent call last):
File "/service/vfc/nfvo/lcm/lcm/pub/utils/restcall.py", line 70, in call_req
ServerNotFoundError: Unable to find the server at msb-iag
File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 963, in connect
django.db.utils.OperationalError: (2003, "Can't connect to MySQL server on 'vfc-db' ([Errno -2] Name or service not known)")
the vfc-db is up/running - in fact, if I exec into the container I can curl msb-iag and use the mysql client to connect to the db using the credentials (and also see the schema has been deployed).
The error seems to indicate a dns/resolution issue - but I have no issues after execing into the container. In fact, if I kill the processes and restart manually the services appear to start (at least Django comes up and the service binds to the port.)
Any thoughts/ideas would be appreciated.