New Cassablanca install - issues with VFC


Koyle, John <john.koyle@...>
 

I'm new to Onap setup/config but I've nearly got everything is up and running within a virtualized k8s cluster.  A number of the VFC pods are throwing the same errors when the startup.

vfc-catalog/vfc-nslcm/vnflcm/vnfmgr all log the following (trimmed errors)

2019-06-24 22:27:09|||||lcm.pub.utils.restcall||140440895162112||call_req||ERROR||Traceback (most recent call last):
  File "/service/vfc/nfvo/lcm/lcm/pub/utils/restcall.py", line 70, in call_req
    raise ex
ServerNotFoundError: Unable to find the server at msb-iag

....
  File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 963, in connect
    raise exc
django.db.utils.OperationalError: (2003, "Can't connect to MySQL server on 'vfc-db' ([Errno -2] Name or service not known)")


the vfc-db is up/running - in fact, if I exec into the container I can curl msb-iag and use the mysql client to connect to the db using the credentials (and also see the schema has been deployed).

The error seems to indicate a dns/resolution issue - but I have no issues after execing into the container.  In fact, if I kill the processes and restart manually the services appear to start (at least Django comes up and the service binds to the port.)

Any thoughts/ideas would be appreciated.

Thanks,
John


Catherine LEFEVRE
 

Thank you John for reaching us.

Yan is our ONAP VFC PTL.

 

Yan,

Can you please look at this issue?

 

Many thanks & regards

Catherine

 

From: onap-users@... [mailto:onap-users@...] On Behalf Of Koyle, John
Sent: Tuesday, June 25, 2019 12:40 AM
To: onap-users@...
Subject: [Onap-users] New Cassablanca install - issues with VFC

 

I'm new to Onap setup/config but I've nearly got everything is up and running within a virtualized k8s cluster.  A number of the VFC pods are throwing the same errors when the startup.

 

vfc-catalog/vfc-nslcm/vnflcm/vnfmgr all log the following (trimmed errors)

 

2019-06-24 22:27:09|||||lcm.pub.utils.restcall||140440895162112||call_req||ERROR||Traceback (most recent call last):

  File "/service/vfc/nfvo/lcm/lcm/pub/utils/restcall.py", line 70, in call_req

    raise ex

ServerNotFoundError: Unable to find the server at msb-iag

 

....

  File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 963, in connect

    raise exc

django.db.utils.OperationalError: (2003, "Can't connect to MySQL server on 'vfc-db' ([Errno -2] Name or service not known)")

 

 

the vfc-db is up/running - in fact, if I exec into the container I can curl msb-iag and use the mysql client to connect to the db using the credentials (and also see the schema has been deployed).

 

The error seems to indicate a dns/resolution issue - but I have no issues after execing into the container.  In fact, if I kill the processes and restart manually the services appear to start (at least Django comes up and the service binds to the port.)

 

Any thoughts/ideas would be appreciated.

 

Thanks,

John


Koyle, John
 

I spent a bit more time with this, and the issue seems container specific and DNS related.

Connecting to one of the failing VFC containers and trying to curl http://msb-iag/api/microservices/v1/services fails about 30-40% of the time with the following error:

# curl http://msb-iag/api/microservices/v1/services
curl: (6) Could not resolve host: msb-iag

Doing a tcpdump on the dns requests show only checks to the host - without the domain search suffix .onap.svc.cluster.local.  Presumably the failing for the the MySQL connections is the same issue.



Interestingly enough, I spun up a simple busybox container in the same cluster, and using wget to the same URL above there are never any name resolution failures.

TIA!
John



From: onap-users@... <onap-users@...> on behalf of Catherine LEFEVRE <catherine.lefevre@...>
Sent: Friday, July 5, 2019 10:26 AM
To: Koyle, John; onap-users@...; Yan Yang
Subject: Re: [Onap-users] New Cassablanca install - issues with VFC
 

Message received from external source. Exercise caution when opening attachments, clicking links, or exchanging information.

 

Thank you John for reaching us.

Yan is our ONAP VFC PTL.

 

Yan,

Can you please look at this issue?

 

Many thanks & regards

Catherine

 

From: onap-users@... [mailto:onap-users@...] On Behalf Of Koyle, John
Sent: Tuesday, June 25, 2019 12:40 AM
To: onap-users@...
Subject: [Onap-users] New Cassablanca install - issues with VFC

 

I'm new to Onap setup/config but I've nearly got everything is up and running within a virtualized k8s cluster.  A number of the VFC pods are throwing the same errors when the startup.

 

vfc-catalog/vfc-nslcm/vnflcm/vnfmgr all log the following (trimmed errors)

 

2019-06-24 22:27:09|||||lcm.pub.utils.restcall||140440895162112||call_req||ERROR||Traceback (most recent call last):

  File "/service/vfc/nfvo/lcm/lcm/pub/utils/restcall.py", line 70, in call_req

    raise ex

ServerNotFoundError: Unable to find the server at msb-iag

 

....

  File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 963, in connect

    raise exc

django.db.utils.OperationalError: (2003, "Can't connect to MySQL server on 'vfc-db' ([Errno -2] Name or service not known)")

 

 

the vfc-db is up/running - in fact, if I exec into the container I can curl msb-iag and use the mysql client to connect to the db using the credentials (and also see the schema has been deployed).

 

The error seems to indicate a dns/resolution issue - but I have no issues after execing into the container.  In fact, if I kill the processes and restart manually the services appear to start (at least Django comes up and the service binds to the port.)

 

Any thoughts/ideas would be appreciated.

 

Thanks,

John