ubuntu@kub4:~$ kubectl describe pod  dev-oof-oof-cmso-service-6c485cdff-pbzb6  -n onap 

Name:           dev-oof-oof-cmso-service-6c485cdff-pbzb6

Namespace:      onap

Node:           kub3/192.168.13.151

Start Time:     Tue, 25 Dec 2018 11:20:07 +0000

Labels:         app=oof-cmso-service

                pod-template-hash=270417899

                release=dev-oof

Annotations:    <none>

Status:         Pending

IP:             10.42.224.93

Controlled By:  ReplicaSet/dev-oof-oof-cmso-service-6c485cdff

Init Containers:

  oof-cmso-service-readiness:

    Container ID:  docker://bb4ccdfaf3ba6836e606685de4bbe069da2e5193f165ae466f768dad85b71908

    Image:         oomk8s/readiness-check:2.0.0

    Image ID:      docker-pullable://oomk8s/readiness-check@sha256:7daa08b81954360a1111d03364febcb3dcfeb723bcc12ce3eb3ed3e53f2323ed

    Port:          <none>

    Host Port:     <none>

    Command:

      /root/ready.py

    Args:

      --container-name

      cmso-db

    State:          Terminated

      Reason:       Completed

      Exit Code:    0

      Started:      Tue, 25 Dec 2018 11:22:53 +0000

      Finished:     Tue, 25 Dec 2018 11:25:01 +0000

    Ready:          True

    Restart Count:  0

    Environment:

      NAMESPACE:  onap (v1:metadata.namespace)

    Mounts:

      /var/run/secrets/kubernetes.io/serviceaccount from default-token-rm7hn (ro)

  db-init:

    Container ID:   docker://dbc9fadd1140584043b8f690974a4d626f64d12ef5002108b7b5c29148981e23

    Image:          nexus3.onap.org:10001/onap/optf-cmso-dbinit:1.0.1

    Image ID:       docker-pullable://nexus3.onap.org:10001/onap/optf-cmso-dbinit@sha256:c5722a319fb0d91ad4d533597cdee2b55fc5c51d0a8740cf02cbaa1969c8554f

    Port:           <none>

    Host Port:      <none>

    State:          Waiting

      Reason:       CrashLoopBackOff

    Last State:     Terminated

      Reason:       Error

      Exit Code:    1

      Started:      Tue, 25 Dec 2018 11:48:31 +0000

      Finished:     Tue, 25 Dec 2018 11:48:41 +0000

    Ready:          False

    Restart Count:  9

    Environment:

      DB_HOST:      oof-cmso-dbhost.onap

      DB_PORT:      3306

      DB_USERNAME:  root

      DB_SCHEMA:    cmso

      DB_PASSWORD:  <set to the key 'db-root-password' in secret 'dev-oof-cmso-db'>  Optional: false

    Mounts:

      /share/etc/config from dev-oof-oof-cmso-service-config (rw)

      /share/logs from dev-oof-oof-cmso-service-logs (rw)

      /var/run/secrets/kubernetes.io/serviceaccount from default-token-rm7hn (ro)

Containers:

  oof-cmso-service:

    Container ID:   

    Image:          nexus3.onap.org:10001/onap/optf-cmso-service:1.0.1

    Image ID:       

    Port:           8080/TCP

    Host Port:      0/TCP

    State:          Waiting

      Reason:       PodInitializing

    Ready:          False

    Restart Count:  0

    Liveness:       tcp-socket :8080 delay=120s timeout=50s period=10s #success=1 #failure=3

    Readiness:      tcp-socket :8080 delay=100s timeout=50s period=10s #success=1 #failure=3

    Environment:

      DB_HOST:      oof-cmso-dbhost.onap

      DB_PORT:      3306

      DB_USERNAME:  cmso-admin

      DB_SCHEMA:    cmso

      DB_PASSWORD:  <set to the key 'user-password' in secret 'dev-oof-cmso-db'>  Optional: false

    Mounts:

      /share/debug-logs from dev-oof-oof-cmso-service-logs (rw)

      /share/etc/config from dev-oof-oof-cmso-service-config (rw)

      /share/logs from dev-oof-oof-cmso-service-logs (rw)

      /var/run/secrets/kubernetes.io/serviceaccount from default-token-rm7hn (ro)

Conditions:

  Type              Status

  Initialized       False 

  Ready             False 

  ContainersReady   False 

  PodScheduled      True 

Volumes:

  dev-oof-oof-cmso-service-config:

    Type:      ConfigMap (a volume populated by a ConfigMap)

    Name:      dev-oof-oof-cmso-service

    Optional:  false

  dev-oof-oof-cmso-service-logs:

    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)

    Medium:  

  default-token-rm7hn:

    Type:        Secret (a volume populated by a Secret)

    SecretName:  default-token-rm7hn

    Optional:    false

QoS Class:       BestEffort

Node-Selectors:  <none>

Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s

                 node.kubernetes.io/unreachable:NoExecute for 300s

Events:

  Type     Reason                  Age                From               Message

  ----     ------                  ----               ----               -------

  Normal   Scheduled               30m                default-scheduler  Successfully assigned onap/dev-oof-oof-cmso-service-6c485cdff-pbzb6 to kub3

  Warning  FailedCreatePodSandBox  29m                kubelet, kub3      Failed create pod sandbox: rpc error: code = Unknown desc = [failed to set up sandbox container "7d02bb1144aaaf2479a741c971bad617ea532717e7e72d71e2bfeeac992a7451" network for pod "dev-oof-oof-cmso-service-6c485cdff-pbzb6": NetworkPlugin cni failed to set up pod "dev-oof-oof-cmso-service-6c485cdff-pbzb6_onap" network: No MAC address found, failed to clean up sandbox container "7d02bb1144aaaf2479a741c971bad617ea532717e7e72d71e2bfeeac992a7451" network for pod "dev-oof-oof-cmso-service-6c485cdff-pbzb6": NetworkPlugin cni failed to teardown pod "dev-oof-oof-cmso-service-6c485cdff-pbzb6_onap" network: failed to get IP addresses for "eth0": <nil>]

  Normal   SandboxChanged          29m                kubelet, kub3      Pod sandbox changed, it will be killed and re-created.

  Normal   Pulling                 27m                kubelet, kub3      pulling image "oomk8s/readiness-check:2.0.0"

  Normal   Pulled                  27m                kubelet, kub3      Successfully pulled image "oomk8s/readiness-check:2.0.0"

  Normal   Created                 27m                kubelet, kub3      Created container

  Normal   Started                 27m                kubelet, kub3      Started container

  Normal   Pulling                 23m (x4 over 25m)  kubelet, kub3      pulling image "nexus3.onap.org:10001/onap/optf-cmso-dbinit:1.0.1"

  Normal   Pulled                  23m (x4 over 25m)  kubelet, kub3      Successfully pulled image "nexus3.onap.org:10001/onap/optf-cmso-dbinit:1.0.1"

  Normal   Created                 23m (x4 over 25m)  kubelet, kub3      Created container

  Normal   Started                 23m (x4 over 25m)  kubelet, kub3      Started container

  Warning  BackOff                 4m (x80 over 24m)  kubelet, kub3      Back-off restarting failed container

ubuntu@kub4:~$ kubectl logs  -f  dev-oof-oof-cmso-service-6c485cdff-pbzb6  -c oof-cmso-service-readiness -n onap 

2018-12-25 11:22:54,683 - INFO - Checking if cmso-db  is ready

2018-12-25 11:23:02,186 - INFO - Checking if cmso-db  is ready

2018-12-25 11:23:09,950 - INFO - Checking if cmso-db  is ready

2018-12-25 11:23:12,938 - INFO - cmso-db is not ready.

2018-12-25 11:23:17,963 - INFO - Checking if cmso-db  is ready

2018-12-25 11:23:20,091 - INFO - cmso-db is not ready.

2018-12-25 11:23:25,111 - INFO - Checking if cmso-db  is ready

2018-12-25 11:23:27,315 - INFO - cmso-db is not ready.

2018-12-25 11:23:32,329 - INFO - Checking if cmso-db  is ready

2018-12-25 11:23:35,390 - INFO - cmso-db is not ready.

2018-12-25 11:23:40,407 - INFO - Checking if cmso-db  is ready

2018-12-25 11:23:43,346 - INFO - cmso-db is not ready.

2018-12-25 11:23:48,371 - INFO - Checking if cmso-db  is ready

2018-12-25 11:23:53,848 - INFO - cmso-db is not ready.

2018-12-25 11:23:58,870 - INFO - Checking if cmso-db  is ready

2018-12-25 11:24:02,188 - INFO - cmso-db is not ready.

2018-12-25 11:24:07,207 - INFO - Checking if cmso-db  is ready

2018-12-25 11:24:10,598 - INFO - cmso-db is not ready.

2018-12-25 11:24:15,622 - INFO - Checking if cmso-db  is ready

2018-12-25 11:24:18,936 - INFO - cmso-db is not ready.

2018-12-25 11:24:23,955 - INFO - Checking if cmso-db  is ready

2018-12-25 11:24:26,794 - INFO - cmso-db is not ready.

2018-12-25 11:24:31,813 - INFO - Checking if cmso-db  is ready

2018-12-25 11:24:35,529 - INFO - cmso-db is not ready.

2018-12-25 11:24:40,566 - INFO - Checking if cmso-db  is ready

2018-12-25 11:24:44,374 - INFO - cmso-db is not ready.

2018-12-25 11:24:49,403 - INFO - Checking if cmso-db  is ready

2018-12-25 11:24:53,222 - INFO - cmso-db is not ready.

2018-12-25 11:24:58,238 - INFO - Checking if cmso-db  is ready

2018-12-25 11:25:01,340 - INFO - cmso-db is ready!

ubuntu@kub4:~$ kubectl logs  -f  dev-oof-oof-cmso-service-6c485cdff-pbzb6  -c  db-init  -n onap 

VM_ARGS=

 

  .   ____          _            __ _ _

 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \

( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \

 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )

  '  |____| .__|_| |_|_| |_\__, | / / / /

 =========|_|==============|___/=/_/_/_/

 :: Spring Boot ::        (v2.0.6.RELEASE)

 

2018-12-25 11:48:36.187  INFO 8 --- [           main] o.o.o.c.liquibase.LiquibaseApplication   : Starting LiquibaseApplication on dev-oof-oof-cmso-service-6c485cdff-pbzb6 with PID 8 (/opt/app/cmso-dbinit/app.jar started by root in /opt/app/cmso-dbinit)

2018-12-25 11:48:36.199  INFO 8 --- [           main] o.o.o.c.liquibase.LiquibaseApplication   : No active profile set, falling back to default profiles: default

2018-12-25 11:48:36.310  INFO 8 --- [           main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@d44fc21: startup date [Tue Dec 25 11:48:36 UTC 2018]; root of context hierarchy

2018-12-25 11:48:40.336  INFO 8 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Starting...

2018-12-25 11:48:40.754  INFO 8 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Start completed.

2018-12-25 11:48:41.044  WARN 8 --- [           main] s.c.a.AnnotationConfigApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase' defined in class path resource [org/onap/optf/cmso/liquibase/LiquibaseData.class]: Invocation of init method failed; nested exception is liquibase.exception.LockException: liquibase.exception.DatabaseException: liquibase.exception.DatabaseException: java.sql.SQLTransactionRollbackException: (conn=327) Deadlock found when trying to get lock; try restarting transaction

2018-12-25 11:48:41.045  INFO 8 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Shutdown initiated...

2018-12-25 11:48:41.109  INFO 8 --- [           main] com.zaxxer.hikari.HikariDataSource       : HikariPool-1 - Shutdown completed.

2018-12-25 11:48:41.177  INFO 8 --- [           main] ConditionEvaluationReportLoggingListener : 

 

Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.

2018-12-25 11:48:41.223 ERROR 8 --- [           main] o.s.boot.SpringApplication               : Application run failed

 

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'liquibase' defined in class path resource [org/onap/optf/cmso/liquibase/LiquibaseData.class]: Invocation of init method failed; nested exception is liquibase.exception.LockException: liquibase.exception.DatabaseException: liquibase.exception.DatabaseException: java.sql.SQLTransactionRollbackException: (conn=327) Deadlock found when trying to get lock; try restarting transaction

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1694) ~[spring-beans-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:573) ~[spring-beans-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:495) ~[spring-beans-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]

at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:317) ~[spring-beans-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]

at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) ~[spring-beans-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]

at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:315) ~[spring-beans-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]

at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199) ~[spring-beans-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]

at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:759) ~[spring-beans-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]

at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867) ~[spring-context-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]

at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:548) ~[spring-context-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]

at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:754) [spring-boot-2.0.6.RELEASE.jar!/:2.0.6.RELEASE]

at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:386) [spring-boot-2.0.6.RELEASE.jar!/:2.0.6.RELEASE]

at org.springframework.boot.SpringApplication.run(SpringApplication.java:307) [spring-boot-2.0.6.RELEASE.jar!/:2.0.6.RELEASE]

at org.springframework.boot.SpringApplication.run(SpringApplication.java:1242) [spring-boot-2.0.6.RELEASE.jar!/:2.0.6.RELEASE]

at org.springframework.boot.SpringApplication.run(SpringApplication.java:1230) [spring-boot-2.0.6.RELEASE.jar!/:2.0.6.RELEASE]

at org.onap.optf.cmso.liquibase.LiquibaseApplication.main(LiquibaseApplication.java:45) [classes!/:na]

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_181]

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_181]

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_181]

at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_181]

at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48) [app.jar:na]

at org.springframework.boot.loader.Launcher.launch(Launcher.java:87) [app.jar:na]

at org.springframework.boot.loader.Launcher.launch(Launcher.java:50) [app.jar:na]

at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51) [app.jar:na]

Caused by: liquibase.exception.LockException: liquibase.exception.DatabaseException: liquibase.exception.DatabaseException: java.sql.SQLTransactionRollbackException: (conn=327) Deadlock found when trying to get lock; try restarting transaction

at liquibase.lockservice.StandardLockService.acquireLock(StandardLockService.java:242) ~[liquibase-core-3.5.5.jar!/:na]

at liquibase.lockservice.StandardLockService.waitForLock(StandardLockService.java:170) ~[liquibase-core-3.5.5.jar!/:na]

at liquibase.Liquibase.update(Liquibase.java:196) ~[liquibase-core-3.5.5.jar!/:na]

at liquibase.Liquibase.update(Liquibase.java:192) ~[liquibase-core-3.5.5.jar!/:na]

at liquibase.integration.spring.SpringLiquibase.performUpdate(SpringLiquibase.java:431) ~[liquibase-core-3.5.5.jar!/:na]

at liquibase.integration.spring.SpringLiquibase.afterPropertiesSet(SpringLiquibase.java:388) ~[liquibase-core-3.5.5.jar!/:na]

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1753) ~[spring-beans-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]

at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1690) ~[spring-beans-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]

... 23 common frames omitted

Caused by: liquibase.exception.DatabaseException: liquibase.exception.DatabaseException: java.sql.SQLTransactionRollbackException: (conn=327) Deadlock found when trying to get lock; try restarting transaction

at liquibase.database.AbstractJdbcDatabase.commit(AbstractJdbcDatabase.java:1159) ~[liquibase-core-3.5.5.jar!/:na]

at liquibase.lockservice.StandardLockService.acquireLock(StandardLockService.java:233) ~[liquibase-core-3.5.5.jar!/:na]

... 30 common frames omitted

Caused by: liquibase.exception.DatabaseException: java.sql.SQLTransactionRollbackException: (conn=327) Deadlock found when trying to get lock; try restarting transaction

at liquibase.database.jvm.JdbcConnection.commit(JdbcConnection.java:126) ~[liquibase-core-3.5.5.jar!/:na]

at liquibase.database.AbstractJdbcDatabase.commit(AbstractJdbcDatabase.java:1157) ~[liquibase-core-3.5.5.jar!/:na]

... 31 common frames omitted

Caused by: java.sql.SQLTransactionRollbackException: (conn=327) Deadlock found when trying to get lock; try restarting transaction

at org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.get(ExceptionMapper.java:179) ~[mariadb-java-client-2.2.6.jar!/:na]

at org.mariadb.jdbc.internal.util.exceptions.ExceptionMapper.getException(ExceptionMapper.java:110) ~[mariadb-java-client-2.2.6.jar!/:na]

at org.mariadb.jdbc.MariaDbStatement.executeExceptionEpilogue(MariaDbStatement.java:228) ~[mariadb-java-client-2.2.6.jar!/:na]

at org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:334) ~[mariadb-java-client-2.2.6.jar!/:na]

at org.mariadb.jdbc.MariaDbStatement.execute(MariaDbStatement.java:386) ~[mariadb-java-client-2.2.6.jar!/:na]

at org.mariadb.jdbc.MariaDbConnection.commit(MariaDbConnection.java:709) ~[mariadb-java-client-2.2.6.jar!/:na]

at com.zaxxer.hikari.pool.ProxyConnection.commit(ProxyConnection.java:368) ~[HikariCP-2.7.9.jar!/:na]

at com.zaxxer.hikari.pool.HikariProxyConnection.commit(HikariProxyConnection.java) ~[HikariCP-2.7.9.jar!/:na]

at liquibase.database.jvm.JdbcConnection.commit(JdbcConnection.java:123) ~[liquibase-core-3.5.5.jar!/:na]

... 32 common frames omitted

Caused by: java.sql.SQLException: Deadlock found when trying to get lock; try restarting transaction

Query is: COMMIT

at org.mariadb.jdbc.internal.util.LogQueryTool.exceptionWithQuery(LogQueryTool.java:119) ~[mariadb-java-client-2.2.6.jar!/:na]

at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:200) ~[mariadb-java-client-2.2.6.jar!/:na]

at org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:328) ~[mariadb-java-client-2.2.6.jar!/:na]

... 37 common frames omitted

 

ubuntu@kub4:~$