CD server live for pomba or logging healthcheck on the hour - Jenkins JobBuilder validation non-triggered batch #log #oom #lfn

Michael O'Brien <frank.obrien@...>


   This morning we discussed the recent POMBA healthcheck failure (1 failing 2 passing) that occurred as a regression on the 28th.

   Issue was:

              Dependent or multi-merge reviews – where they need to be merged in sequence – sequence was off on this one.

   Fixed: Prudence Au (my co-PTL) in

   helm install verified:


Basic Pomba AAI-context-builder Health Check | PASS |

Basic Pomba SDC-context-builder Health Check | PASS |

Basic Pomba Network-discovery-context-builder Health Check | PASS |


              In the future anyone on the pomba team or from onap can check the status of a helm deployment every hour – specific to the 6 healthchecks for logging and pomba.  Each build subset with 3 –set overrides takes only 20 min to run.  In the future this type of helm install/deploy can be added to the helm verify already run as part of the JobBuilder.        


   This was a result of a change going into the application that had a dependency on an OOM config merge – the issue was that the change was tested with a local OOM change – instead of merging OOM just before the POMBA merge.

    In the future when each application owns their config (in the works)  - then these sort of multi-commit transactions can be done easier – for now they need to be closely managed (test both patches, post results, post healthcheck/container status – merge in dependency order).

     I am also adding a 4th healthcheck on the kibana pod this week – it is the last to come up on log, clamp and pomba by nature.


     In the meantime I have requested

1: that HC and container status be posted to the review/jira

2: that a clean build be tested ideally with the patch(s) on a separate machine – if possible

3: that each repo merge be linked with a described merge sequence (ie: post in POMBA not to merge before OOM patch is merged)

4: test master immediately after both merges – ideally this is the job of CI/CD – with triggered deployment in the queue

we have CD in at a 4 hour interval to check

we have CD in at a 1 hour interval to check (a subset of the full deploy for speed – and cost reduction)

Note: I am only deploying log, robot and kibana here – the failed HC for other components are expected

Healthcheck pass/fail dashboards

logs on

server on (protected to secure ports 10249-10255 from crypto miners)


thank you



This message and the information contained herein is proprietary and confidential and subject to the Amdocs policy statement,