Re: revised code coverage requirement

Pawel Pawlak
 

+1

Paweł Pawlak

ONAP SECCOM Chair
Leader in IT & Network Convergent Operations
FT/TGI/OLN/TOP/OST

Orange Polska S.A.
Corporate Services Agency

Obrzeżna 7, 02-691 Warszawa
tel. +48 22 699 52 17
fax +48 22 857 99 86
tel. mob. +48 501 501 030

   Czy musisz drukować tę wiadomość? Pomyśl o środowisku.
__________________________________________________________________
Treść tej wiadomości jest własnością Orange Polska i zawiera informacje stanowiące tajemnicę przedsiębiorstwa Orange Polska. Jeżeli nie jesteście Państwo jej adresatem, bądź otrzymaliście ją przez pomyłkę, prosimy o powiadomienie o tym nadawcy oraz trwałe jej usunięcie. Orange Polska Spółka Akcyjna z siedzibą i adresem w Warszawie (02-326) przy Al. Jerozolimskich 160, wpisana do Rejestru Przedsiębiorców prowadzonego przez Sąd Rejonowy dla m.st. Warszawy XII Wydział Gospodarczy Krajowego Rejestru Sądowego pod numerem 0000010681; REGON 012100784, NIP 526-02-50-995; z pokrytym w całości kapitałem zakładowym wynoszącym 3.937.072.437 złotych.

-----Original Message-----
From: Close, Pierre [mailto:pierre.close@...]
Sent: Monday, November 4, 2019 12:32 PM
To: k.opasiak@...; ZWARICO, AMY; Pawlak Paweł 3 - Korpo; onap-seccom@...
Subject: RE: [Onap-seccom] revised code coverage requirement

Amy,

I agree with the general plan, but I'm afraid that global 80% coverage could lead developers to build tests for the sake of only having more coverage, mostly because the amount of available resources for feature development not always allows to add/improve testing at the same time.

Colleagues suggested the following: contextualize the scope of coverage. By this, it is meant to:
- define, for each project, the core parts that need intensive testing (an API sensitive project might prioritize API testing so that all are covered, hardened, so that said APIs are robust)
- concentrate coverage on those areas, even if coverage is lower on other parts, so critical parts are better covered and tested.
This might improve OJSI resolution (and/or reduce findings), and better focuses the effort on testing based on available resources.
This would be even better if we could tell Sonar which parts of the code are the critical ones, but I don't know if that's feasible.

Thoughts are welcome

Best regards,
Pierre

-----Original Message-----
From: onap-seccom@... <onap-seccom@...> On Behalf Of Krzysztof Opasiak via Lists.Onap.Org
Sent: Monday, November 4, 2019 10:02 AM
To: ZWARICO, AMY <az9121@...>; Close, Pierre <pierre.close@...>; Pawlak Paweł 3 - Korpo <Pawel.Pawlak3@...>; onap-seccom@...
Cc: onap-seccom@...
Subject: Re: [Onap-seccom] revised code coverage requirement



On 31.10.2019 13:36, ZWARICO, AMY wrote:
Unfortunately there is urgency, and considerable pushback. I propose that we investigate the "new code tested" feature of SONAR in the Frankfurt release, create a clear definition of "new code", find a project that will test it, and, if all goes well, add the requirement that all new code contributions have 80% statement coverage in the Guilin release.

Thoughts from others?
+1 from my side


-----Original Message-----
From: CLOSE, PIERRE
Sent: Thursday, October 31, 2019 6:14 AM
To: Krzysztof Opasiak <k.opasiak@...>; ZWARICO, AMY
<az9121@...>; Pawlak Paweł 3 - Korpo <Pawel.Pawlak3@...>;
onap-seccom@...
Subject: RE: [Onap-seccom] revised code coverage requirement

Hello Krzysztof,

Please find my comments inline.

Amy,

Is it possible to wait a little longer before agreeing on this requirement? Or do we have an urgency in deciding today -- in such a case, Krzysztof's comments lead me to delay the use of Sonar until we have a solid plan.

Best regards,
Pierre

-----Original Message-----
From: Krzysztof Opasiak <k.opasiak@...>
Sent: Wednesday, October 30, 2019 10:10 PM
To: Close, Pierre <pierre.close@...>; ZWARICO, AMY
<az9121@...>; Pawlak Paweł 3 - Korpo <Pawel.Pawlak3@...>;
onap-seccom@...
Subject: Re: [Onap-seccom] revised code coverage requirement

Hi,

On 30.10.2019 19:24, Pierre Close wrote:
Amy,

That’s a good question. But your explanation makes sense to me: when
a new Sonar scan is run it probably overrides the result of the
previous one, causing those counters to reset… Additionally, what is
considered “new code”? Is a configuration file change seen as new code?

I’ll try to look for an explanation from my colleagues and come back
to you when I got an answer (or maybe the LF knows?)
We would need a clear definition of the new code. Is a change in currently existing class a new code or only newly created classes? Is new function withing a class a new code?
[PC]: Agreed. That's what I was suggesting in my comment above.

Saying that we will just use the metric that the tool provides without
any explanation to the community is not a good thing I believe
[PC]: I think we need to dig further into the capabilities of the tool and verify how we can use the metrics in a proper and meaningful way, otherwise it won't be of any use, as you mention.


In the meantime, since the feature seems available in the tool, we
might want to consider binding the “new code coverage” objective to
the tool – in other words, if the feature is usable and gives what we
expect, we could start monitoring it.

Would that make sense?
As far as I can see (please correct me if I'm wrong) the tool gives
you only the current status of the code and diff since last run. What
we would need is a dashboard as bitergia where you can select any
arbitrary period of time and check the results
[PC]: I was under the impression that you could choose different versions to compare between... Now, your comments being valid, I propose that we take some time to further analyze what we can do, build (if possible) meaningful metrics and dashboards (as you mention), and then decide what route to take -- I know the timing is against us, but throwing all this into the community without further research is not constructive, so I support your statements. Would you agree with this approach?

Best regards,
--
Krzysztof Opasiak
Samsung R&D Institute Poland
Samsung Electronics
--
Krzysztof Opasiak
Samsung R&D Institute Poland
Samsung Electronics

Join onap-seccom@lists.onap.org to automatically receive all group messages.