Topics

revised code coverage requirement

Amy Zwarico
 

Please review https://jira.onap.org/browse/REQ-207 updated based on feedback at the PTL call and discussion during the SECCOM meeting. I would like to finalize this requirement before the TSC call on Thursday 19/10/31.

 

​​​​​Amy Zwarico, LMTS

Chief Security Office / Emerging Services Security

AT&T Services

(205) 613-1667

 

"This e-mail and any files transmitted with it are the property of AT&T,  and are intended solely for the use of the individual or entity to whom this e-mail is addressed. If you are not one of the named recipient(s) or otherwise have reason to believe that you have received this message in error, please notify the sender and delete this message immediately from your electronic device. Any other use, retention, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited."

 

Pierre Close
 

Hello,

 

I have updated the ticket with a link to Sonar coverage stats – this might help reach part of the coverage goals.

 

Best regards,

Pierre

 

From: onap-seccom@... <onap-seccom@...> On Behalf Of ZWARICO, AMY
Sent: Tuesday, October 29, 2019 3:28 PM
To: Pawlak Paweł 3 - Korpo <Pawel.Pawlak3@...>; onap-seccom@...
Subject: [Onap-seccom] revised code coverage requirement

 

***Security Advisory: This Message Originated Outside of AT&T ***
Reference http://cso.att.com/EmailSecurity/IDSP.html for more information.

Please review https://jira.onap.org/browse/REQ-207 updated based on feedback at the PTL call and discussion during the SECCOM meeting. I would like to finalize this requirement before the TSC call on Thursday 19/10/31.

 

​​​​​Amy Zwarico, LMTS

Chief Security Office / Emerging Services Security

AT&T Services

(205) 613-1667

 

"This e-mail and any files transmitted with it are the property of AT&T,  and are intended solely for the use of the individual or entity to whom this e-mail is addressed. If you are not one of the named recipient(s) or otherwise have reason to believe that you have received this message in error, please notify the sender and delete this message immediately from your electronic device. Any other use, retention, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited."

 

Amy Zwarico
 

I’ve used the sonar high level reports, but never drilled into detail. It looks as if we can look at metrics for new code. Pierre, are the numbers coming up 0 because we run the Sonar jobs as part of Jenkins?

 

 

From: CLOSE, PIERRE
Sent: Wednesday, October 30, 2019 12:27 PM
To: ZWARICO, AMY <az9121@...>; Pawlak Paweł 3 - Korpo <Pawel.Pawlak3@...>; onap-seccom@...
Subject: RE: revised code coverage requirement

 

Hello,

 

I have updated the ticket with a link to Sonar coverage stats – this might help reach part of the coverage goals.

 

Best regards,

Pierre

 

From: onap-seccom@... <onap-seccom@...> On Behalf Of ZWARICO, AMY
Sent: Tuesday, October 29, 2019 3:28 PM
To: Pawlak Paweł 3 - Korpo <Pawel.Pawlak3@...>; onap-seccom@...
Subject: [Onap-seccom] revised code coverage requirement

 

***Security Advisory: This Message Originated Outside of AT&T ***
Reference http://cso.att.com/EmailSecurity/IDSP.html for more information.

Please review https://jira.onap.org/browse/REQ-207 updated based on feedback at the PTL call and discussion during the SECCOM meeting. I would like to finalize this requirement before the TSC call on Thursday 19/10/31.

 

​​​​​Amy Zwarico, LMTS

Chief Security Office / Emerging Services Security

AT&T Services

(205) 613-1667

 

"This e-mail and any files transmitted with it are the property of AT&T,  and are intended solely for the use of the individual or entity to whom this e-mail is addressed. If you are not one of the named recipient(s) or otherwise have reason to believe that you have received this message in error, please notify the sender and delete this message immediately from your electronic device. Any other use, retention, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited."

 

Pierre Close
 

Amy,

 

That’s a good question. But your explanation makes sense to me: when a new Sonar scan is run it probably overrides the result of the previous one, causing those counters to reset…  Additionally, what is considered “new code”? Is a configuration file change seen as new code?

 

I’ll try to look for an explanation from my colleagues and come back to you when I got an answer (or maybe the LF knows?)

 

In the meantime, since the feature seems available in the tool, we might want to consider binding the “new code coverage” objective to the tool – in other words, if the feature is usable and gives what we expect, we could start monitoring it.

 

Would that make sense?

 

Best regards,

Pierre

 

From: ZWARICO, AMY <az9121@...>
Sent: Wednesday, October 30, 2019 6:44 PM
To: Close, Pierre <pierre.close@...>; Pawlak Paweł 3 - Korpo <Pawel.Pawlak3@...>; onap-seccom@...
Subject: RE: revised code coverage requirement

 

I’ve used the sonar high level reports, but never drilled into detail. It looks as if we can look at metrics for new code. Pierre, are the numbers coming up 0 because we run the Sonar jobs as part of Jenkins?

 

 

From: CLOSE, PIERRE
Sent: Wednesday, October 30, 2019 12:27 PM
To: ZWARICO, AMY <az9121@...>; Pawlak Paweł 3 - Korpo <Pawel.Pawlak3@...>; onap-seccom@...
Subject: RE: revised code coverage requirement

 

Hello,

 

I have updated the ticket with a link to Sonar coverage stats – this might help reach part of the coverage goals.

 

Best regards,

Pierre

 

From: onap-seccom@... <onap-seccom@...> On Behalf Of ZWARICO, AMY
Sent: Tuesday, October 29, 2019 3:28 PM
To: Pawlak Paweł 3 - Korpo <Pawel.Pawlak3@...>; onap-seccom@...
Subject: [Onap-seccom] revised code coverage requirement

 

***Security Advisory: This Message Originated Outside of AT&T ***
Reference http://cso.att.com/EmailSecurity/IDSP.html for more information.

Please review https://jira.onap.org/browse/REQ-207 updated based on feedback at the PTL call and discussion during the SECCOM meeting. I would like to finalize this requirement before the TSC call on Thursday 19/10/31.

 

​​​​​Amy Zwarico, LMTS

Chief Security Office / Emerging Services Security

AT&T Services

(205) 613-1667

 

"This e-mail and any files transmitted with it are the property of AT&T,  and are intended solely for the use of the individual or entity to whom this e-mail is addressed. If you are not one of the named recipient(s) or otherwise have reason to believe that you have received this message in error, please notify the sender and delete this message immediately from your electronic device. Any other use, retention, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited."

 

Krzysztof Opasiak
 

Hi,

On 30.10.2019 19:24, Pierre Close wrote:
Amy,

That’s a good question. But your explanation makes sense to me: when a
new Sonar scan is run it probably overrides the result of the previous
one, causing those counters to reset…  Additionally, what is considered
“new code”? Is a configuration file change seen as new code?

I’ll try to look for an explanation from my colleagues and come back to
you when I got an answer (or maybe the LF knows?)
We would need a clear definition of the new code. Is a change in
currently existing class a new code or only newly created classes? Is
new function withing a class a new code?

Saying that we will just use the metric that the tool provides without
any explanation to the community is not a good thing I believe


In the meantime, since the feature seems available in the tool, we might
want to consider binding the “new code coverage” objective to the tool –
in other words, if the feature is usable and gives what we expect, we
could start monitoring it.

Would that make sense?
As far as I can see (please correct me if I'm wrong) the tool gives you
only the current status of the code and diff since last run. What we
would need is a dashboard as bitergia where you can select any arbitrary
period of time and check the results

Best regards,
--
Krzysztof Opasiak
Samsung R&D Institute Poland
Samsung Electronics

Pierre Close
 

Hello Krzysztof,

Please find my comments inline.

Amy,

Is it possible to wait a little longer before agreeing on this requirement? Or do we have an urgency in deciding today -- in such a case, Krzysztof's comments lead me to delay the use of Sonar until we have a solid plan.

Best regards,
Pierre

-----Original Message-----
From: Krzysztof Opasiak <k.opasiak@...>
Sent: Wednesday, October 30, 2019 10:10 PM
To: Close, Pierre <pierre.close@...>; ZWARICO, AMY <az9121@...>; Pawlak Paweł 3 - Korpo <Pawel.Pawlak3@...>; onap-seccom@...
Subject: Re: [Onap-seccom] revised code coverage requirement

Hi,

On 30.10.2019 19:24, Pierre Close wrote:
Amy,

That’s a good question. But your explanation makes sense to me: when a
new Sonar scan is run it probably overrides the result of the previous
one, causing those counters to reset…  Additionally, what is
considered “new code”? Is a configuration file change seen as new code?

I’ll try to look for an explanation from my colleagues and come back
to you when I got an answer (or maybe the LF knows?)
We would need a clear definition of the new code. Is a change in currently existing class a new code or only newly created classes? Is new function withing a class a new code?
[PC]: Agreed. That's what I was suggesting in my comment above.

Saying that we will just use the metric that the tool provides without any explanation to the community is not a good thing I believe
[PC]: I think we need to dig further into the capabilities of the tool and verify how we can use the metrics in a proper and meaningful way, otherwise it won't be of any use, as you mention.


In the meantime, since the feature seems available in the tool, we
might want to consider binding the “new code coverage” objective to
the tool – in other words, if the feature is usable and gives what we
expect, we could start monitoring it.

Would that make sense?
As far as I can see (please correct me if I'm wrong) the tool gives you only the current status of the code and diff since last run. What we would need is a dashboard as bitergia where you can select any arbitrary period of time and check the results
[PC]: I was under the impression that you could choose different versions to compare between... Now, your comments being valid, I propose that we take some time to further analyze what we can do, build (if possible) meaningful metrics and dashboards (as you mention), and then decide what route to take -- I know the timing is against us, but throwing all this into the community without further research is not constructive, so I support your statements. Would you agree with this approach?

Best regards,
--
Krzysztof Opasiak
Samsung R&D Institute Poland
Samsung Electronics

Amy Zwarico
 

Unfortunately there is urgency, and considerable pushback. I propose that we investigate the "new code tested" feature of SONAR in the Frankfurt release, create a clear definition of "new code", find a project that will test it, and, if all goes well, add the requirement that all new code contributions have 80% statement coverage in the Guilin release.

Thoughts from others?

-----Original Message-----
From: CLOSE, PIERRE
Sent: Thursday, October 31, 2019 6:14 AM
To: Krzysztof Opasiak <k.opasiak@...>; ZWARICO, AMY <az9121@...>; Pawlak Paweł 3 - Korpo <Pawel.Pawlak3@...>; onap-seccom@...
Subject: RE: [Onap-seccom] revised code coverage requirement

Hello Krzysztof,

Please find my comments inline.

Amy,

Is it possible to wait a little longer before agreeing on this requirement? Or do we have an urgency in deciding today -- in such a case, Krzysztof's comments lead me to delay the use of Sonar until we have a solid plan.

Best regards,
Pierre

-----Original Message-----
From: Krzysztof Opasiak <k.opasiak@...>
Sent: Wednesday, October 30, 2019 10:10 PM
To: Close, Pierre <pierre.close@...>; ZWARICO, AMY <az9121@...>; Pawlak Paweł 3 - Korpo <Pawel.Pawlak3@...>; onap-seccom@...
Subject: Re: [Onap-seccom] revised code coverage requirement

Hi,

On 30.10.2019 19:24, Pierre Close wrote:
Amy,

That’s a good question. But your explanation makes sense to me: when a
new Sonar scan is run it probably overrides the result of the previous
one, causing those counters to reset… Additionally, what is
considered “new code”? Is a configuration file change seen as new code?

I’ll try to look for an explanation from my colleagues and come back
to you when I got an answer (or maybe the LF knows?)
We would need a clear definition of the new code. Is a change in currently existing class a new code or only newly created classes? Is new function withing a class a new code?
[PC]: Agreed. That's what I was suggesting in my comment above.

Saying that we will just use the metric that the tool provides without any explanation to the community is not a good thing I believe
[PC]: I think we need to dig further into the capabilities of the tool and verify how we can use the metrics in a proper and meaningful way, otherwise it won't be of any use, as you mention.


In the meantime, since the feature seems available in the tool, we
might want to consider binding the “new code coverage” objective to
the tool – in other words, if the feature is usable and gives what we
expect, we could start monitoring it.

Would that make sense?
As far as I can see (please correct me if I'm wrong) the tool gives you only the current status of the code and diff since last run. What we would need is a dashboard as bitergia where you can select any arbitrary period of time and check the results
[PC]: I was under the impression that you could choose different versions to compare between... Now, your comments being valid, I propose that we take some time to further analyze what we can do, build (if possible) meaningful metrics and dashboards (as you mention), and then decide what route to take -- I know the timing is against us, but throwing all this into the community without further research is not constructive, so I support your statements. Would you agree with this approach?

Best regards,
--
Krzysztof Opasiak
Samsung R&D Institute Poland
Samsung Electronics

seshu kumar m
 

Hi Amy

I respect the intension of having the 80% coverage on new lines added.
But given the case of ONAP we have most of the involved changes are over the existing code, this would bring in difficulty in measuring the coverage specific to the new code added.

Thanks and Regards,
M Seshu Kumar
Senior System Architect
Single OSS India Branch Department. S/W BU.
Huawei Technologies India Pvt. Ltd.
Survey No. 37, Next to EPIP Area, Kundalahalli, Whitefield
Bengaluru-560066, Karnataka.
Tel: + 91-80-49160700 , Mob: 9845355488

___________________________________________________________________________________________________
This e-mail and its attachments contain confidential information from HUAWEI, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it!
-------------------------------------------------------------------------------------------------------------------------------------------------------------------

-----Original Message-----
From: onap-seccom@... [mailto:onap-seccom@...] On Behalf Of Amy Zwarico
Sent: Thursday, October 31, 2019 8:36 PM
To: CLOSE, PIERRE <pierre.close@...>; Krzysztof Opasiak <k.opasiak@...>; Pawlak Paweł 3 - Korpo <Pawel.Pawlak3@...>; onap-seccom@...
Subject: Re: [Onap-seccom] revised code coverage requirement

Unfortunately there is urgency, and considerable pushback. I propose that we investigate the "new code tested" feature of SONAR in the Frankfurt release, create a clear definition of "new code", find a project that will test it, and, if all goes well, add the requirement that all new code contributions have 80% statement coverage in the Guilin release.

Thoughts from others?

-----Original Message-----
From: CLOSE, PIERRE
Sent: Thursday, October 31, 2019 6:14 AM
To: Krzysztof Opasiak <k.opasiak@...>; ZWARICO, AMY <az9121@...>; Pawlak Paweł 3 - Korpo <Pawel.Pawlak3@...>; onap-seccom@...
Subject: RE: [Onap-seccom] revised code coverage requirement

Hello Krzysztof,

Please find my comments inline.

Amy,

Is it possible to wait a little longer before agreeing on this requirement? Or do we have an urgency in deciding today -- in such a case, Krzysztof's comments lead me to delay the use of Sonar until we have a solid plan.

Best regards,
Pierre

-----Original Message-----
From: Krzysztof Opasiak <k.opasiak@...>
Sent: Wednesday, October 30, 2019 10:10 PM
To: Close, Pierre <pierre.close@...>; ZWARICO, AMY <az9121@...>; Pawlak Paweł 3 - Korpo <Pawel.Pawlak3@...>; onap-seccom@...
Subject: Re: [Onap-seccom] revised code coverage requirement

Hi,

On 30.10.2019 19:24, Pierre Close wrote:
Amy,

That’s a good question. But your explanation makes sense to me: when a
new Sonar scan is run it probably overrides the result of the previous
one, causing those counters to reset… Additionally, what is
considered “new code”? Is a configuration file change seen as new code?

I’ll try to look for an explanation from my colleagues and come back
to you when I got an answer (or maybe the LF knows?)
We would need a clear definition of the new code. Is a change in currently existing class a new code or only newly created classes? Is new function withing a class a new code?
[PC]: Agreed. That's what I was suggesting in my comment above.

Saying that we will just use the metric that the tool provides without any explanation to the community is not a good thing I believe
[PC]: I think we need to dig further into the capabilities of the tool and verify how we can use the metrics in a proper and meaningful way, otherwise it won't be of any use, as you mention.


In the meantime, since the feature seems available in the tool, we
might want to consider binding the “new code coverage” objective to
the tool – in other words, if the feature is usable and gives what we
expect, we could start monitoring it.

Would that make sense?
As far as I can see (please correct me if I'm wrong) the tool gives you only the current status of the code and diff since last run. What we would need is a dashboard as bitergia where you can select any arbitrary period of time and check the results
[PC]: I was under the impression that you could choose different versions to compare between... Now, your comments being valid, I propose that we take some time to further analyze what we can do, build (if possible) meaningful metrics and dashboards (as you mention), and then decide what route to take -- I know the timing is against us, but throwing all this into the community without further research is not constructive, so I support your statements. Would you agree with this approach?

Best regards,
--
Krzysztof Opasiak
Samsung R&D Institute Poland
Samsung Electronics

Krzysztof Opasiak
 

On 31.10.2019 13:36, ZWARICO, AMY wrote:
Unfortunately there is urgency, and considerable pushback. I propose that we investigate the "new code tested" feature of SONAR in the Frankfurt release, create a clear definition of "new code", find a project that will test it, and, if all goes well, add the requirement that all new code contributions have 80% statement coverage in the Guilin release.

Thoughts from others?
+1 from my side


-----Original Message-----
From: CLOSE, PIERRE
Sent: Thursday, October 31, 2019 6:14 AM
To: Krzysztof Opasiak <k.opasiak@...>; ZWARICO, AMY <az9121@...>; Pawlak Paweł 3 - Korpo <Pawel.Pawlak3@...>; onap-seccom@...
Subject: RE: [Onap-seccom] revised code coverage requirement

Hello Krzysztof,

Please find my comments inline.

Amy,

Is it possible to wait a little longer before agreeing on this requirement? Or do we have an urgency in deciding today -- in such a case, Krzysztof's comments lead me to delay the use of Sonar until we have a solid plan.

Best regards,
Pierre

-----Original Message-----
From: Krzysztof Opasiak <k.opasiak@...>
Sent: Wednesday, October 30, 2019 10:10 PM
To: Close, Pierre <pierre.close@...>; ZWARICO, AMY <az9121@...>; Pawlak Paweł 3 - Korpo <Pawel.Pawlak3@...>; onap-seccom@...
Subject: Re: [Onap-seccom] revised code coverage requirement

Hi,

On 30.10.2019 19:24, Pierre Close wrote:
Amy,

That’s a good question. But your explanation makes sense to me: when a
new Sonar scan is run it probably overrides the result of the previous
one, causing those counters to reset… Additionally, what is
considered “new code”? Is a configuration file change seen as new code?

I’ll try to look for an explanation from my colleagues and come back
to you when I got an answer (or maybe the LF knows?)
We would need a clear definition of the new code. Is a change in currently existing class a new code or only newly created classes? Is new function withing a class a new code?
[PC]: Agreed. That's what I was suggesting in my comment above.

Saying that we will just use the metric that the tool provides without any explanation to the community is not a good thing I believe
[PC]: I think we need to dig further into the capabilities of the tool and verify how we can use the metrics in a proper and meaningful way, otherwise it won't be of any use, as you mention.


In the meantime, since the feature seems available in the tool, we
might want to consider binding the “new code coverage” objective to
the tool – in other words, if the feature is usable and gives what we
expect, we could start monitoring it.

Would that make sense?
As far as I can see (please correct me if I'm wrong) the tool gives you only the current status of the code and diff since last run. What we would need is a dashboard as bitergia where you can select any arbitrary period of time and check the results
[PC]: I was under the impression that you could choose different versions to compare between... Now, your comments being valid, I propose that we take some time to further analyze what we can do, build (if possible) meaningful metrics and dashboards (as you mention), and then decide what route to take -- I know the timing is against us, but throwing all this into the community without further research is not constructive, so I support your statements. Would you agree with this approach?

Best regards,
--
Krzysztof Opasiak
Samsung R&D Institute Poland
Samsung Electronics
--
Krzysztof Opasiak
Samsung R&D Institute Poland
Samsung Electronics

Pierre Close
 

Amy,

I agree with the general plan, but I'm afraid that global 80% coverage could lead developers to build tests for the sake of only having more coverage, mostly because the amount of available resources for feature development not always allows to add/improve testing at the same time.

Colleagues suggested the following: contextualize the scope of coverage. By this, it is meant to:
- define, for each project, the core parts that need intensive testing (an API sensitive project might prioritize API testing so that all are covered, hardened, so that said APIs are robust)
- concentrate coverage on those areas, even if coverage is lower on other parts, so critical parts are better covered and tested.
This might improve OJSI resolution (and/or reduce findings), and better focuses the effort on testing based on available resources.
This would be even better if we could tell Sonar which parts of the code are the critical ones, but I don't know if that's feasible.

Thoughts are welcome

Best regards,
Pierre

-----Original Message-----
From: onap-seccom@... <onap-seccom@...> On Behalf Of Krzysztof Opasiak via Lists.Onap.Org
Sent: Monday, November 4, 2019 10:02 AM
To: ZWARICO, AMY <az9121@...>; Close, Pierre <pierre.close@...>; Pawlak Paweł 3 - Korpo <Pawel.Pawlak3@...>; onap-seccom@...
Cc: onap-seccom@...
Subject: Re: [Onap-seccom] revised code coverage requirement



On 31.10.2019 13:36, ZWARICO, AMY wrote:
Unfortunately there is urgency, and considerable pushback. I propose that we investigate the "new code tested" feature of SONAR in the Frankfurt release, create a clear definition of "new code", find a project that will test it, and, if all goes well, add the requirement that all new code contributions have 80% statement coverage in the Guilin release.

Thoughts from others?
+1 from my side


-----Original Message-----
From: CLOSE, PIERRE
Sent: Thursday, October 31, 2019 6:14 AM
To: Krzysztof Opasiak <k.opasiak@...>; ZWARICO, AMY
<az9121@...>; Pawlak Paweł 3 - Korpo <Pawel.Pawlak3@...>;
onap-seccom@...
Subject: RE: [Onap-seccom] revised code coverage requirement

Hello Krzysztof,

Please find my comments inline.

Amy,

Is it possible to wait a little longer before agreeing on this requirement? Or do we have an urgency in deciding today -- in such a case, Krzysztof's comments lead me to delay the use of Sonar until we have a solid plan.

Best regards,
Pierre

-----Original Message-----
From: Krzysztof Opasiak <k.opasiak@...>
Sent: Wednesday, October 30, 2019 10:10 PM
To: Close, Pierre <pierre.close@...>; ZWARICO, AMY
<az9121@...>; Pawlak Paweł 3 - Korpo <Pawel.Pawlak3@...>;
onap-seccom@...
Subject: Re: [Onap-seccom] revised code coverage requirement

Hi,

On 30.10.2019 19:24, Pierre Close wrote:
Amy,

That’s a good question. But your explanation makes sense to me: when
a new Sonar scan is run it probably overrides the result of the
previous one, causing those counters to reset… Additionally, what is
considered “new code”? Is a configuration file change seen as new code?

I’ll try to look for an explanation from my colleagues and come back
to you when I got an answer (or maybe the LF knows?)
We would need a clear definition of the new code. Is a change in currently existing class a new code or only newly created classes? Is new function withing a class a new code?
[PC]: Agreed. That's what I was suggesting in my comment above.

Saying that we will just use the metric that the tool provides without
any explanation to the community is not a good thing I believe
[PC]: I think we need to dig further into the capabilities of the tool and verify how we can use the metrics in a proper and meaningful way, otherwise it won't be of any use, as you mention.


In the meantime, since the feature seems available in the tool, we
might want to consider binding the “new code coverage” objective to
the tool – in other words, if the feature is usable and gives what we
expect, we could start monitoring it.

Would that make sense?
As far as I can see (please correct me if I'm wrong) the tool gives
you only the current status of the code and diff since last run. What
we would need is a dashboard as bitergia where you can select any
arbitrary period of time and check the results
[PC]: I was under the impression that you could choose different versions to compare between... Now, your comments being valid, I propose that we take some time to further analyze what we can do, build (if possible) meaningful metrics and dashboards (as you mention), and then decide what route to take -- I know the timing is against us, but throwing all this into the community without further research is not constructive, so I support your statements. Would you agree with this approach?

Best regards,
--
Krzysztof Opasiak
Samsung R&D Institute Poland
Samsung Electronics
--
Krzysztof Opasiak
Samsung R&D Institute Poland
Samsung Electronics

Pawel Pawlak
 

+1

Paweł Pawlak

ONAP SECCOM Chair
Leader in IT & Network Convergent Operations
FT/TGI/OLN/TOP/OST

Orange Polska S.A.
Corporate Services Agency

Obrzeżna 7, 02-691 Warszawa
tel. +48 22 699 52 17
fax +48 22 857 99 86
tel. mob. +48 501 501 030

   Czy musisz drukować tę wiadomość? Pomyśl o środowisku.
__________________________________________________________________
Treść tej wiadomości jest własnością Orange Polska i zawiera informacje stanowiące tajemnicę przedsiębiorstwa Orange Polska. Jeżeli nie jesteście Państwo jej adresatem, bądź otrzymaliście ją przez pomyłkę, prosimy o powiadomienie o tym nadawcy oraz trwałe jej usunięcie. Orange Polska Spółka Akcyjna z siedzibą i adresem w Warszawie (02-326) przy Al. Jerozolimskich 160, wpisana do Rejestru Przedsiębiorców prowadzonego przez Sąd Rejonowy dla m.st. Warszawy XII Wydział Gospodarczy Krajowego Rejestru Sądowego pod numerem 0000010681; REGON 012100784, NIP 526-02-50-995; z pokrytym w całości kapitałem zakładowym wynoszącym 3.937.072.437 złotych.

-----Original Message-----
From: Close, Pierre [mailto:pierre.close@...]
Sent: Monday, November 4, 2019 12:32 PM
To: k.opasiak@...; ZWARICO, AMY; Pawlak Paweł 3 - Korpo; onap-seccom@...
Subject: RE: [Onap-seccom] revised code coverage requirement

Amy,

I agree with the general plan, but I'm afraid that global 80% coverage could lead developers to build tests for the sake of only having more coverage, mostly because the amount of available resources for feature development not always allows to add/improve testing at the same time.

Colleagues suggested the following: contextualize the scope of coverage. By this, it is meant to:
- define, for each project, the core parts that need intensive testing (an API sensitive project might prioritize API testing so that all are covered, hardened, so that said APIs are robust)
- concentrate coverage on those areas, even if coverage is lower on other parts, so critical parts are better covered and tested.
This might improve OJSI resolution (and/or reduce findings), and better focuses the effort on testing based on available resources.
This would be even better if we could tell Sonar which parts of the code are the critical ones, but I don't know if that's feasible.

Thoughts are welcome

Best regards,
Pierre

-----Original Message-----
From: onap-seccom@... <onap-seccom@...> On Behalf Of Krzysztof Opasiak via Lists.Onap.Org
Sent: Monday, November 4, 2019 10:02 AM
To: ZWARICO, AMY <az9121@...>; Close, Pierre <pierre.close@...>; Pawlak Paweł 3 - Korpo <Pawel.Pawlak3@...>; onap-seccom@...
Cc: onap-seccom@...
Subject: Re: [Onap-seccom] revised code coverage requirement



On 31.10.2019 13:36, ZWARICO, AMY wrote:
Unfortunately there is urgency, and considerable pushback. I propose that we investigate the "new code tested" feature of SONAR in the Frankfurt release, create a clear definition of "new code", find a project that will test it, and, if all goes well, add the requirement that all new code contributions have 80% statement coverage in the Guilin release.

Thoughts from others?
+1 from my side


-----Original Message-----
From: CLOSE, PIERRE
Sent: Thursday, October 31, 2019 6:14 AM
To: Krzysztof Opasiak <k.opasiak@...>; ZWARICO, AMY
<az9121@...>; Pawlak Paweł 3 - Korpo <Pawel.Pawlak3@...>;
onap-seccom@...
Subject: RE: [Onap-seccom] revised code coverage requirement

Hello Krzysztof,

Please find my comments inline.

Amy,

Is it possible to wait a little longer before agreeing on this requirement? Or do we have an urgency in deciding today -- in such a case, Krzysztof's comments lead me to delay the use of Sonar until we have a solid plan.

Best regards,
Pierre

-----Original Message-----
From: Krzysztof Opasiak <k.opasiak@...>
Sent: Wednesday, October 30, 2019 10:10 PM
To: Close, Pierre <pierre.close@...>; ZWARICO, AMY
<az9121@...>; Pawlak Paweł 3 - Korpo <Pawel.Pawlak3@...>;
onap-seccom@...
Subject: Re: [Onap-seccom] revised code coverage requirement

Hi,

On 30.10.2019 19:24, Pierre Close wrote:
Amy,

That’s a good question. But your explanation makes sense to me: when
a new Sonar scan is run it probably overrides the result of the
previous one, causing those counters to reset… Additionally, what is
considered “new code”? Is a configuration file change seen as new code?

I’ll try to look for an explanation from my colleagues and come back
to you when I got an answer (or maybe the LF knows?)
We would need a clear definition of the new code. Is a change in currently existing class a new code or only newly created classes? Is new function withing a class a new code?
[PC]: Agreed. That's what I was suggesting in my comment above.

Saying that we will just use the metric that the tool provides without
any explanation to the community is not a good thing I believe
[PC]: I think we need to dig further into the capabilities of the tool and verify how we can use the metrics in a proper and meaningful way, otherwise it won't be of any use, as you mention.


In the meantime, since the feature seems available in the tool, we
might want to consider binding the “new code coverage” objective to
the tool – in other words, if the feature is usable and gives what we
expect, we could start monitoring it.

Would that make sense?
As far as I can see (please correct me if I'm wrong) the tool gives
you only the current status of the code and diff since last run. What
we would need is a dashboard as bitergia where you can select any
arbitrary period of time and check the results
[PC]: I was under the impression that you could choose different versions to compare between... Now, your comments being valid, I propose that we take some time to further analyze what we can do, build (if possible) meaningful metrics and dashboards (as you mention), and then decide what route to take -- I know the timing is against us, but throwing all this into the community without further research is not constructive, so I support your statements. Would you agree with this approach?

Best regards,
--
Krzysztof Opasiak
Samsung R&D Institute Poland
Samsung Electronics
--
Krzysztof Opasiak
Samsung R&D Institute Poland
Samsung Electronics