Topics

Problem regenerating schema for AAI v14 #aai

Keong Lim
 

Hi all,

I have updated the AAI v14 OXM file with new schema changes, as per https://wiki.onap.org/display/DW/AAI-CCVPN+Schema+Proposal+for+Casablanca+Release.

I have followed the instructions in the tutorial: https://wiki.onap.org/pages/viewpage.action?pageId=10783023
e.g. steps 7 and 20:

===
Rebuild aai-common first:
$ cd ~/LF/AAI/aai-common
$ mvn clean install
Should result in BUILD SUCCESS
etc
===

I can find the generated files "aai_swagger_v14.html", "aai_swagger_v14.yaml", which are new and expanded compared to their v13 versions.

However, the "aai_schema_v14.xsd" is not changed at all. There is a placeholder file already committed to src/main/resources/aai_schema directory, with the contents as a copy of the v13 version. It appears this file was simply copied to the target/classes/aai_schema directory, rather than being regenerated from the new v14 OXM file.

Why is the v14 XSD file not automatically regenerated by this mvn command?
What command is required to regenerate the XSD file?

The pom.xml file in aai-core appear to have <execution> tags for <id>autoGenerateYaml</id> and <id>autoGenerateHtml</id>, but nothing for "auto generate XSD". Is this correct or broken?


Thanks,
Keong


Customer Experience and Platform Integration R&D Dept
--
Keong Lim, Huawei Technologies Co. Ltd (keong.lim@...)
Ground Floor, Suite 1, 5 Lakeside Drive, BURWOOD EAST VIC 3151 AUSTRALIA
--
"If ye love wealth better than liberty, the tranquillity of servitude than the
animating contest of freedom-go from us in peace. We ask not your counsels
or arms. Crouch down and lick the hands which feed you. May your chains sit
lightly upon you, and may posterity forget that ye were our countrymen!"
- Samuel Adams

Venkata Harish K Kajur
 

Hi Keong,

The html and yaml gets automatically generated when you run install but the xsd was never part of the autogenerate maven profile.
For Beijing and earlier release, we never auto generated it. In order to run the generation of the xsd, here is the command:

cd ~/LF/aai/aai-common/aai-core/
mvn -PgenerateXsd install -DskipTests -Dgendoc.version=v14

I have actually noticed that recently within the internal release and realized that it makes sense for xsd to be also auto generated.
I am not sure why it was left out from the autogenerate profile but when developing the model driven feature, I ensured that the auto generate also does xsd generation.
Once that feature gets delivered to Casablanca, there wouldn't be a need to run the profile manually.

Thanks,
Harish

-----Original Message-----
From: onap-discuss@... <onap-discuss@...> On Behalf Of Keong Lim
Sent: Monday, July 30, 2018 1:18 AM
To: onap-discuss@...
Subject: [onap-discuss] Problem regenerating schema for AAI v14 #aai
Sensitivity: Confidential

Hi all,

I have updated the AAI v14 OXM file with new schema changes, as per https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.onap.org_display_DW_AAI-2DCCVPN-2BSchema-2BProposal-2Bfor-2BCasablanca-2BRelease&d=DwIFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=f44eG3iZaja2ozEA2yRZnQ&m=wGqk6dbredJelbUbsEpMZwAY7EZaQ2IW6NPX6da4Rdk&s=I5IYHqPmi6w9XAFjJu9BmOCMuFWZ6GONpoOzMWG0hCQ&e=.

I have followed the instructions in the tutorial: https://urldefense.proofpoint.com/v2/url?u=https-3A__wiki.onap.org_pages_viewpage.action-3FpageId-3D10783023&d=DwIFAg&c=LFYZ-o9_HUMeMTSQicvjIg&r=f44eG3iZaja2ozEA2yRZnQ&m=wGqk6dbredJelbUbsEpMZwAY7EZaQ2IW6NPX6da4Rdk&s=2QdJLPrk8KGnZTXeCxUdx2nDm4yfOZoVSHCE161etzo&e=
e.g. steps 7 and 20:

===
Rebuild aai-common first:
$ cd ~/LF/AAI/aai-common
$ mvn clean install
Should result in BUILD SUCCESS
etc
===

I can find the generated files "aai_swagger_v14.html", "aai_swagger_v14.yaml", which are new and expanded compared to their v13 versions.

However, the "aai_schema_v14.xsd" is not changed at all. There is a placeholder file already committed to src/main/resources/aai_schema directory, with the contents as a copy of the v13 version. It appears this file was simply copied to the target/classes/aai_schema directory, rather than being regenerated from the new v14 OXM file.

Why is the v14 XSD file not automatically regenerated by this mvn command?
What command is required to regenerate the XSD file?

The pom.xml file in aai-core appear to have <execution> tags for <id>autoGenerateYaml</id> and <id>autoGenerateHtml</id>, but nothing for "auto generate XSD". Is this correct or broken?


Thanks,
Keong


Customer Experience and Platform Integration R&D Dept
--
Keong Lim, Huawei Technologies Co. Ltd (keong.lim@...) Ground Floor, Suite 1, 5 Lakeside Drive, BURWOOD EAST VIC 3151 AUSTRALIA
--
"If ye love wealth better than liberty, the tranquillity of servitude than the
animating contest of freedom-go from us in peace. We ask not your counsels
or arms. Crouch down and lick the hands which feed you. May your chains sit
lightly upon you, and may posterity forget that ye were our countrymen!"
- Samuel Adams

Keong Lim
 

Thanks Harish!

After cleaning up the mess from my working directory, your command successfully created the new XSD file.

Updated the wiki https://wiki.onap.org/pages/viewpage.action?pageId=10783023 with a note and a link to your message.


Keong

-----Original Message-----
From: KAJUR, HARISH V [mailto:vk250x@...]
Sent: Monday, 30 July 2018 16:01
To: onap-discuss@...; Keong Lim <Keong.Lim@...>
Subject: RE: Problem regenerating schema for AAI v14 #aai
Sensitivity: Confidential

Hi Keong,

The html and yaml gets automatically generated when you run install but the xsd was never part of the autogenerate maven profile.
For Beijing and earlier release, we never auto generated it. In order to run the generation of the xsd, here is the command:

cd ~/LF/aai/aai-common/aai-core/
mvn -PgenerateXsd install -DskipTests -Dgendoc.version=v14

I have actually noticed that recently within the internal release and realized that it makes sense for xsd to be also auto generated.
I am not sure why it was left out from the autogenerate profile but when developing the model driven feature, I ensured that the auto generate also does xsd generation.
Once that feature gets delivered to Casablanca, there wouldn't be a need to run the profile manually.

Thanks,
Harish

Keong Lim
 

Hi Harish,

My workspace has the following directory structure:

- aai/
- aai/aai-common
- aai/aai-common/aai-schema
- aai/aai-common/aai-core
- etc

Sometimes after running the command to regenerate the XSD files:

cd ~/LF/aai/aai-common/aai-core/
mvn -PgenerateXsd install -DskipTests -Dgendoc.version=v14
I see that there is a new sub-directory created, i.e.

- aai/aai-schema

This new directory contains all freshly generated XSD files for all versions (despite the gendoc.version being for one specific version).
Are these files being generated to the wrong directory?

If they are intended to overwrite the aai-schema files, then it should go into "aai/aai-common/aai-schema".
If they are simply the build products of aai-core, then it should go into "aai/aai-common/aai-core/target".
As it happens now, it is polluting the workspace with extra files that need to be manually cleaned up.


Regarding the build order in aai-common, maven reports the order is:

- aai-aai-common
- aai-annotations
- aai-schema
- aai-schema-ingest
- aai-core
- aai-auth
- aai-utils

When the aai-schema sub-component is built, it appears to copy the schema files from "aai-schema/src/" to "aai-schema/target/" subdirectory.
Later, when the aai-core sub-component is built, the auto-generation overwrites the "aai-schema/src/" sub-directory.
A subsequent build in aai-schema then produces a different result from the first time.

Is it appropriate for a later component to be rewriting the src of an earlier component?
This makes for a not-very-reproducible build.

Could the "generateXSD", "generateHtml" and "generateYaml" be done in "aai-schema" itself?
Alternatively, could the auto-generated files be stored in aai-core, separately from the hand-written files (i.e. the OXM)?


Thanks,
Keong

-----Original Message-----
From: KAJUR, HARISH V [mailto:vk250x@...]
Sent: Monday, 30 July 2018 16:01
To: onap-discuss@...; Keong Lim <Keong.Lim@...>
Subject: RE: Problem regenerating schema for AAI v14 #aai
Sensitivity: Confidential

Hi Keong,

The html and yaml gets automatically generated when you run install but the xsd was never part of the autogenerate maven profile.
For Beijing and earlier release, we never auto generated it. In order to run the generation of the xsd, here is the command:

cd ~/LF/aai/aai-common/aai-core/
mvn -PgenerateXsd install -DskipTests -Dgendoc.version=v14

I have actually noticed that recently within the internal release and realized that it makes sense for xsd to be also auto generated.
I am not sure why it was left out from the autogenerate profile but when developing the model driven feature, I ensured that the auto generate also does xsd generation.
Once that feature gets delivered to Casablanca, there wouldn't be a need to run the profile manually.

Thanks,
Harish

Venkata Harish K Kajur
 

Hi Keong,

I also noticed this and as I mentioned earlier, I also fixed that internally as part of model driven feature and will be taking to Casablanca release.
This was one of the reasons why the generateXsd step was a manual process.
I also noticed that aai-schema shouldn't be a dependency to aai-core so it was removed.
So aai-core gets built first and the generation of schema gets done and then aai-schema will be build last.
I was planning to move the schema generation steps to aai-schema but there was some tight coupling that would take too long to fix in this release.
I have plans on making changes in the next release to remove that coupling and move the schema generation stuff to aai-schema.

Thanks,
Harish

-----Original Message-----
From: Keong Lim <Keong.Lim@...>
Sent: Tuesday, July 31, 2018 4:43 AM
To: KAJUR, HARISH V <vk250x@...>; onap-discuss@...
Subject: RE: Problem regenerating schema for AAI v14 #aai
Sensitivity: Confidential

Hi Harish,

My workspace has the following directory structure:

- aai/
- aai/aai-common
- aai/aai-common/aai-schema
- aai/aai-common/aai-core
- etc

Sometimes after running the command to regenerate the XSD files:

cd ~/LF/aai/aai-common/aai-core/
mvn -PgenerateXsd install -DskipTests -Dgendoc.version=v14
I see that there is a new sub-directory created, i.e.

- aai/aai-schema

This new directory contains all freshly generated XSD files for all versions (despite the gendoc.version being for one specific version).
Are these files being generated to the wrong directory?

If they are intended to overwrite the aai-schema files, then it should go into "aai/aai-common/aai-schema".
If they are simply the build products of aai-core, then it should go into "aai/aai-common/aai-core/target".
As it happens now, it is polluting the workspace with extra files that need to be manually cleaned up.


Regarding the build order in aai-common, maven reports the order is:

- aai-aai-common
- aai-annotations
- aai-schema
- aai-schema-ingest
- aai-core
- aai-auth
- aai-utils

When the aai-schema sub-component is built, it appears to copy the schema files from "aai-schema/src/" to "aai-schema/target/" subdirectory.
Later, when the aai-core sub-component is built, the auto-generation overwrites the "aai-schema/src/" sub-directory.
A subsequent build in aai-schema then produces a different result from the first time.

Is it appropriate for a later component to be rewriting the src of an earlier component?
This makes for a not-very-reproducible build.

Could the "generateXSD", "generateHtml" and "generateYaml" be done in "aai-schema" itself?
Alternatively, could the auto-generated files be stored in aai-core, separately from the hand-written files (i.e. the OXM)?


Thanks,
Keong

-----Original Message-----
From: KAJUR, HARISH V [mailto:vk250x@...]
Sent: Monday, 30 July 2018 16:01
To: onap-discuss@...; Keong Lim <Keong.Lim@...>
Subject: RE: Problem regenerating schema for AAI v14 #aai
Sensitivity: Confidential

Hi Keong,

The html and yaml gets automatically generated when you run install but the xsd was never part of the autogenerate maven profile.
For Beijing and earlier release, we never auto generated it. In order to run the generation of the xsd, here is the command:

cd ~/LF/aai/aai-common/aai-core/
mvn -PgenerateXsd install -DskipTests -Dgendoc.version=v14

I have actually noticed that recently within the internal release and realized that it makes sense for xsd to be also auto generated.
I am not sure why it was left out from the autogenerate profile but when developing the model driven feature, I ensured that the auto generate also does xsd generation.
Once that feature gets delivered to Casablanca, there wouldn't be a need to run the profile manually.

Thanks,
Harish

Keong Lim
 

Hi Harish,

with the merge of https://gerrit.onap.org/r/#/c/60045/ can you please confirm that the build order of aai-common should now be:

- aai-aai-common
- aai-schema-ingest
- aai-annotations
- aai-core
- aai-schema
- aai-auth
- aai-utils

and that all of the XSD/HTML/YAML are automatically regenerated with this command:

cd ~/LF/aai/aai-common/
mvn clean install -DskipTests


Also, since the HTML/YAML files are no longer checked into the source repo, the generated documentation can be browsed from the JAR files published in nexus repo like so:

https://nexus.onap.org/service/local/repositories/releases/archive/org/onap/aai/aai-common/aai-schema/1.2.4/aai-schema-1.2.4.jar/!/aai_swagger_html/aai_swagger_v13.html

The nexus repo can only be browsed like this after logging in.


Thanks,
Keong

Keong Lim
 

Hi Harish,

I tried to regenerate the AAI XSD/HTML/YAML files using the newly merged code and it seems to be producing different results from the checked-in XSD:

====
@@ -16,26 +16,26 @@ xmlns:jaxb="http://java.sun.com/xml/ns/jaxb"
<xs:element name="inventory-item">
<xs:complexType>
<xs:sequence>
<xs:element name="inventory-item-type" type="xs:string" minOccurs="0"/>
<xs:element name="inventory-item-link" type="xs:string" minOccurs="0"/>
- <xs:element ref="tns:inventory-item-data" minOccurs="0" maxOccurs="unbounded"/>
- <xs:element ref="tns:tagged-inventory-item-list" minOccurs="0" maxOccurs="unbounded"/>
+ <xs:element ref="tns:inventory-item-data" minOccurs="0" maxOccurs="5000"/>
+ <xs:element ref="tns:tagged-inventory-item-list" minOccurs="0" maxOccurs="5000"/>
</xs:sequence>
</xs:complexType>
</xs:element>
====

Can you please confirm that the change from "unbounded" to "5000" is intentional?
Is this new maxOccurs limit going to affect existing ONAP systems and databases?
Is there a migration plan for existing database that exceed 5000 items?
Is there a configuration setting or growth plan for going beyond 5000 items?


Thanks,
Keong

Keong Lim
 

Hi Harish,

In https://gerrit.onap.org/r/#/c/60465/ the tests are picking up some new errors post-merge related to "PrivateEdgeIntegrationTest":

=====
07:29:29 Tests run: 3, Failures: 0, Errors: 3, Skipped: 0, Time elapsed: 0.008 sec <<< FAILURE!
07:29:29 getCachedInstanceConnectionName(org.onap.aai.dbmap.AAIGraphTest) Time elapsed: 0.003 sec <<< ERROR!
07:29:29 java.lang.NoClassDefFoundError: Could not initialize class org.onap.aai.dbmap.AAIGraph$Helper
...
07:29:49 Tests in error:
07:29:49 testPutWithGenericVnfToExistingModelAndUpdateWithNewModelInfoThatDoesntExistAndCheckIfReturnsNotFoundAndOldEdgeShouldNotBeDeleted[QueryStyle.TRAVERSAL Version.v10](org.onap.aai.rest.PrivateEdgeIntegrationTest)
07:29:49 testPutWithGenericVnfToExistingModelAndUpdateWithNewModelInfoThatDoesntExistAndCheckIfReturnsNotFoundAndOldEdgeShouldNotBeDeleted[QueryStyle.TRAVERSAL Version.v10](org.onap.aai.rest.PrivateEdgeIntegrationTest): Could not initialize class org.onap.aai.dbmap.AAIGraph$Helper
=====

Not sure why my new schema element has triggered this failure, as the specific test cases failing do not appear to be related to this new schema element.
Does it need to have a "private edge" property introduced to it?
What are the guidelines for this "private edge" property?

Thanks,
Keong

Venkata Harish K Kajur
 

Hi Keong,

Yes, with the latest merge, this is the intended order of aai-common.
If you want to skip the schema generation for some reason, you need to run:

mvn clean install -DskipTests -Daai.generate.schema=false

Also, you don't need to login to nexus to directly view this page:

https://nexus.onap.org/service/local/repositories/releases/archive/org/onap/aai/aai-common/aai-schema/1.2.4/aai-schema-1.2.4.jar/!/aai_swagger_html/aai_swagger_v13.html

You would need to login to get to the top level page.
For every release, this link should be updated the wiki.
There seems to be a size limit on the files being uploaded so this is the only way to do it now.

Thanks,
Harish

-----Original Message-----
From: onap-discuss@... <onap-discuss@...> On Behalf Of Keong Lim
Sent: Monday, August 13, 2018 9:49 PM
To: onap-discuss@...
Subject: Re: [onap-discuss] Problem regenerating schema for AAI v14 #aai

Hi Harish,

with the merge of https://urldefense.proofpoint.com/v2/url?u=https-3A__gerrit.onap.org_r_-23_c_60045_&d=DwIFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=f44eG3iZaja2ozEA2yRZnQ&m=s7oKqLKPYJC-WAnYJMSBDikyfE8GYLIhsONnBl8OMF4&s=sKEdz32K5JgCdlPaSpF5k9Y9UMkb0CHKqW-pQ3j14LI&e= can you please confirm that the build order of aai-common should now be:

- aai-aai-common
- aai-schema-ingest
- aai-annotations
- aai-core
- aai-schema
- aai-auth
- aai-utils

and that all of the XSD/HTML/YAML are automatically regenerated with this command:

cd ~/LF/aai/aai-common/
mvn clean install -DskipTests


Also, since the HTML/YAML files are no longer checked into the source repo, the generated documentation can be browsed from the JAR files published in nexus repo like so:

https://urldefense.proofpoint.com/v2/url?u=https-3A__nexus.onap.org_service_local_repositories_releases_archive_org_onap_aai_aai-2Dcommon_aai-2Dschema_1.2.4_aai-2Dschema-2D1.2.4.jar_-21_aai-5Fswagger-5Fhtml_aai-5Fswagger-5Fv13.html&d=DwIFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=f44eG3iZaja2ozEA2yRZnQ&m=s7oKqLKPYJC-WAnYJMSBDikyfE8GYLIhsONnBl8OMF4&s=dXU0MK1kW3pu69OdFa-TTXEaw3WjfI5OvDvoT1aQ228&e=

The nexus repo can only be browsed like this after logging in.


Thanks,
Keong

Keong Lim
 

Thanks to Harish, it appears the test failures showing:

07:29:29 java.lang.NoClassDefFoundError: Could not initialize class org.onap.aai.dbmap.AAIGraph$Helper

have nothing to do with "privateEdge" property but with needing to use camelCase for java attributes instead of all-caps for the acronyms.

I will rename the java attributes to use camelCase.

Shouldn't this be detected as part of the schema generation phase, rather than the unit test phase?

The command given was using "-DskipTests" so this never showed up before. In the near future, the AAI schema will be dynamically modified at run-time, so how will that scenario detect these errors?

Where in the error logs did you diagnose this particular problem? it was not obvious to me from what I could see. It only complained that the Helper could not be found and all that does is instantiate a graph object.

Can the error messages be improved? It is giving false negatives on too many unrelated test cases.

Thanks,
Keong