INTEGRATION

Integration Jakarta Release Notes

Integration Releases

Repository

Revision

demo

a112ebd160d3acc20e6766d8550d757a0581797d

integration

fd5b23e8ff35aff3eacf0d20b6910b0257bcced0

integration/csit

8f88f01b9fbbdd2af363a747ecf43b9de15a9b6e

integration/data-provider

a5d76c710471e8c6819a8d5963ee73349a48013c

integration/docker/onap-java11

1a12b2560c0b168c64e1b38a25d8284904b8ef01

integration/docker/onap-python

5fb8770183d2bef14d47124b100dc4dec4ee1999

integration/ietf-actn-tools

2e7a9a57f2610268002badc23748d483b653042e

integration/seccom

687008f10b08ded20cb222d732523fea270b9c8e

integration/simulators/5G-core-nf-simulator

7ee6ee299f53e226663645010b9467860a656c07

integration/simulators/A1-policy-enforcement-simulator

a1408cfc73163ed209049b47098d86e2f94a538a

integration/simulators/core-nssmf-simulator

c3bbe92494ba80d10628dbe102eddaadaac9d421

integration/simulators/nf-simulator

745a7b4c9bc5606a342f5fcb250b606836f73627

integration/simulators/nf-simulator/avcn-manager

0e280f888bcc42909eee1aefe65a5a8b79a7477b

integration/simulators/nf-simulator/netconf-server

c7bd72db1a290848fdad94c2b3688c790b4e778f

integration/simulators/nf-simulator/pm-https-server

98b44a558e1993f0ceafbf22b250ba0975042229

integration/simulators/nf-simulator/ves-client

4ac8f0ad9b5f627697f2d3a67bb47c3315737528

integration/simulators/pnf-simulator

c942281da3cc77db1aee06ef69398fd57b970f31

integration/simulators/ran-nssmf-simulator

220120e53977d6df4eda174dca710320a2bcba8f

integration/simulators/ran-simulator

b395e82cedb969d392fd4a1a2b13cfa8cf64cce7

integration/usecases/A1-policy-enforcement

f29388582dc68a8f1cfdff4aa65f3839e9e1aaac

integration/usecases/A1-policy-enforcement-r-apps

3fc8bbb9080cd637318fb920056f5a51014e5c1f

integration/xtesting

1b5b1a3dc73431c671971a22abd875cd292c1692

oparent

f95bafef87324b634305e1b98cf17838aa998d26

testsuite

898230f3ead89797c4e4f774ccef38f7c23c0b7e

testsuite/cds

b1f769130a6abe8ac67f42fe74b834eb25f33c07

testsuite/cds-mock-odl

4784fbe07e25a6894d6dfccf76f78e8710f20f11

testsuite/cds-mock-server

7db71adaf139e54f2186cfd19d468f5a1123835d

testsuite/cds-mock-ssh

a43ce8950dcc36363c406b1cc4043dc7d623c9f4

testsuite/oom

5809297b562203922c13c843c085049148d78edb

testsuite/python-testing-utils

17d603cf6655198fa61873e4d715889998a92b69

testsuite/pythonsdk-tests

81584955687e053d0697a63131b026fa0eda47c7

testsuite/robot-utils

a67403bdb58cfdfc272801d9c4448a22bd40046d

Important

  • Creation of an Jakarta Daily CI/CD chain

  • Creation of Java and Python baseline images for Jakarta

  • Update of Seccom waivers and version recommendations

  • Adapt robot tests to DCAE project changes

  • New tests (basic_cnf_macro)

  • New repositories (see dedicated section)

  • Release ONAP Data Provider tool

  • Automate repositories INFO.yaml updates

  • Bug fixes

  • ONAP tests library gating tests

Quick Links:

Code changes

Integration Repo

Release Date

2021-10-14

Version: 10.0.0 (aka Jakarta)

Onaptests (pythonsdk_tests)

Main changes:

pythonsdk_tests Changes

Issue-ID

Description

SDC-3949

Log exception stack trace on test failure

TEST-384

[TEST] Change of the k8sregion name

TEST-382

[TEST] Change of the namespace for basic_cnf_macro

INT-2078

Use Python ONAP SDK 9.4.0

TEST-354

[TEST] PNF macro nad CDS resource resoulution tests are failing on retry

INT-2078

[PYTHON] Improve requirement versioning

TEST-380

[TEST] Add cds resource resolution missing cleanup method call

TEST-376

[TEST] Basic CNF macro

TEST-356

[TEST] Do not create simulators in “onap” namespace

TEST-369

[TEST] Fix cds-resource-resolution test

TEST-368

[TEST] Do not show “No execution start time…” error log message for skipped substeps

TEST-358

[TEST] Added support for multi-vnf macro instantiation

INT-1994

[INTEGRATION] Use the newest onapsdk (9.0.0)

Robot (Testsuite)

Version: 1.10.0

Main changes:

Testsuite Changes

Issue-ID

Description

INT-1895

[ROBOT] Change performance measurements topic for 5gbulkpm

INT-1895

[ROBOT] Add timout flag for helm unintsll operation

INT-1895

[ROBOT] Change pm mapper readiness check to HTTP

INT-2012

[Healthcheck] Remove clamp Healthcheck

NT-1895

[ROBOT] Adopt DCAEMOD test cases to helm based deployment

INT-2093

[DCAE] Deprecate DCAE basic healthcheck

INT-1895

[ROBOT] Fix failing CMPv2 and 5gbulkpm test cases in CI

TEST-383

[INTEGRATION] Fix failing HVVES test case in Jakarta release

OOM-2921

[INT] Remove Portal robot test from full and medium test suite

INT-1895

[ROBOT] ADD CMPv2 test cases that use helm based components

POLICY-3887

Add policy cleanup if policy is existing before policy health check execution

INT-1895

[ROBOT] FIX BULKPM test cases for Jakarta release

SO-3829

[HEALTCHECK] remove mariadbgalera connectivity tests from core

SO-3829

[SO] MariaDB Galera connectivity with SO Issue-ID:SO-3829

INT-1895

[ROBOT] ADD HTTPS based BULKPM test cases that use helm based components

POLICY-3815

[POLICY] Change API and PAP usernames

INT-1895

[ROBOT] ADD BULKPM test cases that use helm based component services

INT-2006

[TIMEOUT] Extend healthcheck timeout value

O-Parent

Version: 3.3.1
Oparent Changes

Issue-ID

Description

CPS-475

Enable @surpressWarnings for Checkstyle

Demo Artifacts (Heat Templates)

Version: 1.10.0
Demo Changes

Issue-ID

Description

INT-2094

[APACHE] Add Apache CNF use case files

INT-2069

Make Network Slicing usecase more user friendly

INT-1960

[vFW CNF CDS] Fix issue with multiple tenants creation for k8s region

INT-1960

[vFW CNF CDS] vf_module and vnf name generation improvment in CBA

The demo artifacts are pushed to https://nexus.onap.org/content/repositories/releases/org/onap/demo/vnf

Use Cases and Requirements

See dedicated Jakarta Use Cases and requirements page

Maturity Testing Notes

Maturity testing page

Open JIRAs/Known issues

Integration

Integration JIRA page

Testsuite

Testsuite JIRA page

Jakarta Use Cases and Requirements

Description

This session includes use cases and functional requirements which have been officially verified in Jakarta release by the ONAP community.

For each use case or functional requirement, you can find contact names and a link to the associated documentation.

This documentation deals with

  1. What has been implemented

  2. Step by step instructions to deploy and execute the tests, including the links to download the related assets and resources

  3. Known issues and workarounds

Jakarta Use Cases

Description

This session includes use cases and functional requirements which have been officially verified in Jakarta release by the ONAP community.

For each use case or functional requirement, you can find contact names and a link to the associated documentation.

This documentation deals with

  1. What has been implemented

  2. Step by step instructions to deploy and execute the tests, including the links to download the related assets and resources

  3. Known issues and workarounds

Use cases
Jakarta Official Use Cases

Ref

Summary

Link

Contacts

REQ-440

E2E Network Slicing

official doc

Kevin Tang

REQ-429

5G OOF SON

official doc

    1. Shankaranarayanan

REQ-459

CCVPN-Transport Slicing

official doc

Henry Yu

Automated Use Cases

These use cases have been run on the Daily Jakarta CI chains and are used to validate the integration of any new dockers in OOM. New tests are indicated in bold.

Infrastructure Healthcheck Tests

Tests

Description

Code

Comments

onap-helm

Verify Helm chart status, the test has been updated to take into account Helm3

python module

onap-k8s

Check common resources of the ONAP Kubernetes namespace

python module

kubernetes python library

nodeport_check_certs

This test list the nodeports and tries to get SSL information to evaluate the validity of the certificates (expiration and issuer) used on the nodeports

python module

pyopenssl, kubernetes python libraries

Healthcheck Tests

Tests

Description

Code

Comments

core

Robot healthcheck tests of the core components (AA&I, DMAAP, Portal, SDC, SDNC, SO)

robot tests

robotframework

full

Robot healthcheck tests for all the components, holmes healthcheck have been reintroduced

robot tests

robotframework

healthdist

Check the onboarding and distribution of the vFW

robot tests

robotframework

postinstall

Check dmaap and AA&I Design model DB tests

robot tests

robotframework

ves-collector (new)

Suite for checking handling events by VES Collector

code

robotframework

hv-ves

HV-VES ‘Sunny Scenario’ Robot Framework test - message is sent to the collector and Kafka topic is checked if the message has been published. Content is decoded and checked.

code

robotframework

basic_onboard

onboard a model, subset of most of the other basic_* tests, created to perform stability testing

code

onap_pythonsdk, pythonsdk-tests

dcaemod

Check new DCAEmod

robot tests

robotframework

cps-healthcheck

Call liveness and readiness probes of the CPS module

robot tests

robotframework

Smoke Tests

Tests

Description

Code

Comments

basic_vm

Onboard, distribute and instantiate an Openstack VM using à la carte BPMN, replaced the former basic_vm test

code

onap_pythonsdk, pythonsdk-tests

basic_network

Onboard, distribute and instantiate a Neutron network

code

onap_pythonsdk, pythonsdk-tests

basic_cnf

Onboard (new), distribute and instantiate a Kubernetes pods

code

onap_pythonsdk, pythonsdk-tests

5gbulkpm

5G Bulk PM Usecase functionality. The test has been significantly enhanced in Honolulu

code

robotframework

pnf-registrate

Executes the PNF registration test cases including setup and teardown

code

robotframework

cmpv2

CMPv2 Usecase functionality

code

robotframework

basic_vm_macro

Instantiate a VM using macro bpmn

code

onap_pythonsdk, pythonsdk-tests

pnf_macro

Run PNF simulator, onboard, distribute and instantiate service including PNF

code

onap_pythonsdk, pythonsdk-tests

cds_resource_resolution

Upload blueprint file into CDS, execute test workflow and check results

code

onap_pythonsdk, pythonsdk-tests

basic_cnf_macro

Onboard (new), distribute and instantiate a Kubernetes pods using SO’s macro flow

code

onap_pythonsdk, pythonsdk-tests

Security Tests

Tests

Description

Code

Comments

root_pods

check that pods are nor using root user or started as root

bash script

kubectl

unlimitted_pods

check that limits are set for pods

bash script

kubectl

cis_kubernetes

perform the k8s cis test suite (upstream src aquasecurity)

bash script

kube-bench

nonssl_endpoints

check that all public HTTP endpoints exposed in ONAP cluster use SSL tunnels

Go script

kubetl, nmap

http_public_endpoints

check that there is no public http endpoints exposed in ONAP cluster

bash script

kubectl,nmap

jdpw_ports

check that there are no internal java ports

bash script

kubectl, procfs

kube_hunter

security suite to search k8s vulnerabilities (upstream src aquasecurity)

kube-Hunter

kube-Hunter

versions

check that Java and Python are available only in versions recommended by SECCOM. This test is long and run only in Weekly CI chains

python module

cerberus, kubernetes python lib,

tern

Check the component licenses within the ONAP dockers

bash script

kubectl

Functional Requirements

functional requirements table

Issue key

Summary

Contact

Comment

REQ-463

ONAP to support Multi Tenancy (part 2)

Olivier Phénix

REQ-457

Extend ORAN A1 Adapter and add A1 Policy Management

John Keeney

REQ-458

ONAP CNF orchestration - Enhancements

Lukasz Rajewski

Changes of this feature are described in vFW CNF use case

REQ-446

NF Software Upgrade enhancement

Zu Qiang

REQ-433

ONAP/3GPP & O-RAN Alignment-Standards Defined Notifications over VES (Honolulu)

Damian Nowak

Description how to configure Standards Defined Notifications in VES

REQ-432

IPv4/IPv6 dual stack support in ONAP (Honolulu)

Damian Nowak

REQ-431

CMPv2 Enhancements for R7

Pawel Baniewski

cmpv2 automated test integrated in CI/CD, see automated test page

REQ-400

ETSI-Alignment for Guilin and Honolulu

Byung-Woo Jun

Non Functional Requirements

non functional requirements table

Issue key

Summary

Contact

Comment

REQ-453

Smart Operator Intent Translation in UUI based on IBN - R8 5G Slicing Support

Dong Wang

REQ-430

PNF Plug & Play in R8

Damian Nowak

REQ-428

5G Service Modeling in R8 - Modeling Work

Benjamin Cheung

REQ-427

Configuration Persistence Service in R8

Toine Siebelink

Deprecated Use Cases and Functional Requirements

Each ONAP release deals with lots of use cases and functional requirements. When possible, it is strongly recommended to automate the use cases. In this case Integration team can take over the maintenance part of the use case. If not automated, the use cases are fully under the responsibility of the use case team and usually valid for the release the team was involved in. However, these use cases, their artifacts remain in the repository. Anyone can give a try even if the use cases are no more supported.

This section deals with such use cases. These use cases have been part of one release but have not been tested on the last releases. They might fully deprecated or usable through minor adaptations. The entry points are the use case owners.

deprecated use case table

Use Case

Link

Last Valid Version

Comments

vFirewall with closed loop

official doc

Guilin

Shall still be OK in Honolulu but not tested yet

Scale Out

official doc

Guilin

Shall still be OK in Honolulu but not tested yet

vCPE Use Case

official doc

El Alto

No resources to test on Frankfurt

vIPsec with HPA Use Case

official doc

El Alto

No resources to test on Frankfurt

Change Management Schedule Optimization

official doc

El Alto

No resources to test on Frankfurt

Change Management Flexible Designer and Orchestrator

official doc

El Alto

No resources to test on Frankfurt

vFirewall/vDNS with HPA

official doc

Frankfurt

No resources to test on Guilin

BBS (Broadband Service)

official doc

Frankfurt

No resources to test on Guilin

vFirewall CNF with multicloud k8s plugin

official doc

Frankfurt

No resources to test on Guilin

EdgeXFoundry CNF with multicloud k8s plugin

official doc

Frankfurt

No resources to test on Guilin

vCPE with Tosca

official doc

Frankfurt

No resources to test on Guilin

E2E Automation vLB with CDS

wiki page

Frankfurt

No resources to test on Guilin

vFirewall In-Place Software Upgrade with Traffic Distribution

official doc

Frankfurt

APPC in maintenance mode

5G Bulk PM

official doc

Frankfurt

No tested in Guilin

5G NRM Network Resource Model (Configuration management)

official doc

Frankfurt

No tested in Guilin

5G NETCONF configuration

official doc

Frankfurt

No tested in Guilin

PNF Software Upgrade using direct Netconf Yang interface with PNF

official doc

Frankfurt

No tested in Guilin

PNF Software Upgrade with EM with Ansible

official doc

Frankfurt

No tested in Guilin

PNF Software Upgrade with EM with Netconf

official doc

Frankfurt

No tested in Guilin

PNF Software Upgrade in association to schema updates

official doc

Frankfurt

No tested in Guilin

VSP Compliance and Validation Check within SDC

wiki page

Frankfurt

No tested in Guilin

Enable PNF software version at onboarding

wiki page

Frankfurt

No tested in Guilin

xNF communication security enhancements

wiki page

Frankfurt

No tested in Guilin

ETSI Alignment SO plugin to support SOL003 to connect to an external VNFM

wiki page

Frankfurt

No tested in Guilin

Integration of CDS as an Actor

official doc

Frankfurt

No tested in Guilin

3rd Party Operational Domain Manager

wiki page

Frankfurt

No tested in Guilin

Configuration & persistency

wiki page

Frankfurt

No tested in Guilin

Integration Resources

Integration repositories

Important

The Integration project deals with lots of code repositories.

Most of the repositories are internal ONAP repositories.

├── demo
├── integration
│   ├── csit
|   ├── data-provider
│   ├── docker
│   │   ├── onap-java11
│   │   └── onap-python
│   ├── ietf-actn-tools
│   ├── seccom
│   ├── simulators
│   │   ├──5G-core-nf-simulator
│   │   ├──A1-policy-enforcement-simulator
│   │   ├──core-nssmf-simulator
│   │   ├──nf-simulator
│   │   │  ├──avcn-manager
│   │   │  ├──netconf-server
│   │   │  ├──pm-https-server
│   │   │  └──ves-client
│   │   ├──pnf-simulator
│   │   ├──ran-nssmf-simulator
│   │   └──ran-simulator
│   ├── usecases
│   │   ├── A1-policy-enforcement
│   │   ├── A1-policy-enforcement-r-apps
│   └── xtesting
├── oparent
│   └── cia
└── testsuite
   ├── cds
   ├── cds-mock-odl
   ├── cds-mock-server
   ├── cds-mock-ssh
   ├── oom
   ├── python-testing-utils
   ├── pythonsdk-tests
   └── robot-utils

Please note that integration and teststuite are repositories and groups hosting several sub-repositories.

Integration

The integration repository is the historical repository. As a consequence it includes several elements in the same repository:

  • Deployment scripts (deployment directory)

  • Tests: the first non robot tests (security, vCPE,..)

  • Simulators/emulators (test/mocks)

  • Integration and use cases documentation (docs)

  • Tools (bootstrap, S3Ptools)

Since Frankfurt version, we created more smaller repositories especially for the use cases and the simulators. It shall help improving the maintenance of the different elements. It shall also help identifying, leveraging and adopting existing simulators rather than systematically re-inventing the wheel.

Integration Repositories

Repository

Description

Link

integration

Historical main repository including documentation, simulators (e.g. mass PNF simulator), non robot tests (e.g. security tests, vCPE Tosca,..), …

link

integration/csit

Repository hosting some tooling to start component functional tests in Jenkins (To be deprecated in Guilin as such tests must be reinsourced by the projects)

link

integration/data-provider

Project that provides a tool to automate common ONAP resource creation

link

integration/docker/onap-java11

Java11 baseline image conformed to SECCOM recommendations

link

integration/docker/onap-python

Python baseline image conformed to SECCOM recommendations

link

integration/ietf-actn-tools

IETF ACTN tools introduced in Honolulu)

link

integration/seccom

Repory hosting seccom recommended versions and security test waivers

link

integration/usecases/A1-policy-enforcement

A1 policy enforcement introduced in Honolulu

link

integration/usecases/A1-policy-enforcement-r-apps

A1 policy enforcement (analyticis part) introduced in Honolulu

link

integration/xtesting

Repository in charge to build th xtesting dockers used in CI/CD chains

link

Integration Simulators

Repository

Description

Link

integration/simulators/A1-policy-enforcement-simulator

A1 Policy Enforcement Simulator

link

integration/simulators/5G-core-nf-simulator

5G core nf simulator

link

integration/simulators/core-nssmf-simulator

Core NSSMF Simulator

link

integration/simulators/nf-simulator

NF simulator

link

integration/simulators/nf-simulator/avcn-manager

NF simulator avcn manager

link

integration/simulators/nf-simulator/netconf-server

NF simulator netconf server

link

integration/simulators/nf-simulator/pm-https-server

NF simulator pm https server

link

integration/simulators/nf-simulator/ves-client

NF simulator ves client

link

integration/simulators/pnf-simulator

PNF Simulator

link

integration/simulators/ran-simulator

RAN simulator

link

integration/simulators/ran-nssmf-simulator

RAN NSSMF simulator

link

Testsuite

The testsuite repository and its sub repositories deal exclusively with tests.

The testsuite repository includes all the robotframework scripts. The robot pod that can be installed as part of the ONAP cluster is built from this repository.

Several tooling repositories are associated with the robot tests (heatbridge, robot-python-testing-utils).

Testsuite Repositories

Repository

Description

Link

testsuite

repository hosting the robot test suites

link

testsuite/cds

Repository hosting (standalone) CDS test suites shared by Bell Canada team, not yet integrated in CI/CD

link

testsuite/cds-mock-odl

needed for cds regression tests

link

testsuite/cds-mock-server

needed for cds regression tests

link

testsuite/cds-mock-ssh

needed for cds regression tests

link

testsuite/oom

Helm chart for robot pod (to be deprecated in Honolulu and moved back to OOM)

link

testsuite/python-testing-utils

Python and robot util libraries used for robot tests

link

testsuite/pythonsdk-tests

Repository hosting the test scenarios leveraging python-onapsdk for end to end smoke tests

link

testsuite/robot-utils

Repository aiming to provide a robot wrapper for python-onapsdk

link

Demo

In this repository you will find any artifacts needed for demo, PoC and use cases if they do not have their own repository (mainly old use cases).

Demo Repository

Repository

Description

Link

demo

Historical repository to host use case artifacts (heat templates, json files,..)

link

Oparent

Oparent Repository

Repository

Description

Link

oparent

Java dependencies for JAVA projects

link

oparent/cia

Dockerfile optimization and best practices

link

Archived repositories

Some repositories are archived and marked as “read-only” due to the lack of any activity in them.

Archived Repositories

Repository

Description

Link

integration/benchmark

Benchmark project

link

integration/devtool

Devtool project

link

integration/simulators/dc-simulator

Data Center simulator

link

integration/simulators/masspnf-simulator

Mass PNF Simulator

link

integration/terraform

Terraform based alternative infrastructure installation

link

integration/terragrunt

Compagnon repository of terraform

link

integration/usecases/bbs

BBS use case introduced in Dublin and extracted from global repository in frankfurt

link

integration/usecases/mdons

MDONS use case introduced in Frankfurt

link

testsuite/heatbridge

python utils to manage the heatbridge function to enrich cloud information to AAI (deprecated)

link

External repositories

Additionally, the Integration team also deals with external gitlab.com repositories.

Integration external repositories table

Repository

Description

Link

pythononapsdk

Python SDK for ONAP, used to create use cases

link

xtesting-onap

Repository in charge of the test section of the CD chains

link

integration-view

Repository integration hosting the itegration portal including the hosting of the web site

link

The python-onapsdk has been developed outside of ONAP as gitlab provided more enhanced built-in features for this kind of development.

The xtesting-onap repository is also hosted in gitlab.com as the CD part of Integration work is based on public gitlab-ci chains.

Integration Labs

Important

The Integration team deals with several community labs:

  • The Azure staging lab

  • The Orange openlab

  • The DT lab

  • The Nokia dualstack lab

  • The University New Hempshire lab

Additionally integration contributors may deal with their own lab pushing results in the integration portal (See DT http://testresults.opnfv.org/onap-integration/dt/dt.html)

Azure staging lab

An additional Azure staging lab has been created for Guilin. It is installed as any daily/weekly/gating labs (see CI/CD sections). Contact the Integration team to get an access.

Orange Openlab

This lab is for community use. It is always provided with the last stable version, i.e. Honolulu release during Istanbul development time.

See Orange Openlab access procedure for details.

DT lab

The DT lab reported Master daily results in addition of Istanbul daily results. Results are shared with the community in https://logs.onap.org/onap-integration/daily/onap-master-daily-dell/

Nokia lab

Nokia setup a lab to support the dual stack IPv4/IPv6 tests. Results are shared with the community in https://logs.onap.org/onap-integration/daily/onap_daily_nokia_dualstack_master/

University New Hempshire lab

Lab for community use. See ONAP UNH lab for more information.

Tests

Important

Integration is in charge of several types of tests:

  • Use Cases: developed by use case teams, usually complex, demonstrating high value capabilities of ONAP. They may be partially automated and even integrated in CD.

  • CSIT Tests: functional tests created by the projects, partially hosted in CSIT repository

  • Automatic Test Cases: these use cases are usually more simple and aim to validate that ONAP is working properly. These tests have been developed to validate ONAP as a software solution. In theory all the main functions shall be covered by such tests in order to have more robust CI/CD and then avoid regressions. These tests are usually developed and maintained by the integration team.

We may also indicate that when the development of the test framework python-onapsdk follows standard development quality rules and imposes the creation of unit/functional/integration tests. As an example python-onapsdk requires a unit test coverage of 98% before merging a new feature, which is far above the project criteria in SonarCloud today.

Use Cases

The use cases of the last release are described in Verified Use cases.

CSIT Tests

The CSIT tests are functional tests executed by the projects on mocked environment to validate their components. Historically it was hosted in a CSIT repository.

Integration team invited the projects to bring back such tests back to home repository for 2 main reasons:

  • integration cannot be a bottleneck: +2/merge from integration needed for each project

  • most of the tests are abandoned and not maintained when hosted in a third party repository leading to CI/CD resource waste and misleading test reporting

Automated Tests

These tests are run daily/weekly on each new gate (new patchset in OOM, CLAMP or SO). They can be in any language (bash, go, python,…), leveraging any test framework (robotframework, MTS, python-onapsdk). They are all embedded in xtesting dockers.

Hint

Automatic tests are currently divided in 4 different categories:

  • infrastructure-healthcheck: tests from OOM checking the ONAP namespace, certificates…

  • healthcheck: basic tests on components

  • smoke tests: end to end tests

  • security tests

A dashboard summarizing the status and providing the links to the test result page or the logs is automatically created at the end of the execution of the tests.

_images/test-dashboard.png

Test dashboard (Guilin version)

All the pages and artifacts are pushed to LF backend:

A video has been recorded to help launching some of the automated tests on ONAP Guilin. See Running ONAP tests in Guilin Video

Infrastructure Healthcheck Tests
Infrastructure Healthcheck Tests

Tests

Description

Code

Comments

onap-helm

Verify Helm chart status, the test has been updated to take into account Helm3

python module

onap-k8s

Check common resources of the ONAP Kubernetes namespace

python module

kubernetes python library

nodeport_check_certs

This test list the nodeports and tries to get SSL information to evaluate the validity of the certificates (expiration and issuer) used on the nodeports

python module

pyopenssl, kubernetes python libraries

See Infrastructure Healthcheck README to adapt then run infrastructure healthcheck tests on your own system.

Please note that the onap-k8s is run 2 times in CD chains. It is run just after the installation (onap-k8s) and at the end of the test execution (onap-k8s-teardown) in order to collect the logs of the different components during the test execution.

_images/test-onap-k8s.png
Healthcheck Tests
Healthcheck Tests

Tests

Description

Code

Comments

core

Robot healthcheck tests of the core components (AA&I, DMAAP, Portal, SDC, SDNC, SO)

robot tests

robotframework

full

Robot healthcheck tests for all the components, holmes healthcheck have been reintroduced

robot tests

robotframework

healthdist

Check the onboarding and distribution of the vFW

robot tests

robotframework

postinstall

Check dmaap and AA&I Design model DB tests

robot tests

robotframework

ves-collector (new)

Suite for checking handling events by VES Collector

code

robotframework

hv-ves

HV-VES ‘Sunny Scenario’ Robot Framework test - message is sent to the collector and Kafka topic is checked if the message has been published. Content is decoded and checked.

code

robotframework

basic_onboard

onboard a model, subset of most of the other basic_* tests, created to perform stability testing

code

onap_pythonsdk, pythonsdk-tests

dcaemod

Check new DCAEmod

robot tests

robotframework

cps-healthcheck

Call liveness and readiness probes of the CPS module

robot tests

robotframework

See Healthcheck README to adapt then run healthcheck tests on your own system.

Smoke Tests
Smoke Tests

Tests

Description

Code

Comments

basic_vm

Onboard, distribute and instantiate an Openstack VM using à la carte BPMN, replaced the former basic_vm test

code

onap_pythonsdk, pythonsdk-tests

basic_network

Onboard, distribute and instantiate a Neutron network

code

onap_pythonsdk, pythonsdk-tests

basic_cnf

Onboard (new), distribute and instantiate a Kubernetes pods

code

onap_pythonsdk, pythonsdk-tests

5gbulkpm

5G Bulk PM Usecase functionality. The test has been significantly enhanced in Honolulu

code

robotframework

pnf-registrate

Executes the PNF registration test cases including setup and teardown

code

robotframework

cmpv2

CMPv2 Usecase functionality

code

robotframework

basic_vm_macro

Instantiate a VM using macro bpmn

code

onap_pythonsdk, pythonsdk-tests

pnf_macro

Run PNF simulator, onboard, distribute and instantiate service including PNF

code

onap_pythonsdk, pythonsdk-tests

cds_resource_resolution

Upload blueprint file into CDS, execute test workflow and check results

code

onap_pythonsdk, pythonsdk-tests

basic_cnf_macro

Onboard (new), distribute and instantiate a Kubernetes pods using SO’s macro flow

code

onap_pythonsdk, pythonsdk-tests

There are 2 main families of smoke tests:

  • RobotFramework based tests, usually run from inside the cluster as a k8s job

  • Pythonsdk based tests. These tests (also known as onaptests) are consuming several SDKs: the Openstack and Kubernetes SDK for the management of the cloud resources and the python ONAP SDK for the interactions with ONAP

To launch the the robot based tests, please see Robot smoke test README Standard Robot html pages are generated. See Robot page.

To launch the pythonsdk based tests, please see Python smoke test README

An html page is generated by the pythonsdk-test tests.

_images/test-basic-cnf.png
Security Tests
Security Tests

Tests

Description

Code

Comments

root_pods

check that pods are nor using root user or started as root

bash script

kubectl

unlimitted_pods

check that limits are set for pods

bash script

kubectl

cis_kubernetes

perform the k8s cis test suite (upstream src aquasecurity)

bash script

kube-bench

nonssl_endpoints

check that all public HTTP endpoints exposed in ONAP cluster use SSL tunnels

Go script

kubetl, nmap

http_public_endpoints

check that there is no public http endpoints exposed in ONAP cluster

bash script

kubectl,nmap

jdpw_ports

check that there are no internal java ports

bash script

kubectl, procfs

kube_hunter

security suite to search k8s vulnerabilities (upstream src aquasecurity)

kube-Hunter

kube-Hunter

versions

check that Java and Python are available only in versions recommended by SECCOM. This test is long and run only in Weekly CI chains

python module

cerberus, kubernetes python lib,

tern

Check the component licenses within the ONAP dockers

bash script

kubectl

See Security test README to adapt then run the security tests on your own system.

Note for security tests, integration team follows SECCOM recommendations and apply waivers granted by SECCOM if needed through xfail lists.

Stability tests

Ensuring the stability of ONAP is one of the missions of the Integration team. CI chains and stability tests are performed to help stabilising the release. See Integration stability tests for details.

CI/CD

Important

Integration team deals with 2 different CI/CD systems.

  • Jenkins CI/CD, CI managed by LF IT and CD by Integration team

  • GitLab-CI managed by Integration team

Continuous Integration

The CI part provides the following features:

  • Repository verification (format of the INFO.yaml)

  • Patchset verification thanks to json/yaml/python/go/rst/md linters. These Jenkins verification jobs are hosted in the ci-management repository. They can vote +1/-1 on patchset submission. Integration team systematically enables linters on any new repository

  • Docker build: Integration team builds testsuite dockers and xtesting dockers. These dockers are built then pushed to Nexus through a jjb also hosted in the ci-management repository.

The different verification chains are defined in https://jenkins.onap.org/:

The Jenkins jobs (jjb) are hosted in https://git.onap.org/ci-management/.

Continuous Deployment

GitLab CD

This CD is leveraging public gitlab-ci mechanism and used to deploy several ONAP labs:

  • Daily Master: daily run using OOM Master

  • Weekly Master: run once a week with longer tests

  • Gating: run on OOM, clamp or SO patchset submission. It means a full ONAP deployment on demand based on new patchset declared in gerrit.

See Integration CI guideline for details.

Simulators

Simulators are regularly created for use cases. The goal of this section is to:

  • Highlight the existing Simulators

  • Provide recommendations when starting developing a new simulator

Important

Before developing a new simulator, check that it does not exist…and refactor/contribute to existing simulators rather than recreating new ones.

Existing simulators

Simulators

Name

Description

Link

Contacts

NF Simulator

Evolution of the pnf simulator, the Network service simulator

official doc

K.Kuzmicki

A1 Policy Enforcement Simulator

Simulator that supports the A1-P OSC_2.1.0 interface and also provides internal API to manage the RAN elements (Cells, Ues) and allows to customize and send VES Events

official readme

Krystian Kędroń

Mass PNF Simulator

Mimic the PNF for benchmark purposes

official readme

Tamas Bakai

Ran simulator

RAN-SIM is a Radio Access Network Simulator, it is used to simulate the various functionalities of an eNodeB

official readme

Priyadharshini B

DC simulator

Data Center simulator

official readme

Xin Miao

Recommendations

The simulator code

We recommend to create a dedicated repository (ask Integration team).

Simulator repositories

Repository

Description

Link

integration/simulators/A1-policy-enforcement-simulator

A1 Policy Enforcement Simulator

link

integration/simulators/5G-core-nf-simulator

5G core nf simulator

link

integration/simulators/core-nssmf-simulator

Core NSSMF Simulator

link

integration/simulators/nf-simulator

NF simulator

link

integration/simulators/nf-simulator/avcn-manager

NF simulator avcn manager

link

integration/simulators/nf-simulator/netconf-server

NF simulator netconf server

link

integration/simulators/nf-simulator/pm-https-server

NF simulator pm https server

link

integration/simulators/nf-simulator/ves-client

NF simulator ves client

link

integration/simulators/pnf-simulator

PNF Simulator

link

integration/simulators/ran-simulator

RAN simulator

link

integration/simulators/ran-nssmf-simulator

RAN NSSMF simulator

link

Dockerization

From this repository, create a jenkins job to automatically build the dockers.

Helm Chart

It is recommended to create a helm chart in order to run the simulators.

Wrapper for simulators

1. In order to deploy the Helm release with a simulator, place a YAML file describing the Helm release in src/onaptests/templates/helm_charts.

The structure of the YAML file should be like in the example below. Dependencies contain all the charts that need to be pulled.

# Helm release information
api_version:       # API_VERSION
app_version:       # APP_VERSION
chart_name:        # SIMULATOR_NAME
version:           # CHART_VERSION

# Helm charts that need to be pulled
dependencies:
- name:            # SIMULATOR_NAME
  version:         # CHART_VERSION
  repository:      # URL
  local_repo_name: # REPO_NAME
  1. Install the Helm release:

from onaptests.steps.wrapper.helm_charts import HelmChartStep

chart = HelmChartStep(
    cleanup         = BOOLEAN,
    chart_info_file = YAML_FILE_NAME  # name, not the path
)
chart.execute()
  1. Start the simulator via an API call:

 start = SimulatorStartStep(
     cleanup   = BOOLEAN,
     https     = BOOLEAN,
     host      = HOSTNAME,
     port      = PORT,
     endpoint  = START_ENDPOINT,  # if applicable
     method    = REQUEST_METHOD,  # GET, POST etc.
     data      = PAYLOAD  # {"json": {...}, ...}
)
start.execute()
  1. Undeploy the Helm release:

chart.cleanup()

Tooling

Important

Integration team deals with lots of tools to complete its missions. The goal of this section is to highlight some of them and redirect to their official documentation. These tools can be used for CI/CD, Testing or platform management.

Upstream tools are privileged but when needed specific developments can be done.

Please note that none of these tools are imposed to test developers, in other words, any kind of test is accepted and can be integrated, the list of tools is just indicative.

Integration Project

Integration portal

A portal is built to report the status of the different labs collaborating in Integration, see http://testresults.opnfv.org/onap-integration/

_images/ONAP_CI_3.png

The code of this web site is shared on a public gitlab project.

Communication channels

The main communication channel for real time support is the official ONAP Slack #integration-team chan (https://onapproject.slack.com/).

You can also send a mail to onap-discuss AT lists.onap.org with [ONAP] [Integration] prefix in the title.

Repository management

Since Integration team manages few dozens of different repositories a tool was provided to aid the process of mass editing the INFO.yaml files. It can be found here.

Testing

Test frameworks
Robotframework

robotframework is a well known test framework. Lots of ONAP tests are leveraging this framework. This framework is fully developed upstream even if some extensions (python modules) were created especially to deal with OpenStack (see python-testing-utils project).

Some GUI tests (using Robotframework Selenium extension) had been initiated but not maintained, as a consequence they are not integrated in CI/CD.

Python-onapsdk

The Openstack and Kubernetes python SDK are references widely adopted by the developers and the industry. Developing a python ONAP SDK aimed to follow the examples of the infrastructure SDK with the same expectations in term of code quality. After an evaluation of the CLI project (JAVA SDK re-exposing primitives through python system calls), and a first prototype (onap_tests used until Frankfurt for end to end tests) it was decided to develop a new python SDK.

This SDK has been developed in gitlab.com to benefit from the numerous built-in options offered by gitlab and ensure the best possible code quality.

The project is fully Open Source, released under the Apache v2 license. Integration committers are invited to join the project. The main maintainers are ONAP integration and OOM committers.

Any new feature shall respect the code quality criteria:

  • unit test coverage > 98%

  • functional tests (several components mock objects have been developed)

Attention

Python-onapsdk is a SDK, it means it is a tool allowing to communicate with ONAP. It is a middleware that can be used by test projects but it is NOT a test.

A compagnon project has been created in ONAP: pythonsdk-tests.

The pythonsdk-test project defines tests based on python-onapsdk.

The tests are hosted in this repository. They consume the different needed SDK: python-onapsdk but also the kubernetes, the OpenStack SDK and or any needed additional middlewares. The project developed the notion of steps that can been combined and reorganized as need to design a test. This project interacts with ONAP only through the python-onapsdk library. The tests are described in The Integration Test page.

The available steps are:

  • [CLAMP] OnboardClampStep: Onboard a SDC including a TCA blueprint

  • [CDS] ExposeCDSBlueprintprocessorNodePortStep: expose CDS blueprint nodeport (Guilin workaround)

  • [CDS] BootstrapBlueprintprocessor: Bootstrap a blueprint processor

  • [CDS] DataDictionaryUploadStep: Upload a Data Dictionary to CDS

  • [CDZ] CbaEnrichStep: Enrich CBA

  • [K8S plugin] K8SProfileStep: Create K8S profile

  • [SO] YamlTemplateVfModuleAlaCarteInstantiateStep: Instantiate VF module described in YAML using SO a’la carte method

  • [SO] YamlTemplateVlAlaCarteInstantiateStep: Instantiate network link described in YAML using SO a’la carte method.

  • [SO] YamlTemplateVfModuleAlaCarteInstantiateStep: Instantiate VF module described in YAML using SO a’la carte method

  • [SO] YamlTemplateVnfAlaCarteInstantiateStep: Instantiate vnf described in YAML using SO a’la carte method

  • [SO] YamlTemplateServiceAlaCarteInstantiateStep: Instantiate service described in YAML using SO a’la carte method

  • [AAI] ConnectServiceSubToCloudRegionStep: Connect service subscription with cloud region

  • [AAI] CustomerServiceSubscriptionCreateStep: Create customer’s service subscription

  • [AAI] CustomerCreateStep: Create customer

  • [AAI] LinkCloudRegionToComplexStep: Connect cloud region with complex

  • [AAI] ComplexCreateStep: Create complex

  • [AAI] RegisterCloudRegionStep: Register cloud region

  • [SDC] YamlTemplateServiceOnboardStep: Onboard service described in YAML file in SDC

  • [SDC] YamlTemplateVfOnboardStep: Onboard vf described in YAML file in SDC

  • [SDC] YamlTemplateVspOnboardStep: Onboard vsp described in YAML file in SDC

  • [SDC] VendorOnboardStep: Onboard vendor in SDC

You can reuse the existing steps to compose your test and/or code your own step if it is not supported yet.

The procedure to start a test is described in pythonsdk-test README

CI/CD

The CI/CD is key for integration. It consolidates the trustability in the solution by the automated verification of the deployment and the execution of tests. Integration tests complete the component tests (unit and functional known as CSIT tests).

Xtesting

As the tests can be very heterogeneous (framework, language, outputs), the integration team integrates the tests in simple isolated execution context based on docker called xtesting dockers.

Xtesting is a python library harmonizing the way to setup, run, teardown, manage the artifacts, manage the reporting of the tests (automatic push of the results on a DB backend). It was developed by OPNFV functest project. This python library is included in an alpine docker and contains the needed tests, their associated libraries as well as a testcases.yaml listing these tests. These docker files are built on any change in the integration/xtesting repository and daily to take into account the upstream changes.

The integration project manages 5 xtesting dockers, see Integration Test page.

Important

xtesting is a CI/CD framework, neither a test nor a test framework

Testers can provide tests independently from xtesting. However to be part of the CI/CD chains, an integration of the test in xtesting will be required.

The configuration files are provided as volumes and defined in each docker. The use of this CI/CD abstraction for the tests simplify the integration of the test suites in any CI/CD systems and harmonize the inputs and the outputs.

The official documentation can be found on xtesting official web site

Integration Test database

The integration team shares a Test Result Database with the OPNFV project. All the test results of the CD are automatically pushed to this database. It is possible to retrieve the results through the Test API associated with this test Database.

The following information are available:

It is possible to get results according to several criteria (version, case name, lab, period, last, CI id,..) See the OPNFV test API documentation.

Any company running ONAP Integration tests can be referenced to push their results to this database. This Database is hosted on a LF OPNFV server. Results are backuped daily. Integration committers can have access to this server.

VNF demo Artifacts

VNF demo artifacts are hosted in the demo repositories and published in https://nexus.onap.org/content/repositories/releases/org/onap/demo/vnf/.

Integration Missions

Important

The Integration project is in charge of:

  • Providing testing environment

  • Supporting the use case teams

  • Managing ONAP CI/CD chains

  • Developing tests

  • Providing baseline images

  • Validating the ONAP releases

The different activities may be summarized as follows (proportions are indicative):

  • Community support

  • Lab support

  • Use case support

  • Test development

  • Management of daily/weekly CI chains

  • Build baseline images

  • Automate tests

  • Validate the release

For each release, the integration team provides the following artifacts:

  • A daily CI chain corresponding to the release

  • Staging labs to perform the pairwise testing (when not automated) and support the use case teams

  • Baseline Java and Python images

  • oparent library to manage Java dependencies

  • Test suites and tools to check the various ONAP components

  • Use-case documentation and artifacts

  • A testsuite docker included in the ONAP cluster to execute the robot based tests

  • Configuration files (scripts, Heat templates, CSAR files) to help installing and testing ONAP

  • Wiki release follow-up tables (blocking points, docker versions,…)

Please see the integration wiki page for details.