INTEGRATION

Integration Istanbul Release Notes

Integration Releases

Repository

Revision

demo

129d553d323653461d14f248cb9e8b6f266eb5e3

integration

81ef97d2e7f8bf1d50f590447725efd05ba9014b

integration/csit

85321a52e56422c1bb552c10387d27066decee84

integration/docker/onap-java11

ad17a4087baaefb1adf9ad0dde3029b1da7dfc7d

integration/docker/onap-python

a0588db586ae626b4fadcfde450039ab69ed4171

integration/ietf-actn-tools

2770f33536cfb828b7416adff8eafdd0a141a0b4

integration/seccom

538d7faf46a58fdf8ff857770a9e65aa9c312913

integration/simulators/5G-core-nf-simulator

ef3045db013ce7826c0c6f2a3597f7a1d1033106

integration/simulators/nf-simulator

f739bd6b7e48edad36966a00894dc08de0c4de21

integration/simulators/nf-simulator/avcn-manager

8be57ab7523dc201a54cf4f5b0251159af1fda38

integration/simulators/nf-simulator/netconf-server

2cf18a71ceccf0ae6508a4ffabd1adef90772211

integration/simulators/nf-simulator/pm-https-server

3f7d8d756e0951c95b894cdbec785ed4f24052e4

integration/simulators/nf-simulator/ves-client

7397ac984b69f7605e6ccb0b5ab02a74b73e818b

integration/simulators/pnf-simulator

20fbea02fedad63770a5a7e3958adb71992a7878

integration/simulators/ran-nssmf-simulator

85ccf7677389c9bd3f43c31d220347e9ebd4d404

integration/simulators/ran-simulator

a3165afba7f422e61fecdce5c5c24d8b91d8bd40

integration/usecases/A1-policy-enforcement

ada51c6695b785865eea3ccea45a13026bd03518

integration/usecases/A1-policy-enforcement-r-apps

3ac3f212d7127e887625a344c3e4846269a01df0

integration/xtesting

a93cdb28c8e9e05f8d794556dce52bca11b59028

oparent

c6044f697e3345c8b907a4f8d5f0cfa3d4069071

testsuite

b4680596e5c876b12dc416c305de69371d5aedd5

testsuite/cds

9f83b04496cd16ae280af1e939eff20ff7195fc6

testsuite/cds-mock-odl

ae2aebead9224fb0bbbe6aaf0b782b27a501dee9

testsuite/cds-mock-server

7db71adaf139e54f2186cfd19d468f5a1123835d

testsuite/cds-mock-ssh

a43ce8950dcc36363c406b1cc4043dc7d623c9f4

testsuite/oom

f3b6c657d38373ce2aaac9a2041285b75525bfc2

testsuite/python-testing-utils

1911416dec8d483ccaa320b5ed0a6fbbafe1f9c2

testsuite/pythonsdk-tests

1a77bcd6eb5ab695a232004b88e5e59d9d9ff6a3

testsuite/robot-utils

8b80d8c8bebad201185639807ec50110fb857638

Important

  • Creation of an Istanbul Daily CI/CD chain

  • Creation of Java and Python baseline images for Istanbul

  • Update of Seccom waivers and version recommendations

  • Stability tests (basic_onboard, basic_vm)

  • New tests (cps-healthcheck)

  • New repositories (see dedicated section)

  • Bug fixes

  • ONAP tests library gating tests

Quick Links:

Code changes

Integration Repo

Release Date

2021-10-14

Version: 9.0.0 (aka Istanbul)
Integration Changes

Issue-ID

Description

INT-1979

E2E Network slicing - Istanbul Release Documentation

INT-1976

Updating release notes for CCVPN - Intent Based Networking and Cloud Leased Line Service.

DOC-765

Remove indirect deps

DOC-765

Make funcparserlib + lfdocs-conf work on RTD

DOC-765

Adds 2 upper-constraints to pin all the dependencies.

REQ-910

Update docs for OOF-SON use case

INT-1966

[DOC] Remove direct gating link

DCAEGEN2-2692

Fix problem with expired certs in FTPES tests

DCAEGEN2-2692

Update DFC tests to use file based conifg

TEST-360

[SECURITY] Fix waiver management of check_for_nonssl_ports test

INT-1735

Update Vagrantfile for tern

INT-1601

Enable VID

INT-1601

noheat: deploy ONAP

INT-1601

noheat: deploy MetalLB, cert-manager and prometheus

INT-1601

noheat: deploy helm with plugins & chartmuseum

INT-1601

noheat: install Ansible kubernetes collection

INT-1601

noheat: use Python 3 as Ansible interpreter

INT-1601

noheat: clone OOM repository

INT-1601

noheat: deploy kubernetes

INT-1601

noheat: deploy Docker

INT-1601

noheat deployment: use nfs0 as nexus3 bastion

INT-1601

noheat deployment: setup NFS server and clients

INT-1601

noheat deployment: add operator0 key to itself

INT-1601

noheat deployment: loosen security groups constraints

INT-1601

noheat deployment: Add groups to dynamic inventory

INT-1601

Update requirements of OpenStack noheat deployment

INT-1940

[CODESEARCH] Update documentation

INT-1940

[CODESEARCH] Always run the ‘run_codesearch’ provisioner

INT-1940

[CODESEARCH] Rework the nameserver provisioning

INT-1940

[CODESEARCH] Drop ssh authentication for Gerrit endpoint

INT-1940

[CODESEARCH] Rework how –git option works

INT-1942

[CODESEARCH] Add option to define custom polling interval

INT-1941

[CODESEARCH] Upgrade Vagrant box to utilise newest 20.04 LTS

INT-1922

Standard defined schema files in VES Collector

INT-1601

Set up network for in-cluster deployment stage

INT-1601

Add missing dependencies and artifacts for in-cluster deployment stage

INT-1601

Prepare operation machine for in-cluster deployment stage

Onaptests (pythonsdk_tests)

Main changes:

pythonsdk_tests Changes

Issue-ID

Description

TEST-364

[TEST] Do not use VID API in tests

INT-1978

[BASIC_CLAMP] Adapt tca clamp plugin

TEST-361

Update HEAT-file for ubuntu20 used for basic_vm_macro test

TEST-357

[TEST] Get cleanup reports from substeps also if parent step has no cleanup report

MULTICLOUD-1377

Change rb-definition-version identifier

TEST-352

[TEST] Use the newest ONAP SDK version

TEST-349

[TEST] Basic macro stability scenario

TEST-347

[TEST] Check if cds blueprintsprocessor service type is ‘NodePort’

TEST-341

[TEST] Wait for instantiated simulator longer

TEST-341

[TEST] Create “pnf-macro-test-simulator” resources

TEST-341

[TEST] Use nf-simulator/vesclient

TEST-315

[OPTIM] Tune SDC delay before certification

INT-1894

[TEST] Patch ip values in pnf-simulator event

TEST-333

[TEST] Do not try to recreate already created SDC resources

TEST-338

[CLAMP] Fix Policy exception in basic_clamp

TEST-337

[TEST] Catch ConnectionError exception during pnf simulator startup

TEST-336

[TEST] Catch k8s connection exceptions

TEST-334

[CLAMP] Update clamp to allow re-play of the test

TEST-332

[EXCEPTIONS] Distinguish onaptests and onapsdk exception

Robot (Testsuite)

Version: 1.9.0

Main changes:

Testsuite Changes

Issue-ID

Description

TEST-364

[TEST] Do not use VID API in pnf-registrate test

TEST-365

[HEALTHCHECK] Remove VID from healthcheck

DCAEGEN2-2718

Use configMap instead of Consul to configure hv-ves

TEST-353

Cmpv2 robot testcase fails when using proxy

MULTICLOUD-1362

[MULTICLOUD] Exclude multicloud healthcheck tests

OPTFRA-969

[OOF] Remove CMSO related test cases

TEST-346

CMPv2 test case fails with error No keyword with name

TEST-345

Refactor ves client and mongo db host name usage in CMPv2 test

TEST-339

[POLICY] Increase Timeout for Enhanced Policy Healthcheck Test

O-Parent

Version: 3.2.2
Oparent Changes

Issue-ID

Description

INT-1948

Set default branch in oparent gitreview file

INT-1948

Bump oparent to 3.2.2

INT-1948

Release oparent 3.2.1 Istanbul

INT-1938

Upgrade depenencies for Istanbul

POLICY-3209

Step version of spring to 5.3.8

POLICY-3206

Update Checkstyle to version 8.43

INT-1711

Bump oparent 3.2.1-SNAPSHOT

INT-1711

Release 3.2.0 oparent

INT-1766

Upgrade latest dependencies

INT-1756

Bump master to 3.2.0-SNAPSHOT

Demo Artifacts (Heat Templates)

Version: 1.9.0
Demo Changes

Issue-ID

Description

INT-1960

vFW CNF CDS Change Blueprint to BluePrint

INT-1960

vFW CNF CDS usecase automation scripts update

INT-1899

[vFW_CNF_CDS] Improve logging and fix tests typo

INT-1899

[vFW_CNF_CDS] Add workflow health-check and K8sHealthCheck.kt script

INT-1899

[vFW_CNF_CDS] Sort generated json files

INT-1899

[vFW_CNF_CDS] Add Healthcheck automation

INT-1899

[vFW_CNF_CDS] Provide Helm Tests for CNF

INT-1899

[vFW_CNF_CDS] Correct Makefile target

The demo artifacts are pushed to https://nexus.onap.org/content/repositories/releases/org/onap/demo/vnf

Use Cases and Requirements

See dedicated Istanbul Use Cases and requirements page

Maturity Testing Notes

Maturity testing page

Open JIRAs/Known issues

Integration

Integration JIRA page

Testsuite

Testsuite JIRA page

Istanbul Use Cases and Requirements

Description

This session includes use cases and functional requirements which have been officially verified in Istanbul release by the ONAP community.

For each use case or functional requirement, you can find contact names and a link to the associated documentation.

This documentation deals with

  1. What has been implemented

  2. Step by step instructions to deploy and execute the tests, including the links to download the related assets and resources

  3. Known issues and workarounds

Istanbul Use Cases

Description

This session includes use cases and functional requirements which have been officially verified in Istanbul release by the ONAP community.

For each use case or functional requirement, you can find contact names and a link to the associated documentation.

This documentation deals with

  1. What has been implemented

  2. Step by step instructions to deploy and execute the tests, including the links to download the related assets and resources

  3. Known issues and workarounds

Use cases
Istanbul Official Use Cases

Ref

Summary

Link

Contacts

REQ-440

E2E Network Slicing

official doc

Lin Meng

REQ-429

5G OOF SON

official doc

Swaminathan Seetharaman

REQ-459

CCVPN-Transport Slicing

official doc

Henry Yu
Old Use Cases Tested in Istanbul

Summary

Link

Contacts

vFirewall CNF With CDS

official doc

L.Rajewski, K.Banka

5G Realtime PM and High Volume Stream Data Collection

official doc

M.Przybysz

5G PNF Plug and Play

official doc

M.Przybysz K.Kuzmicki

5G PNF Pre-Onboarding & Onboarding

official doc

M.Przybysz K.Kuzmicki D.Melia A.Walshe

MDONS extension

official doc

X.Miao

Automated Use Cases

These use cases have been run on the Daily Honolulu CI chains and are used to validate the integration of any new dockers in OOM. New tests are indicated in bold.

Infrastructure Healthcheck Tests

Tests

Description

Code

Comments

onap-helm

Verify Helm chart status, the test has been updated to take into account Helm3

python module

onap-k8s

Check common resources of the ONAP Kubernetes namespace

python module

kubernetes python library

nodeport_check_certs

This test list the nodeports and tries to get SSL information to evaluate the validity of the certificates (expiration and issuer) used on the nodeports

python module

pyopenssl, kubernetes python libraries

internal_check_certs

This test list the internal ports and tries to get SSL information to evaluate the validity of the certificates (expiration and issuer) used

python module

pyopenssl, kubernetes python libraries
Healthcheck Tests

Tests

Description

Code

Comments

core

Robot healthcheck tests of the core components (AA&I, DMAAP, Portal, SDC, SDNC, SO)

robot tests

robotframework

full

Robot healthcheck tests for all the components, holmes healthcheck have been reintroduced

robot tests

robotframework

healthdist

Check the onboarding and distribution of the vFW

robot tests

robotframework

postinstall

Check dmaap and AA&I Design model DB tests

robot tests

robotframework

ves-collector (new)

Suite for checking handling events by VES Collector

code

robotframework

hv-ves

HV-VES ‘Sunny Scenario’ Robot Framework test - message is sent to the collector and Kafka topic is checked if the message has been published. Content is decoded and checked.

code

robotframework

basic_cds

CDS blueprint enrichment, open a nodeport on CDS then enrich CDS CBA

code

onap_pythonsdk, pythonsdk-tests

basic_onboard

onboard a model, subset of most of the other basic_* tests, created to perform stability testing

code

onap_pythonsdk, pythonsdk-tests

dcaemod

Check new DCAEmod

robot tests

robotframework

cps-healthcheck

Call liveness and readiness probes of the CPS module

robot tests

robotframework
Smoke Tests

Tests

Description

Code

Comments

basic_vm

Onboard, distribute and instantiate an Openstack VM using à la carte BPMN, replaced the former basic_vm test

code

onap_pythonsdk, pythonsdk-tests

basic_network

Onboard, distribute and instantiate a Neutron network

code

onap_pythonsdk, pythonsdk-tests

basic_cnf

Onboard (new), distribute and instantiate a Kubernetes pods

code

onap_pythonsdk, pythonsdk-tests

5gbulkpm

5G Bulk PM Usecase functionality. The test has been significantly enhanced in Honolulu

code

robotframework

pnf-registrate

Executes the PNF registration test cases including setup and teardown

code

robotframework

cmpv2

CMPv2 Usecase functionality

code

robotframework

basic_clamp

distribute a model, enrich it with a tca blueprint, call Clamp to create a loop then instantiate it in Policy and DCAE

code

onap_pythonsdk, pythonsdk-tests

basic_vm_macro

Instantiate a VM using macro bpmn

code

onap_pythonsdk, pythonsdk-tests
Security Tests

Tests

Description

Code

Comments

root_pods

check that pods are nor using root user or started as root

bash script

kubectl

unlimitted_pods

check that limits are set for pods

bash script

kubectl

cis_kubernetes

perform the k8s cis test suite (upstream src aquasecurity)

bash script

kube-bench

nonssl_endpoints

check that all public HTTP endpoints exposed in ONAP cluster use SSL tunnels

Go script

kubetl, nmap

http_public_endpoints

check that there is no public http endpoints exposed in ONAP cluster

bash script

kubectl,nmap

jdpw_ports

check that there are no internal java ports

bash script

kubectl, procfs

kube_hunter

security suite to search k8s vulnerabilities (upstream src aquasecurity)

kube-Hunter

kube-Hunter

versions

check that Java and Python are available only in versions recommended by SECCOM. This test is long and run only in Weekly CI chains

python module

cerberus, kubernetes python lib,

tern

Check the component licenses within the ONAP dockers

bash script

kubectl

Functional Requirements

functional requirements table

Issue key

Summary

Contact

Comment

REQ-463

ONAP to support Multi Tenancy (part 2)

Olivier Phénix

REQ-457

Extend ORAN A1 Adapter and add A1 Policy Management

John Keeney

REQ-458

ONAP CNF orchestration - Enhancements

Lukasz Rajewski

Changes of this feature are described in vFW CNF use case

REQ-446

NF Software Upgrade enhancement

Zu Qiang

REQ-433

ONAP/3GPP & O-RAN Alignment-Standards Defined Notifications over VES (Honolulu)

Damian Nowak

Description how to configure Standards Defined Notifications in VES

REQ-432

IPv4/IPv6 dual stack support in ONAP (Honolulu)

Damian Nowak

REQ-431

CMPv2 Enhancements for R7

Pawel Baniewski

cmpv2 automated test integrated in CI/CD, see automated test page

REQ-400

ETSI-Alignment for Guilin and Honolulu

Byung-Woo Jun

Non Functional Requirements

non functional requirements table

Issue key

Summary

Contact

Comment

REQ-453

Smart Operator Intent Translation in UUI based on IBN - R8 5G Slicing Support

Dong Wang

REQ-430

PNF Plug & Play in R8

Damian Nowak

REQ-428

5G Service Modeling in R8 - Modeling Work

Benjamin Cheung

REQ-427

Configuration Persistence Service in R8

Toine Siebelink

Deprecated Use Cases and Functional Requirements

Each ONAP release deals with lots of use cases and functional requirements. When possible, it is strongly recommended to automate the use cases. In this case Integration team can take over the maintenance part of the use case. If not automated, the use cases are fully under the responsibility of the use case team and usually valid for the release the team was involved in. However, these use cases, their artifacts remain in the repository. Anyone can give a try even if the use cases are no more supported.

This section deals with such use cases. These use cases have been part of one release but have not been tested on the last releases. They might fully deprecated or usable through minor adaptations. The entry points are the use case owners.

deprecated use case table

Use Case

Link

Last Valid Version

Comments

vFirewall with closed loop

official doc

Guilin

Shall still be OK in Honolulu but not tested yet

Scale Out

official doc

Guilin

Shall still be OK in Honolulu but not tested yet

vCPE Use Case

official doc

El Alto

No resources to test on Frankfurt

vIPsec with HPA Use Case

official doc

El Alto

No resources to test on Frankfurt

Change Management Schedule Optimization

official doc

El Alto

No resources to test on Frankfurt

Change Management Flexible Designer and Orchestrator

official doc

El Alto

No resources to test on Frankfurt

vFirewall/vDNS with HPA

official doc

Frankfurt

No resources to test on Guilin

BBS (Broadband Service)

official doc

Frankfurt

No resources to test on Guilin

vFirewall CNF with multicloud k8s plugin

official doc

Frankfurt

No resources to test on Guilin

EdgeXFoundry CNF with multicloud k8s plugin

official doc

Frankfurt

No resources to test on Guilin

vCPE with Tosca

official doc

Frankfurt

No resources to test on Guilin

E2E Automation vLB with CDS

wiki page

Frankfurt

No resources to test on Guilin

vFirewall In-Place Software Upgrade with Traffic Distribution

official doc

Frankfurt

APPC in maintenance mode

5G Bulk PM

official doc

Frankfurt

No tested in Guilin

5G OOF and PCI

official doc

Frankfurt

No tested in Guilin

5G NRM Network Resource Model (Configuration management)

official doc

Frankfurt

No tested in Guilin

5G NETCONF configuration

official doc

Frankfurt

No tested in Guilin

5G OOF SON

official doc

Frankfurt

No tested in Guilin

5G E2E Network Slicing

official doc

Frankfurt

No tested in Guilin

PNF Software Upgrade using direct Netconf Yang interface with PNF

official doc

Frankfurt

No tested in Guilin

PNF Software Upgrade with EM with Ansible

official doc

Frankfurt

No tested in Guilin

PNF Software Upgrade with EM with Netconf

official doc

Frankfurt

No tested in Guilin

PNF Software Upgrade in association to schema updates

official doc

Frankfurt

No tested in Guilin

VSP Compliance and Validation Check within SDC

wiki page

Frankfurt

No tested in Guilin

Enable PNF software version at onboarding

wiki page

Frankfurt

No tested in Guilin

xNF communication security enhancements

wiki page

Frankfurt

No tested in Guilin

ETSI Alignment SO plugin to support SOL003 to connect to an external VNFM

wiki page

Frankfurt

No tested in Guilin

Integration of CDS as an Actor

official doc

Frankfurt

No tested in Guilin

3rd Party Operational Domain Manager

wiki page

Frankfurt

No tested in Guilin

Configuration & persistency

wiki page

Frankfurt

No tested in Guilin

Integration Resources

Integration repositories

Important

The Integration project deals with lots of code repositories.

Most of the repositories are internal ONAP repositories.

├── demo
├── integration
│   ├── csit
│   ├── docker
│   │   ├── onap-java11
│   │   └── onap-python
│   ├── ietf-actn-tools
│   ├── integration
│   ├── seccom
│   ├── simulators
│   │   ├──5G-core-nf-simulator
│   │   ├──A1-policy-enforcement-simulator
│   │   ├──core-nssmf-simulator
│   │   ├──nf-simulator
│   │   │  ├──avcn-manager
│   │   │  ├──netconf-server
│   │   │  ├──pm-https-server
│   │   │  └──ves-client
│   │   ├──pnf-simulator
│   │   ├──ran-nssmf-simulator
│   │   └──ran-simulator
│   ├── usecases
│   │   ├── A1-policy-enforcement
│   │   ├── A1-policy-enforcement-r-apps
│   └── xtesting
├── oparent
│   └── cia
└── testsuite
   ├── cds
   ├── cds-mock-odl
   ├── cds-mock-server
   ├── cds-mock-ssh
   ├── oom
   ├── python-testing-utils
   ├── pythonsdk-tests
   └── robot-utils

Please note that integration and teststuite are repositories and groups hosting several sub-repositories.

Integration

The integration repository is the historical repository. As a consequence it includes several elements in the same repository:

  • Deployment scripts (deployment directory)

  • Tests: the first non robot tests (security, vCPE,..)

  • Simulators/emulators (test/mocks)

  • Integration and use cases documentation (docs)

  • Tools (bootstrap, S3Ptools)

Since Frankfurt version, we created more smaller repositories especially for the use cases and the simulators. It shall help improving the maintenance of the different elements. It shall also help identifying, leveraging and adopting existing simulators rather than systematically re-inventing the wheel.

Integration Repositories

Repository

Description

Link

integration

Historical main repository including documentation, simulators (e.g. mass PNF simulator), non robot tests (e.g. security tests, vCPE Tosca,..), …

link

integration/csit

Repository hosting some tooling to start component functional tests in Jenkins (To be deprecated in Guilin as such tests must be reinsourced by the projects)

link

integration/docker/onap-java11

Java11 baseline image conformed to SECCOM recommendations

link

integration/docker/onap-python

Python baseline image conformed to SECCOM recommendations

link

integration/ietf-actn-tools

IETF ACTN tools introduced in Honolulu)

link

integration/seccom

Repory hosting seccom recommended versions and security test waivers

link

integration/usecases/A1-policy-enforcement

A1 policy enforcement introduced in Honolulu

link

integration/usecases/A1-policy-enforcement-r-apps

A1 policy enforcement (analyticis part) introduced in Honolulu

link

integration/xtesting

Repository in charge to build th xtesting dockers used in CI/CD chains

link
Integration Simulators

Repository

Description

Link

integration/simulators/A1-policy-enforcement-simulator

A1 Policy Enforcement Simulator

link

integration/simulators/5G-core-nf-simulator

5G core nf simulator

link

integration/simulators/core-nssmf-simulator

Core NSSMF Simulator

link

integration/simulators/nf-simulator

NF simulator

link

integration/simulators/nf-simulator/avcn-manager

NF simulator avcn manager

link

integration/simulators/nf-simulator/netconf-server

NF simulator netconf server

link

integration/simulators/nf-simulator/pm-https-server

NF simulator pm https server

link

integration/simulators/nf-simulator/ves-client

NF simulator ves client

link

integration/simulators/pnf-simulator

PNF Simulator

link

integration/simulators/ran-simulator

RAN simulator

link

integration/simulators/ran-nssmf-simulator

RAN NSSMF simulator

link

Testsuite

The testsuite repository and its sub repositories deal exclusively with tests.

The testsuite repository includes all the robotframework scripts. The robot pod that can be installed as part of the ONAP cluster is built from this repository.

Several tooling repositories are associated with the robot tests (heatbridge, robot-python-testing-utils).

Testsuite Repositories

Repository

Description

Link

testsuite

repository hosting the robot test suites

link

testsuite/cds

Repository hosting (standalone) CDS test suites shared by Bell Canada team, not yet integrated in CI/CD

link

testsuite/cds-mock-odl

needed for cds regression tests

link

testsuite/cds-mock-server

needed for cds regression tests

link

testsuite/cds-mock-ssh

needed for cds regression tests

link

testsuite/oom

Helm chart for robot pod (to be deprecated in Honolulu and moved back to OOM)

link

testsuite/python-testing-utils

Python and robot util libraries used for robot tests

link

testsuite/pythonsdk-tests

Repository hosting the test scenarios leveraging python-onapsdk for end to end smoke tests

link

testsuite/robot-utils

Repository aiming to provide a robot wrapper for python-onapsdk

link

Demo

In this repository you will find any artifacts needed for demo, PoC and use cases if they do not have their own repository (mainly old use cases).

Demo Repository

Repository

Description

Link

demo

Historical repository to host use case artifacts (heat templates, json files,..)

link

Oparent

Oparent Repository

Repository

Description

Link

oparent

Java dependencies for JAVA projects

link

oparent/cia

Dockerfile optimization and best practices

link

Archived repositories

Some repositories are archived and marked as “read-only” due to the lack of any activity in them.

Archived Repositories

Repository

Description

Link

integration/benchmark

Benchmark project

link

integration/devtool

Devtool project

link

integration/simulators/dc-simulator

Data Center simulator

link

integration/simulators/masspnf-simulator

Mass PNF Simulator

link

integration/terraform

Terraform based alternative infrastructure installation

link

integration/terragrunt

Compagnon repository of terraform

link

integration/usecases/bbs

BBS use case introduced in Dublin and extracted from global repository in frankfurt

link

integration/usecases/mdons

MDONS use case introduced in Frankfurt

link

testsuite/heatbridge

python utils to manage the heatbridge function to enrich cloud information to AAI (deprecated)

link

External repositories

Additionally, the Integration team also deals with external gitlab.com repositories.

Integration external repositories table

Repository

Description

Link

pythononapsdk

Python SDK for ONAP, used to create use cases

link

xtesting-onap

Repository in charge of the test section of the CD chains

link

integration-view

Repository integration hosting the itegration portal including the hosting of the web site

link

The python-onapsdk has been developed outside of ONAP as gitlab provided more enhanced built-in features for this kind of development.

The xtesting-onap repository is also hosted in gitlab.com as the CD part of Integration work is based on public gitlab-ci chains.

Integration Labs

Important

The Integration team deals with several community labs:

  • The Windriver/Intel lab

  • The Azure staging lab

  • The Orange openlab

  • The DT lab

  • The Nokia dualstack lab

Additionally integration contributors may deal with their own lab pushing results in the integration portal (See DT http://testresults.opnfv.org/onap-integration/dt/dt.html)

Windriver/Intel lab

The Historical Community Lab

This lab is the historical lab of ONAP integration team based on OpenStack Ocata.

The figure hereafter shows all the ONAP projects consuming Windriver/Intel lab resources (April 2020).

_images/windriver_servers.png

This lab is mainly used by the projects for the development. A staging lab named SB-00 is also available.

If you want to use this lab, you need a VPN access. The procedure is described in the wiki.

Environment Installation Scripts

In addition to the official OOM scripts, Integration used to provide some extra scripts/guidelines to install your OpenStack infrastructure thanks to a heat template. See Integration heat guideline for details. These scripts were used mainly in Windriver labs but are not actively maintained.

Azure staging lab

An additional Azure staging lab has been created for Guilin. It is installed as any daily/weekly/gating labs (see CI/CD sections). Contact the Integration team to get an access.

Orange Openlab

This lab is for community use. It is always provided with the last stable version, i.e. Honolulu release during Istanbul development time.

See Orange Openlab access procedure for details.

DT lab

The DT lab reported Master daily results in addition of Istanbul daily results. Results are shared with the community in https://logs.onap.org/onap-integration/daily/onap-master-daily-dell/

Nokia lab

Nokia setup a lab to support the dual stack IPv4/IPv6 tests. Results are shared with the community in https://logs.onap.org/onap-integration/daily/onap_daily_nokia_dualstack_master/

Tests

Important

Integration is in charge of several types of tests:

  • Use Cases: developed by use case teams, usually complex, demonstrating high value capabilities of ONAP. They may be partially automated and even integrated in CD.

  • CSIT Tests: functional tests created by the projects, partially hosted in CSIT repository

  • Automatic Test Cases: these use cases are usually more simple and aim to validate that ONAP is working properly. These tests have been developed to validate ONAP as a software solution. In theory all the main functions shall be covered by such tests in order to have more robust CI/CD and then avoid regressions. These tests are usually developed and maintained by the integration team.

We may also indicate that when the development of the test framework python-onapsdk follows standard development quality rules and imposes the creation of unit/functional/integration tests. As an example python-onapsdk requires a unit test coverage of 98% before merging a new feature, which is far above the project criteria in SonarCloud today.

Use Cases

The use cases of the last release are described in Verified Use cases.

CSIT Tests

The CSIT tests are functional tests executed by the projects on mocked environment to validate their components. Historically it was hosted in a CSIT repository.

Integration team invited the projects to bring back such tests back to home repository for 2 main reasons:

  • integration cannot be a bottleneck: +2/merge from integration needed for each project

  • most of the tests are abandoned and not maintained when hosted in a third party repository leading to CI/CD resource waste and misleading test reporting

Automated Tests

These tests are run daily/weekly on each new gate (new patchset in OOM, CLAMP or SO). They can be in any language (bash, go, python,…), leveraging any test framework (robotframework, MTS, python-onapsdk). They are all embedded in xtesting dockers.

Hint

Automatic tests are currently divided in 4 different categories:

  • infrastructure-healthcheck: tests from OOM checking the ONAP namespace, certificates…

  • healthcheck: basic tests on components

  • smoke tests: end to end tests

  • security tests

A dashboard summarizing the status and providing the links to the test result page or the logs is automatically created at the end of the execution of the tests.

_images/test-dashboard.png

Test dashboard (Guilin version)

All the pages and artifacts are pushed to LF backend:

A video has been recorded to help launching some of the automated tests on ONAP Guilin. See Running ONAP tests in Guilin Video

Infrastructure Healthcheck Tests
Infrastructure Healthcheck Tests

Tests

Description

Code

Comments

onap-helm

Verify Helm chart status, the test has been updated to take into account Helm3

python module

onap-k8s

Check common resources of the ONAP Kubernetes namespace

python module

kubernetes python library

nodeport_check_certs

This test list the nodeports and tries to get SSL information to evaluate the validity of the certificates (expiration and issuer) used on the nodeports

python module

pyopenssl, kubernetes python libraries

internal_check_certs

This test list the internal ports and tries to get SSL information to evaluate the validity of the certificates (expiration and issuer) used

python module

pyopenssl, kubernetes python libraries

See Infrastructure Healthcheck README to adapt then run infrastructure healthcheck tests on your own system.

Please note that the onap-k8s is run 2 times in CD chains. It is run just after the installation (onap-k8s) and at the end of the test execution (onap-k8s-teardown) in order to collect the logs of the different components during the test execution.

_images/test-onap-k8s.png
Healthcheck Tests
Healthcheck Tests

Tests

Description

Code

Comments

core

Robot healthcheck tests of the core components (AA&I, DMAAP, Portal, SDC, SDNC, SO)

robot tests

robotframework

full

Robot healthcheck tests for all the components, holmes healthcheck have been reintroduced

robot tests

robotframework

healthdist

Check the onboarding and distribution of the vFW

robot tests

robotframework

postinstall

Check dmaap and AA&I Design model DB tests

robot tests

robotframework

ves-collector (new)

Suite for checking handling events by VES Collector

code

robotframework

hv-ves

HV-VES ‘Sunny Scenario’ Robot Framework test - message is sent to the collector and Kafka topic is checked if the message has been published. Content is decoded and checked.

code

robotframework

basic_cds

CDS blueprint enrichment, open a nodeport on CDS then enrich CDS CBA

code

onap_pythonsdk, pythonsdk-tests

basic_onboard

onboard a model, subset of most of the other basic_* tests, created to perform stability testing

code

onap_pythonsdk, pythonsdk-tests

dcaemod

Check new DCAEmod

robot tests

robotframework

cps-healthcheck

Call liveness and readiness probes of the CPS module

robot tests

robotframework

See Healthcheck README to adapt then run healthcheck tests on your own system.

Smoke Tests
Smoke Tests

Tests

Description

Code

Comments

basic_vm

Onboard, distribute and instantiate an Openstack VM using à la carte BPMN, replaced the former basic_vm test

code

onap_pythonsdk, pythonsdk-tests

basic_network

Onboard, distribute and instantiate a Neutron network

code

onap_pythonsdk, pythonsdk-tests

basic_cnf

Onboard (new), distribute and instantiate a Kubernetes pods

code

onap_pythonsdk, pythonsdk-tests

5gbulkpm

5G Bulk PM Usecase functionality. The test has been significantly enhanced in Honolulu

code

robotframework

pnf-registrate

Executes the PNF registration test cases including setup and teardown

code

robotframework

cmpv2

CMPv2 Usecase functionality

code

robotframework

basic_clamp

distribute a model, enrich it with a tca blueprint, call Clamp to create a loop then instantiate it in Policy and DCAE

code

onap_pythonsdk, pythonsdk-tests

basic_vm_macro

Instantiate a VM using macro bpmn

code

onap_pythonsdk, pythonsdk-tests

There are 2 main families of smoke tests:

  • RobotFramework based tests, usually run from inside the cluster as a k8s job

  • Pythonsdk based tests. These tests (also known as onaptests) are consuming several SDKs: the Openstack and Kubernetes SDK for the management of the cloud resources and the python ONAP SDK for the interactions with ONAP

To launch the the robot based tests, please see Robot smoke test README Standard Robot html pages are generated. See Robot page.

To launch the pythonsdk based tests, please see Python smoke test README

An html page is generated by the pythonsdk-test tests.

_images/test-basic-cnf.png
Security Tests
Security Tests

Tests

Description

Code

Comments

root_pods

check that pods are nor using root user or started as root

bash script

kubectl

unlimitted_pods

check that limits are set for pods

bash script

kubectl

cis_kubernetes

perform the k8s cis test suite (upstream src aquasecurity)

bash script

kube-bench

nonssl_endpoints

check that all public HTTP endpoints exposed in ONAP cluster use SSL tunnels

Go script

kubetl, nmap

http_public_endpoints

check that there is no public http endpoints exposed in ONAP cluster

bash script

kubectl,nmap

jdpw_ports

check that there are no internal java ports

bash script

kubectl, procfs

kube_hunter

security suite to search k8s vulnerabilities (upstream src aquasecurity)

kube-Hunter

kube-Hunter

versions

check that Java and Python are available only in versions recommended by SECCOM. This test is long and run only in Weekly CI chains

python module

cerberus, kubernetes python lib,

tern

Check the component licenses within the ONAP dockers

bash script

kubectl

See Security test README to adapt then run the security tests on your own system.

Note for security tests, integration team follows SECCOM recommendations and apply waivers granted by SECCOM if needed through xfail lists.

Stability/Resiliency tests

Ensuring the stability of ONAP is one of the missions of the Integration team. CI chains and stability tests are performed to help stabilising the release. See Integration stability tests for details.

CI/CD

Important

Integration team deals with 2 different CI/CD systems.

  • Jenkins CI/CD, CI managed by LF IT and CD by Integration team

  • GitLab-ci managed by Integration and OOM team

Continuous Integration

The CI part provides the following features:

  • Repository verification (format of the INFO.yaml)

  • Patchset verification thanks to json/yaml/python/go/rst/md linters. These Jenkins verification jobs are hosted in the ci-management repository. They can vote +1/-1 on patchset submission. Integration team systematically enables linters on any new repository

  • Docker build: Integration team builds testsuite dockers and xtesting dockers. These dockers are built then pushed to Nexus through a jjb also hosted in the ci-management repository.

The different verification chains are defined in https://jenkins.onap.org/:

The Jenkins jobs (jjb) are hosted in https://git.onap.org/ci-management/.

Continuous Deployment

There are 2 Continuous Deployment architectures.

Jenkins CD on Windriver/Intel lab

The CD part on Windriver/Intel is based on Jenkins.

It is based on a standalone VM hosting a Jenkins server. The credentials of this VM as well as the Jenkins server have been provided to integration committers.

Several jobs can be triggered from this Jenkins interface. Historically several chains were run daily (staging/release) but due to performance issues, they have all been stopped. Only SB-00 has been kept for use case support. The Jenkins interface was however used to launch the installation of SB-00.

This Jenkins script is leveraging resources available in OOM and integration repositories.

The replacement of this CD by a GitLab runner based CD to unify the CD management was planned, but finalizing the operation in Guilin was not possible due to performance issues.

GitLab CD

This CD is leveraging public gitlab-ci mechanism and used to deploy several ONAP labs:

  • Daily Master: daily run using OOM Master

  • Daily Guilin: daily run using the last stable version during Honolulu Release processing

  • Daily Honolulu: daily run setup at RC0 (candidate dockers available for integration)

  • Daily Istanbul: daily run setup at RC0 (candidate dockers available for integration)

  • Weekly Master: run once a week with longer tests

  • Weekly Istanbul: run once a week with longer tests

  • Gating: run on OOM, clamp or SO patchset submission. It means a full ONAP deployment on demand based on new patchset declared in gerrit.

See Integration CI guideline for details.

Simulators

Simulators are regularly created for use cases. The goal of this section is to:

  • Highlight the existing Simulators

  • Provide recommendations when starting developing a new simulator

Important

Before developing a new simulator, check that it does not exist…and refactor/contribute to existing simulators rather than recreating new ones.

Existing simulators

Simulators

Name

Description

Link

Contacts

NF Simulator

Evolution of the pnf simulator, the Network service simulator

official doc

K.Kuzmicki

A1 Policy Enforcement Simulator

Simulator that supports the A1-P OSC_2.1.0 interface and also provides internal API to manage the RAN elements (Cells, Ues) and allows to customize and send VES Events

official readme

Krystian Kędroń

Mass PNF Simulator

Mimic the PNF for benchmark purposes

official readme

Tamas Bakai

Ran simulator

RAN-SIM is a Radio Access Network Simulator, it is used to simulate the various functionalities of an eNodeB

official readme

Priyadharshini B

DC simulator

Data Center simulator

official readme

Xin Miao

Recommendations

The simulator code

We recommend to create a dedicated repository (ask Integration team).

Simulator repositories

Repository

Description

Link

integration/simulators/A1-policy-enforcement-simulator

A1 Policy Enforcement Simulator

link

integration/simulators/5G-core-nf-simulator

5G core nf simulator

link

integration/simulators/core-nssmf-simulator

Core NSSMF Simulator

link

integration/simulators/nf-simulator

NF simulator

link

integration/simulators/nf-simulator/avcn-manager

NF simulator avcn manager

link

integration/simulators/nf-simulator/netconf-server

NF simulator netconf server

link

integration/simulators/nf-simulator/pm-https-server

NF simulator pm https server

link

integration/simulators/nf-simulator/ves-client

NF simulator ves client

link

integration/simulators/pnf-simulator

PNF Simulator

link

integration/simulators/ran-simulator

RAN simulator

link

integration/simulators/ran-nssmf-simulator

RAN NSSMF simulator

link
Dockerization

From this repository, create a jenkins job to automatically build the dockers.

Helm Chart

It is recommended to create a helm chart in order to run the simulators.

Wrapper for simulators

1. In order to deploy the Helm release with a simulator, place a YAML file describing the Helm release in src/onaptests/templates/helm_charts.

The structure of the YAML file should be like in the example below. Dependencies contain all the charts that need to be pulled.

# Helm release information
api_version:       # API_VERSION
app_version:       # APP_VERSION
chart_name:        # SIMULATOR_NAME
version:           # CHART_VERSION

# Helm charts that need to be pulled
dependencies:
- name:            # SIMULATOR_NAME
  version:         # CHART_VERSION
  repository:      # URL
  local_repo_name: # REPO_NAME

  1. Install the Helm release:

from onaptests.steps.wrapper.helm_charts import HelmChartStep

chart = HelmChartStep(
    cleanup         = BOOLEAN,
    chart_info_file = YAML_FILE_NAME  # name, not the path
)
chart.execute()

  1. Start the simulator via an API call:

 start = SimulatorStartStep(
     cleanup   = BOOLEAN,
     https     = BOOLEAN,
     host      = HOSTNAME,
     port      = PORT,
     endpoint  = START_ENDPOINT,  # if applicable
     method    = REQUEST_METHOD,  # GET, POST etc.
     data      = PAYLOAD  # {"json": {...}, ...}
)
start.execute()

  1. Undeploy the Helm release:

chart.cleanup()

Tooling

Important

Integration team deals with lots of tools to complete its missions. The goal of this section is to highlight some of them and redirect to their official documentation. These tools can be used for CI/CD, Testing or platform management.

Upstream tools are privileged but when needed specific developments can be done.

Please note that none of these tools are imposed to test developers, in other words, any kind of test is accepted and can be integrated, the list of tools is just indicative.

Integration Project

Integration portal

A portal is built to report the status of the different labs collaborating in Integration, see http://testresults.opnfv.org/onap-integration/

_images/ONAP_CI_3.png

The code of this web site is shared on a public gitlab project.

Communication channels

The main communication channel for real time support is the official ONAP Slack #integration-team chan (https://onapproject.slack.com/).

You can also send a mail to onap-discuss AT lists.onap.org with [ONAP] [Integration] prefix in the title.

Testing

Robotframework

robotframework is a well known test framework. Lots of ONAP tests are leveraging this framework. This framework is fully developed upstream even if some extensions (python modules) were created especially to deal with OpenStack (see python-testing-utils project).

Some GUI tests (using Robotframework Selenium extension) had been initiated but not maintained, as a consequence they are not integrated in CI/CD.

Python-onapsdk

The Openstack and Kubernetes python SDK are references widely adopted by the developers and the industry. Developing a python ONAP SDK aimed to follow the examples of the infrastructure SDK with the same expectations in term of code quality. After an evaluation of the CLI project (JAVA SDK re-exposing primitives through python system calls), and a first prototype (onap_tests used until Frankfurt for end to end tests) it was decided to develop a new python SDK.

This SDK has been developed in gitlab.com to benefit from the numerous built-in options offered by gitlab and ensure the best possible code quality.

The project is fully Open Source, released under the Apache v2 license. Integration committers are invited to join the project. The main maintainers are ONAP integration and OOM committers.

Any new feature shall respect the code quality criteria:

  • unit test coverage > 98%

  • functional tests (several components mock objects have been developed)

Attention

Python-onapsdk is a SDK, it means it is a tool allowing to communicate with ONAP. It is a middleware that can be used by test projects but it is NOT a test.

A compagnon project has been created in ONAP: pythonsdk-tests.

The pythonsdk-test project defines tests based on python-onapsdk.

The tests are hosted in this repository. They consume the different needed SDK: python-onapsdk but also the kubernetes, the OpenStack SDK and or any needed additional middlewares. The project developed the notion of steps that can been combined and reorganized as need to design a test. This project interacts with ONAP only through the python-onapsdk library. The tests are described in The Integration Test page.

The available steps are:

  • [CLAMP] OnboardClampStep: Onboard a SDC including a TCA blueprint

  • [CDS] ExposeCDSBlueprintprocessorNodePortStep: expose CDS blueprint nodeport (Guilin workaround)

  • [CDS] BootstrapBlueprintprocessor: Bootstrap a blueprint processor

  • [CDS] DataDictionaryUploadStep: Upload a Data Dictionary to CDS

  • [CDZ] CbaEnrichStep: Enrich CBA

  • [K8S plugin] K8SProfileStep: Create K8S profile

  • [SO] YamlTemplateVfModuleAlaCarteInstantiateStep: Instantiate VF module described in YAML using SO a’la carte method

  • [SO] YamlTemplateVlAlaCarteInstantiateStep: Instantiate network link described in YAML using SO a’la carte method.

  • [SO] YamlTemplateVfModuleAlaCarteInstantiateStep: Instantiate VF module described in YAML using SO a’la carte method

  • [SO] YamlTemplateVnfAlaCarteInstantiateStep: Instantiate vnf described in YAML using SO a’la carte method

  • [SO] YamlTemplateServiceAlaCarteInstantiateStep: Instantiate service described in YAML using SO a’la carte method

  • [AAI] ConnectServiceSubToCloudRegionStep: Connect service subscription with cloud region

  • [AAI] CustomerServiceSubscriptionCreateStep: Create customer’s service subscription

  • [AAI] CustomerCreateStep: Create customer

  • [AAI] LinkCloudRegionToComplexStep: Connect cloud region with complex

  • [AAI] ComplexCreateStep: Create complex

  • [AAI] RegisterCloudRegionStep: Register cloud region

  • [SDC] YamlTemplateServiceOnboardStep: Onboard service described in YAML file in SDC

  • [SDC] YamlTemplateVfOnboardStep: Onboard vf described in YAML file in SDC

  • [SDC] YamlTemplateVspOnboardStep: Onboard vsp described in YAML file in SDC

  • [SDC] VendorOnboardStep: Onboard vendor in SDC

You can reuse the existing steps to compose your test and/or code your own step if it is not supported yet.

The procedure to start a test is described in pythonsdk-test README

CI/CD

The CI/CD is key for integration. It consolidates the trustability in the solution by the automated verification of the deployment and the execution of tests. Integration tests complete the component tests (unit and functional known as CSIT tests).

As the tests can be very heterogeneous (framework, language, outputs), the integration team integrates the tests in simple isolated execution context based on docker called xtesting dockers.

Xtesting is a python library harmonizing the way to setup, run, teardown, manage the artifacts, manage the reporting of the tests (automatic push of the results on a DB backend). It was developed by OPNFV functest project. This python library is included in an alpine docker and contains the needed tests, their associated libraries as well as a testcases.yaml listing these tests. These docker files are built on any change in the integration/xtesting repository and daily to take into account the upstream changes.

The integration project manages 5 xtesting dockers, see Integration Test page.

Important

xtesting is a CI/CD framework, neither a test nor a test framework

Testers can provide tests independently from xtesting. However to be part of the CI/CD chains, an integration of the test in xtesting will be required.

The configuration files are provided as volumes and defined in each docker. The use of this CI/CD abstraction for the tests simplify the integration of the test suites in any CI/CD systems and harmonize the inputs and the outputs.

The official documentation can be found on xtesting official web site

The integration team shares a Test Result Database with the OPNFV project. All the test results of the CD are automatically pushed to this database. It is possible to retrieve the results through the Test API associated with this test Database.

The following information are available:

It is possible to get results according to several criteria (version, case name, lab, period, last, CI id,..) See the OPNFV test API documentation.

Any company running ONAP Integration tests can be referenced to push their results to this database. This Database is hosted on a LF OPNFV server. Results are backuped daily. Integration committers can have access to this server.

VNF demo artifacts are hosted in the demo repositories and published in https://nexus.onap.org/content/repositories/releases/org/onap/demo/vnf/.

Integration Missions

Important

The Integration project is in charge of:

  • Providing testing environment

  • Supporting the use case teams

  • Managing ONAP CI/CD chains

  • Developing tests

  • Providing baseline images

  • Validating the ONAP releases

The different activities may be summarized as follows (proportions are indicative):

  • Community support

  • Lab support

  • Use case support

  • Test development

  • Management of daily/weekly CI chains

  • Build baseline images

  • Automate tests

  • Validate the release

For each release, the integration team provides the following artifacts:

  • A daily CI chain corresponding to the release

  • Staging labs to perform the pairwise testing (when not automated) and support the use case teams

  • Baseline Java and Python images

  • oparent library to manage Java dependencies

  • Test suites and tools to check the various ONAP components

  • Use-case documentation and artifacts

  • A testsuite docker included in the ONAP cluster to execute the robot based tests

  • Configuration files (scripts, Heat templates, CSAR files) to help installing and testing ONAP

  • Wiki release follow-up tables (blocking points, docker versions,…)

Please see the integration wiki page for details.