Tests

Important

Integration is in charge of several types of tests:

Use Cases: developed by use case teams, usually complex, demonstrating high value capabilities of ONAP. They may be partially automated and even integrated in CD.
CSIT Tests: functional tests created by the projects, partially hosted in CSIT repository
Automatic Test Cases: these use cases are usually more simple and aim to validate that ONAP is working properly. These tests have been developed to validate ONAP as a software solution. In theory all the main functions shall be covered by such tests in order to have more robust CI/CD and then avoid regressions. These tests are usually developed and maintained by the integration team.

We may also indicate that when the development of the test framework python-onapsdk follows standard development quality rules and imposes the creation of unit/functional/integration tests. As an example python-onapsdk requires a unit test coverage of 98% before merging a new feature, which is far above the project criteria in SonarCloud today.

Use Cases

The use cases of the last release are described in Verified Use cases.

CSIT Tests

The CSIT tests are functional tests executed by the projects on mocked environment to validate their components. Historically it was hosted in a CSIT repository.

Integration team invited the projects to bring back such tests back to home repository for 2 main reasons:

integration cannot be a bottleneck: +2/merge from integration needed for each project
most of the tests are abandoned and not maintained when hosted in a third party repository leading to CI/CD resource waste and misleading test reporting

Automated Tests

These tests are run daily/weekly on each new gate (new patchset in OOM, CLAMP or SO). They can be in any language (bash, go, python,…), leveraging any test framework (robotframework, MTS, python-onapsdk). They are all embedded in xtesting dockers.

Hint

Automatic tests are currently divided in 4 different categories:

infrastructure-healthcheck: tests from OOM checking the ONAP namespace, certificates…
healthcheck: basic tests on components
smoke tests: end to end tests
security tests

A dashboard summarizing the status and providing the links to the test result page or the logs is automatically created at the end of the execution of the tests.

_images/test-dashboard.png — Test dashboard (Guilin version)

All the pages and artifacts are pushed to LF backend:

Daily chains: https://logs.onap.org/onap-integration/daily
Weekly chains: https://logs.onap.org/onap-integration/weekly
Gating chains: the result link is indicated in gerrit

A video has been recorded to help launching some of the automated tests on ONAP Guilin. See Running ONAP tests in Guilin Video

Infrastructure Healthcheck Tests

Infrastructure Healthcheck Tests
Tests	Description	Code	Comments
onap-helm	Verify Helm chart status, the test has been updated to take into account Helm3	python module
onap-k8s	Check common resources of the ONAP Kubernetes namespace	python module	kubernetes python library
nodeport_check_certs	This test list the nodeports and tries to get SSL information to evaluate the validity of the certificates (expiration and issuer) used on the nodeports	python module	pyopenssl, kubernetes python libraries

See Infrastructure Healthcheck README to adapt then run infrastructure healthcheck tests on your own system.

Please note that the onap-k8s is run 2 times in CD chains. It is run just after the installation (onap-k8s) and at the end of the test execution (onap-k8s-teardown) in order to collect the logs of the different components during the test execution.

Healthcheck Tests

Healthcheck Tests
Tests	Description	Code	Comments
core	Robot healthcheck tests of the core components (AA&I, DMAAP, Portal, SDC, SDNC, SO)	robot tests	robotframework
full	Robot healthcheck tests for all the components, holmes healthcheck have been reintroduced	robot tests	robotframework
healthdist	Check the onboarding and distribution of the vFW	robot tests	robotframework
postinstall	Check dmaap and AA&I Design model DB tests	robot tests	robotframework
ves-collector (new)	Suite for checking handling events by VES Collector	code	robotframework
hv-ves	HV-VES ‘Sunny Scenario’ Robot Framework test - message is sent to the collector and Kafka topic is checked if the message has been published. Content is decoded and checked.	code	robotframework
basic_onboard	onboard a model, subset of most of the other basic_* tests, created to perform stability testing	code	onap_pythonsdk, pythonsdk-tests
dcaemod	Check new DCAEmod	robot tests	robotframework
cps-healthcheck	Call liveness and readiness probes of the CPS module	robot tests	robotframework

See Healthcheck README to adapt then run healthcheck tests on your own system.

Smoke Tests

Smoke Tests
Tests	Description	Code	Comments
basic_vm	Onboard, distribute and instantiate an Openstack VM using à la carte BPMN, replaced the former basic_vm test	code	onap_pythonsdk, pythonsdk-tests
basic_network	Onboard, distribute and instantiate a Neutron network	code	onap_pythonsdk, pythonsdk-tests
basic_cnf	Onboard (new), distribute and instantiate a Kubernetes pods	code	onap_pythonsdk, pythonsdk-tests
5gbulkpm	5G Bulk PM Usecase functionality. The test has been significantly enhanced in Honolulu	code	robotframework
pnf-registrate	Executes the PNF registration test cases including setup and teardown	code	robotframework
cmpv2	CMPv2 Usecase functionality	code	robotframework
basic_vm_macro	Instantiate a VM using macro bpmn	code	onap_pythonsdk, pythonsdk-tests
pnf_macro	Run PNF simulator, onboard, distribute and instantiate service including PNF	code	onap_pythonsdk, pythonsdk-tests
cds_resource_resolution	Upload blueprint file into CDS, execute test workflow and check results	code	onap_pythonsdk, pythonsdk-tests
basic_cnf_macro	Onboard (new), distribute and instantiate a Kubernetes pods using SO’s macro flow	code	onap_pythonsdk, pythonsdk-tests

There are 2 main families of smoke tests:

RobotFramework based tests, usually run from inside the cluster as a k8s job
Pythonsdk based tests. These tests (also known as onaptests) are consuming several SDKs: the Openstack and Kubernetes SDK for the management of the cloud resources and the python ONAP SDK for the interactions with ONAP

To launch the the robot based tests, please see Robot smoke test README Standard Robot html pages are generated. See Robot page.

To launch the pythonsdk based tests, please see Python smoke test README

An html page is generated by the pythonsdk-test tests.

Security Tests

Security Tests
Tests	Description	Code	Comments
root_pods	check that pods are nor using root user or started as root	bash script	kubectl
unlimitted_pods	check that limits are set for pods	bash script	kubectl
cis_kubernetes	perform the k8s cis test suite (upstream src aquasecurity)	bash script	kube-bench
nonssl_endpoints	check that all public HTTP endpoints exposed in ONAP cluster use SSL tunnels	Go script	kubetl, nmap
http_public_endpoints	check that there is no public http endpoints exposed in ONAP cluster	bash script	kubectl,nmap
jdpw_ports	check that there are no internal java ports	bash script	kubectl, procfs
kube_hunter	security suite to search k8s vulnerabilities (upstream src aquasecurity)	kube-Hunter	kube-Hunter
versions	check that Java and Python are available only in versions recommended by SECCOM. This test is long and run only in Weekly CI chains	python module	cerberus, kubernetes python lib,
tern	Check the component licenses within the ONAP dockers	bash script	kubectl

See Security test README to adapt then run the security tests on your own system.

Note for security tests, integration team follows SECCOM recommendations and apply waivers granted by SECCOM if needed through xfail lists.

Stability tests

Ensuring the stability of ONAP is one of the missions of the Integration team. CI chains and stability tests are performed to help stabilising the release. See Integration stability tests for details.