Policy Drools PDP component
Both the Performance and the Stability tests were executed against an ONAP installation in the policy-k8s tenant in the windriver lab, from an independent VM running the jmeter tool to inject the load.
General Setup
The installation runs the following components in a single VM:
AAF
AAI
DMAAP
POLICY
The VM has the following hardware spec:
126GB RAM
12 VCPUs
155GB Ephemeral Disk
Jmeter is run from a different VM with the following configuration:
16GB RAM
8 VCPUs
155GB Ephemeral Disk
The drools-pdp container uses the JVM memory settings from a default OOM installation.
Other ONAP components exercised during the stability tests were:
Policy XACML PDP to process guard queries for each transaction.
DMaaP to carry PDP-D and jmeter initiated traffic to complete transactions.
Policy API to create (and delete at the end of the tests) policies for each scenario under test.
Policy PAP to deploy (and undeploy at the end of the tests) policies for each scenario under test.
The following components are simulated during the tests.
SO actor for the vDNS use case.
APPC responses for the vCPE and vFW use cases.
AAI to answer queries for the use cases under test.
SO, and AAI actors were simulated within the PDP-D JVM by enabling the feature-controlloop-utils before running the tests.
PDP-D Setup
The kubernetes charts were modified previous to the installation to add the following script that enables the controlloop-utils feature:
oom/kubernetes/policy/charts/drools/resources/configmaps/features.pre.sh:
#!/bin/sh
sh -c "features enable controlloop-utils"
Stability Test of Policy PDP-D
PDP-D performance
The tests focused on the following use cases:
vCPE
vDNS
vFirewall
For 72 hours the following 5 scenarios ran in parallel:
vCPE success scenario
vCPE failure scenario (failure returned by simulated APPC recipient through DMaaP).
vDNS success scenario.
vDNS failure scenario (failure by introducing in the DCAE ONSET a non-existant vserver-name reference).
vFirewall success scenario.
Five threads ran in parallel, one for each scenario, back to back with no pauses. The transactions were initiated by each jmeter thread group. Each thread initiated a transaction, monitored the transaction, and as soon as the transaction ending was detected, it initiated the next one.
JMeter was run in a docker container with the following command:
docker run --interactive --tty --name jmeter --rm --volume $PWD:/jmeter -e VERBOSE_GC="" egaillardon/jmeter-plugins --nongui --testfile s3p.jmx --loglevel WARN
The results were accessed by using the telemetry API to gather statistics:
vCPE Success scenario
ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e:
# Times are in milliseconds
Control Loop Name: ControlLoop-vCPE-48f0c2c3-a172-4192-9ae3-052274181b6e
Number of Transactions Executed: 114007
Number of Successful Transactions: 112727
Number of Failure Transactions: 1280
Average Execution Time: 434.9942021103967 ms.
vCPE Failure scenario
ControlLoop-vCPE-Fail:
# Times are in milliseconds
Control Loop Name: ControlLoop-vCPE-Fail
Number of Transactions Executed: 114367
Number of Successful Transactions: 114367 (failure transactions are expected)
Number of Failure Transactions: 0 (success transactions are not expected)
Average Execution Time: 433.61750330077734 ms.
vDNS Success scenario
ControlLoop-vDNS-6f37f56d-a87d-4b85-b6a9-cc953cf779b3:
# Times are in milliseconds
Control Loop Name: ControlLoop-vDNS-6f37f56d-a87d-4b85-b6a9-cc953cf779b3
Number of Transactions Executed: 237512
Number of Successful Transactions: 229532
Number of Failure Transactions: 7980
Average Execution Time: 268.028794334602 ms.
vDNS Failure scenario
ControlLoop-vDNS-Fail:
# Times are in milliseconds
Control Loop Name: ControlLoop-vDNS-Fail
Number of Transactions Executed: 1957987
Number of Successful Transactions: 1957987 (failure transactions are expected)
Number of Failure Transactions: 0 (success transactions are not expected)
Average Execution Time: 39.369322166081794
vFirewall Success scenario
ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a:
# Times are in milliseconds
Control Loop Name: ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a
Number of Transactions Executed: 120308
Number of Successful Transactions: 118895
Number of Failure Transactions: 1413
Average Execution Time: 394.8609236293513 ms.
Commentary
There has been a degradation of performance observed in this release when compared with the previous one. Approximately 1% of transactions were not completed as expected for some use cases. Average Execution Times are extended as well. The unexpected results seem to point in the direction of the interactions of the distributed locking feature with the database. These areas as well as the conditions for the test need to be investigated further.
# Common pattern in the audit.log for unexpected transaction completions
a8d637fc-a2d5-49f9-868b-5b39f7befe25||ControlLoop-vFirewall-d0a1dfc6-94f5-4fd4-a5b5-4630b438850a|
policy:usecases:[org.onap.policy.drools-applications.controlloop.common:controller-usecases:1.9.0:usecases]|
2021-10-12T19:48:02.052+00:00|2021-10-12T19:48:02.052+00:00|0|
null:operational.modifyconfig.EVENT.MANAGER.FINAL:1.0.0|dev-policy-drools-pdp-0|
ERROR|400|Target Lock was lost|||VNF.generic-vnf.vnf-name||dev-policy-drools-pdp-0||
dev-policy-drools-pdp-0|microservice.stringmatcher|
{vserver.prov-status=ACTIVE, vserver.is-closed-loop-disabled=false,
generic-vnf.vnf-name=fw0002vm002fw002, vserver.vserver-name=OzVServer}||||
INFO|Session org.onap.policy.drools-applications.controlloop.common:controller-usecases:1.9.0:usecases|
# The "Target Lock was lost" is a common message error in the unexpected results.
END-OF-DOCUMENT