CPS Temporal Documentation
CPS Temporal Overview
CPS Temporal is a service component belonging to CPS overall system.
This service is responsible to provide a time oriented perspective for operational network data. It provides features to store and retrieve sequences of configurations or states along with the associated times when they occurred or have been observed.
CPS Temporal Architecture
CPS Temporal is a dedicated service, distinct and decoupled from CPS Core. CPS Temporal service is an independently deployable unit.
Integration between Core and Temporal is event notification based, asynchronous, send and forget. By doing this, we are avoiding the dependency from CPS Core on CPS Temporal and its API. It reverses the dependency, which makes more sense from a conceptual point of view.
For each data modification handled by CPS Core,
CPS Core is publishing, to a dedicated topic, an event representing the data configuration or state.
CPS Temporal is listening to the same topic for the event and is responsible to keep track of all data over time.
In the future, some other services can be created to listen to the same topic to implement additional functionalities or storage forms.
The event messaging system for this integration is Kafka, whose running instance is deployed independently from CPS. It could be either:
the ONAP Kafka instance from ONAP DMaaP component,
or any specific Kafka instance deployed independently from ONAP
The following diagram is the C4 Model representing CPS System Containers:
CPS Temporal Design
Exposed APIs
CPS Temporal is providing a REST HTTP API to query historical CPS data. Its OPEN API Specification can be found either:
In
openapi.ymlfileAt
https://<cps-temporal-host>:<cps-temporal-port>/swagger/openapi.ymlendpoint available on CPS Temporal running instance
Swagger UI is also available at:
https://<cps-temporal-host>:<cps-temporal-port>/swagger-ui.html
And following Postman collection can be used to send requests to any running instance:
Event Integration
CPS Core and CPS Temporal are integrated with an event driven architecture. Integration between these two components is event notification based.
For each data modification handled by CPS Core,
CPS Core is publishing, to a dedicated Kafka topic, an event representing the data configuration or state.
CPS Temporal is listening to the same topic for the event and is responsible to keep track of all data over time.
Refer to CPS Temporal Modeling for more details on the event structure.
CPS Temporal Modeling
Event Structure
Data manipulated by both CPS Core and CPS Temporal to represent a Data Updated Event is a JSON structure that is defined by following Json Schema:
And following is an example of an event compliant with this schema:
{
"schema": "urn:cps:org.onap.cps:data-updated-event-schema:v2",
"id": "38aa6cc6-264d-4ede-b534-18f5c1f403ea",
"source": "urn:cps:org.onap.cps",
"type": "org.onap.cps.data-updated-event",
"content": {
"observedTimestamp": "2021-06-09T13:00:00.123-0400",
"operation": "UPDATE",
"dataspaceName": "my-dataspace",
"schemaSetName": "my-schema-set",
"anchorName": "my-anchor",
"data": {
"interface": {
"name": "itf-1",
"status": "up"
}
}
}
}
Event versions
The following table lists the data-updated-event schema evolution over releases :
Version
Release
Compatibility Type (with previous version)
Upgrade First
v1
Istanbul
n/a
Any order
v2
Jakarta
Backward
Consumer (Temporal)
Compatibility Types
Several compatibility types exist when an event schema definition is evolving from one release to the next one:
Backward compatibility means that consumers using the new schema can read data produced with the previous schema.
Forward compatibility means that data produced with a new schema can be read by consumers using the previous schema.
Full compatibility means that schemas are both backward and forward compatible: old data can be read with the new schema, and new data can also be read with the previous schema.
CPS Temporal Deployment
Deployment
Refer to CPS-Deployment page for deployment documentation related to CPS Temporal and all CPS components.
Once CPS Temporal is successfully deployed and running 2 pods are started, one for running the service and another one for running the database instance:
NAME READY STATUS RESTARTS AGE
cps-temporal-d4cf495b9-bbn7b 1/1 Running 0 8h
cps-temporal-db-0 1/1 Running 0 8h
Configuration
Application Properties
The following table lists some properties that can be specified as Helm chart values to configure the application to be deployed. This list is not exhaustive.
Property |
Description |
Default Value |
|---|---|---|
config.appUserName |
User name used by CPS Temporal service to configure the authentication for REST API it exposes. This is the user name to be used by CPS Temporal REST clients to authenticate themselves. |
|
config.appUserPassword |
Password used by CPS Temporal service to configure the authentication for REST API it exposes. This is the password to be used by CPS Temporal REST clients to authenticate themselves. If not defined, the password is generated when deploying the application. See also CPS Credentials Retrieval |
Not defined |
timescaledb.config.pgUserName |
Internal user name used bt CPS Temporal to connect to its own database. |
|
timescaledb.config.pgUserPassword |
Internal password used bt CPS Temporal to connect to its own database. If not defined, the password is generated when deploying the application. See also Credentials section. |
Not defined |
config.eventConsumption. spring.kafka.bootstrap-servers |
Kafka hostname and port |
|
config.eventConsumption. spring.kafka.consumer.group-id |
Kafka consumer group id |
|
config.eventConsumption. app.listener.data-updated.topic |
Kafka topic to listen to |
|
config.eventConsumption. spring.kafka.security.protocol |
Kafka security protocol. Some possible values are:
|
|
config.eventConsumption. spring.kafka.properties. sasl.mechanism |
Kafka security SASL mechanism. Required for SASL_PLAINTEXT and SASL_SSL protocols. Some possible values are:
|
Not defined |
config.eventConsumption. spring.kafka.properties. sasl.jaas.config |
Kafka security SASL JAAS configuration. Required for SASL_PLAINTEXT and SASL_SSL protocols. Some possible values are:
|
Not defined |
config.eventConsumption. spring.kafka.ssl.trust-store-type |
Kafka security SASL SSL store type. Required for SASL_SSL protocol. Some possible values are:
|
Not defined |
config.eventConsumption. spring.kafka.ssl.trust-store-location |
Kafka security SASL SSL store file location. Required for SASL_SSL protocol. |
Not defined |
config.eventConsumption. spring.kafka.ssl.trust-store-password |
Kafka security SASL SSL store password. Required for SASL_SSL protocol. |
Not defined |
config.eventConsumption. spring.kafka.properties. ssl.endpoint.identification.algorithm |
Kafka security SASL SSL broker hostname identification verification. Required for SASL_SSL protocol. Possible value is:
|
Not defined |
config.additional. app.query.response.max-page-size |
Maximum number of elements that can be retrieved by a single REST API query request using pagination feature. |
|
config.additional. spring.datasource.hikari. maximumPoolSize |
Maximum number of database connections in the connection pool. |
|
Credentials
- Once the deployment is completed, refer to CPS Credentials Retrieval
for more information related to credentials retrieval.
Running With Docker
For development purposes, CPS Temporal can be ran on any environment using Docker. Refer to README.md and docker-compose.yml files for more details.
Upgrade from Istanbul to Jakarta
Cps data-updated-event schema v2
Cps data-updated-event schema v2 used in Jakarta is backward compatible with data-updated-event schema v1 used in Istanbul.
It means that consumers using the schema v2 can process events generated by producers using the schema v2 or v1.
This implies that cps temporal (consumer) must be upgraded from Istanbul to Jakarta at the same time or before cps core (producer) is upgraded from Istanbul to Jakarta.
Database
In Jakarta, CPS Temporal database is upgraded from TimescaleDB 2.1.1 running PostgresSQL 13.2 to
TimescaleDB 2.5.1 running PosgresSQL 14.1. This is a major PostgresSQL upgrade subject to change data storage
format. Then, any existing CPS Temporal data from Istanbul needs to be migrated before it can be used in Jakarta.
The migration needs to be done in 2 main sequential steps for both TimescaleDB and PostgresSQL:
Upgrade TimescaleDB from 2.1.1 to 2.5.1
Upgrade PostgresSQL form 13.2 to 14.1
Bellow are the detailed steps to be completed for data migration (manual docker steps, to be reviewed for production upgrade).
Stop CPS Temporal Istanbul service.
Backup 2.1.1-pg13 data. This backup is to be used to restore data if needed.
# Start timescale 2.1.1-pg13 db instance
docker run --name postgres-cps-temporal-2.1.1-13 -d \
-e POSTGRES_DB=cpstemporaldb \
-e POSTGRES_USER=cpstemporal \
-e POSTGRES_PASSWORD=cpstemporal \
-v cps-temporal_data:/var/lib/postgresql/data \
-v cps-temporal_backup-2.1.1-13:/var/lib/postgresql/backup \
timescale/timescaledb:2.1.1-pg13
docker exec -it postgres-cps-temporal-2.1.1-13 psql -d cpstemporaldb -U cpstemporal -c "select version();"
docker exec -it postgres-cps-temporal-2.1.1-13 psql -d cpstemporaldb -U cpstemporal -c "\dx timescaledb"
# Backup 2.1.1-pg13 data
docker exec -it postgres-cps-temporal-2.1.1-13 pg_basebackup -U cpstemporal -D /var/lib/postgresql/backup/
docker exec -it postgres-cps-temporal-2.1.1-13 ls -l /var/lib/postgresql/backup/
# Stop db instance
docker container stop postgres-cps-temporal-2.1.1-13
docker container rm postgres-cps-temporal-2.1.1-13
Upgrade data from 2.1.1-pg13 to 2.5.1-pg-13
# Start timescale 2.5.1.-pg13
docker run --name postgres-cps-temporal-2.5.1-13 -d \
-e POSTGRES_DB=cpstemporaldb \
-e POSTGRES_USER=cpstemporal \
-e POSTGRES_PASSWORD=cpstemporal \
-v cps-temporal_data:/var/lib/postgresql/data \
-v cps-temporal_dump-2.5.1-13:/var/lib/postgresql/dump \
timescale/timescaledb:2.5.1-pg13
# Upgrade data to 2.5.1-pg13
docker exec -it postgres-cps-temporal-2.5.1-13 psql -d cpstemporaldb -U cpstemporal -c "\dx timescaledb"
docker exec -it postgres-cps-temporal-2.5.1-13 psql -X -d cpstemporaldb -U cpstemporal -c "ALTER EXTENSION timescaledb UPDATE;"
docker exec -it postgres-cps-temporal-2.5.1-13 psql -d cpstemporaldb -U cpstemporal -c "\dx timescaledb"
Create a dump of 2.5.1-pg-13 data
# Dump 2.5.1-pg13 data
docker exec -it postgres-cps-temporal-2.5.1-13 pg_dump -d cpstemporaldb -U cpstemporal -Fc -f /var/lib/postgresql/dump/dumpfile
docker exec -it postgres-cps-temporal-2.5.1-13 ls -l /var/lib/postgresql/dump/dumpfile
# Stop db instance
docker container stop postgres-cps-temporal-2.5.1-13
docker container rm postgres-cps-temporal-2.5.1-13
Upgrade data from 2.5.1-pg13 to 2.5.1-pg-14, by importing the dump in PostgresSQL 14 instance
# Start timescale 2.5.1-pg14
docker run --name postgres-cps-temporal-2.5.1-14 -p 5432:5432 -d \
-e POSTGRES_DB=cpstemporaldb \
-e POSTGRES_USER=cpstemporal \
-e POSTGRES_PASSWORD=cpstemporal \
-v cps-temporal_data-2.5.1-14:/var/lib/postgresql/data \
-v cps-temporal_dump-2.5.1-13:/var/lib/postgresql/dump \
timescale/timescaledb:2.5.1-pg14
docker exec -it postgres-cps-temporal-2.5.1-14 psql -d cpstemporaldb -U cpstemporal -c "select version();"
docker exec -it postgres-cps-temporal-2.5.1-14 psql -d cpstemporaldb -U cpstemporal -c "\dx timescaledb"
# Upgrade data to 2.5.1-pg14 by restoring the dump
docker exec -it postgres-cps-temporal-2.5.1-14 pg_restore -d cpstemporaldb -U cpstemporal -Fc /var/lib/postgresql/dump/dumpfile
docker exec -it postgres-cps-temporal-2.5.1-14 psql -d cpstemporaldb -U cpstemporal -c "select count(*) from network_data;"
# Stop db instances
docker container stop postgres-cps-temporal-2.5.1-14
docker container rm postgres-cps-temporal-2.5.1-14
Copy 2.5.1-pg14 data to data volume
# Start busybox
docker run -it --rm \
-v cps-temporal_data:/data \
-v cps-temporal_data-2.5.1-14:/data-2.5.1-14 \
busybox:1.34.1
# Run commands in busybox
rm -rf /data/*
cp -rp /data-2.5.1-14/* /data
diff -r /data /data-2.5.1-14
exit
Start Cps Temporal Jakarta service
Cleanup volumes that are not needed anymore
For more details about TimescaleDB and PostgresSQL upgrades, refer to:
CPS Temporal Admin Guide
Configuration
Refer to CPS Temporal Deployment for application properties available to configure the application.
Logging
Refer to CPS Logging for documentation related to CPS Temporal and all CPS components logging information.
Monitoring
Refer to CPS Monitoring page for documentation related to CPS Temporal and all CPS components monitoring information.
CPS Temporal Release Notes
Version: 1.1.0
Docker Image |
onap/cps-temporal:1.1.0 |
Designation |
1.1.0 Jakarta |
Date |
2022 March 15 |
Features
Bug Fixes
Version: 1.0.1
Docker Image |
onap/cps-temporal:1.0.1 |
Designation |
1.0.1 Istanbul |
Date |
2021 January 6 |
Features
None
Bug Fixes
CPS-841 - Update log4j version to 2.17.1 due to security vulnerability
Version: 1.0.0
Docker Image |
onap/cps-temporal:1.0.0 |
Designation |
1.0.0 Istanbul |
Date |
2021 September 14 |
New features
Known Limitations, Issues and Workarounds
None
Security Notes
Known Security Issues:
CPS-488 - Authentication and authorization for REST API is limited to one generic user.
Fixed Security Issues:
None