OOM offline-installer

Offline Installer Package Build Guide

This document describes how to build offline installer packages. The build process should be triggered on a host with internet connectivity. It will retrieve all artifacts required for ONAP deployment based on both - static data list files and dynamically assembled ones. The host used for the procedure in this guide should be preferably a separate build server.

Procedure was completely tested on RHEL 7.9 as it’s the default target installation platform, however with small adaptations it should be applicable also for other platforms. Some discrepancies when Centos 7.9 is used are described below as well.

Part 1. Prerequisites

We assume that procedure is executed on RHEL 7.9 server with ~300G disc space, 16G+ RAM and internet connectivity.

Some additional software packages are required by ONAP Offline platform building tooling. In order to install them following repos have to be configured for RHEL 7.9 platform.

Note

All commands stated in this guide are meant to be run in root shell.

############
# RHEL 7.9 #
############

# Register server
subscription-manager register --username <rhel licence name> --password <password> --auto-attach

# required by custom docker version recommended by ONAP
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

# required by docker dependencies i.e. docker-selinux
subscription-manager repos --enable=rhel-7-server-extras-rpms

# epel is required by npm within blob build
rpm -ivh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

Alternatively

##############
# Centos 7.9 #
##############

# required by custom docker version recommended by ONAP
yum-config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo

# enable epel repo for npm and jq
yum install -y epel-release

Subsequent steps are the same on both platforms:

# install following packages
yum install -y docker-ce-19.03.15 git createrepo expect nodejs npm jq

# install Python 3
yum install -y python36 python36-pip

# ensure docker daemon is running
service docker start

Then it is necessary to clone all installer and build related repositories and prepare the directory structure.

# prepare the onap build directory structure
cd /tmp
git clone https://gerrit.onap.org/r/oom/offline-installer onap-offline
cd onap-offline

# install required pip packages for build and download scripts
pip3 install -r ./build/requirements.txt
pip3 install -r ./build/download/requirements.txt

Part 2. Download artifacts for offline installer

Generate the actual list of docker images that are defined within OOM helm charts. Run the docker-images-collector.sh script (check script for runtime dependencies) from cloned OOM repository.

At the beginning of the generated list file there is the OOM repo commit sha from which it was created - the same commit reference should be used in Part 4. Packages preparation as –application-repository_reference option value.

Following example will create the list to the default path (build/data_lists/onap_docker_images.list):

# clone the OOM repository
git clone https://gerrit.onap.org/r/oom -b <branch> --recurse-submodules /tmp/oom
#run the collector providing path the the project
./build/creating_data/docker-images-collector.sh /tmp/oom/kubernetes/onap

For the list of all available options check script usage info.

Note

replace <branch> with OOM branch you want to build

Note

docker-images-collector.sh script uses oom/kubernetes/onap/resources/overrides/onap-all.yaml file to find what subsystems are enabled. By default all subsystems are enabled there. Modify the file to disable some of them if needed.

Note

Skip this step if you have already all necessary resources and continue with Part 3. Populate local nexus

Create repository containing OS packages to be installed on all nodes:

# run create_repo.sh script to download all required packages with their dependencies
# set destination directory for packages with '-d' parameter
# optionally use '-t' parameter to set target platform (host platform by default)
./offline-installer/build/create_repo.sh -d $(pwd) -t centos|rhel|ubuntu

Note

If script fails due to permissions issue, it could be a problem with SeLinux. It can be fixed by running:

# Change security context of directory
chcon -Rt svirt_sandbox_file_t $(pwd)

Download all required binaries and docker images. Run download.py twice (as shown below) as it does not support mixing downloading docker images to a local directory and pulling them to local docker engine cache in one run. Docker images from infra_docker_images.list need to be saved to resources directory while the rest of the images need to be just pulled locally:

# all data lists are taken from ./build/data_lists/ folder by default
# all resources will be stored in expected folder structure within "../resources" folder
./build/download/download.py --docker ./build/data_lists/infra_docker_images.list ../resources/offline_data/docker_images_infra \
--http ./build/data_lists/infra_bin_utils.list ../resources/downloads \
--http ./build/data_lists/kube_prometheus_stack_http.list ../resources/downloads

# second argument for --docker is not present, images are just pulled and cached
./build/download/download.py --docker ./build/data_lists/rke_docker_images.list \
--docker ./build/data_lists/k8s_docker_images.list \
--docker ./build/data_lists/onap_docker_images.list \
--docker ./build/data_lists/kube_prometheus_stack_docker_images.list

Part 3. Populate local nexus

In order to build nexus blob all docker images required for ONAP offline platform should be available locally (see Part 2).

Note

In case you skipped the Part 2 for the artifacts download, please ensure that the onap docker images are cached and copy of resources data are untarred in ./onap-offline/../resources/

build_nexus_blob.sh script will run the Nexus container and configure it as docker repository. Then it will push all docker images from previously generated list to it. After that the repository container is stopped and its filesystem gets saved to resources directory.

./onap-offline/build/build_nexus_blob.sh

It will load the listed docker images, run the Nexus, configure it as npm, pypi and docker repositories. Then it will push all listed docker images to the repositories. After all is done the repository container is stopped.

Note

By default the script uses data lists from ./build/data_lists/ directory and saves the blob to ../resources/nexus_data.

Note

By default the script uses “nexus” for the container name and binds 8081 and 8082 ports. Should those names/ports be already used please check the script options on how to customize them.

Part 4. Packages preparation

ONAP offline deliverable consist of 3 packages:

Package	Description
sw_package.tar	Contains provisioning software and configuration for infrastructure and ONAP
resources_package.tar	Contains all binary data and config files needed to deploy infrastructure and ONAP
aux_package.tar	Contains auxiliary input files that can be added to ONAP

All packages can be created using build/package.py script. Beside of archiving files gathered in the previous steps, script also builds docker images used on infra server.

From onap-offline directory run:

./build/package.py <helm charts repo> --build-version <version> --application-repository_reference <commit/tag/branch> --output-dir <target\_dir> --resources-directory <target\_dir>

For example:

./build/package.py https://gerrit.onap.org/r/oom --application-repository_reference <branch> --output-dir /tmp/packages --resources-directory /tmp/resources

Note

replace <branch> by branch you want to build

Above command should produce following tar files in the target directory:

sw_package.tar
resources_package.tar
aux_package.tar

Offline Installer - Installation Guide

This document describes offline installation procedure for OOM ONAP, which is done by the ansible based Offline installer.

Before you begin the installation process you should prepare the offline installation packages. Please refer to the Build Guide for instructions on how to create them.

Part 1. Prerequisites

ONAP platform has certain software requirements - see Software requirements and minimum hardware recommendations: 224 GB RAM, 112 vCPUs and 160GB of storage (see Hardware requirements). The minimum count of nodes should not drop below three - otherwise you may have to do some tweaking to make it work, which is not covered here (there is a pod count limit for a single kubernetes node - you can read more about it in this discussion).

Kubernetes cluster

The four nodes/VMs will be running these services:

infra-node:

- nexus
- nginx proxy
- dns
- kubernetes-etcd
- kubernetes-control-plane
- chartmuseum (if using helm v3)

NOTE: kubernetes-* control plane can be colocated directly with k8s nodes and not necessarily on infra node.

kubernetes node 1-3:
```
- kubernetes worker
```

You don’t need to care about these services now - that is the responsibility of the installer (described below). Just start four VMs as seen in below table (or according to your needs as we hinted above):

Kubernetes cluster overview

Note

Offline installer leverages RKE to provision kubernetes cluster. If you’d like to use different k8s installation method please exclude rke.yml ansible playbook from execution and provide your own.

KUBERNETES NODE	OS	NETWORK	CPU	RAM	STORAGE
infra-node	RHEL/CentOS 7.9	`10.8.8.100/24`	`8 vCPUs`	`8 GB`	`100 GB`
kube-node1	RHEL/CentOS 7.9	`10.8.8.101/24`	`16 vCPUs`	`56+ GB`	`100 GB`
kube-node2	RHEL/CentOS 7.9	`10.8.8.102/24`	`16 vCPUs`	`56+ GB`	`100 GB`
kube-node3	RHEL/CentOS 7.9	`10.8.8.103/24`	`16 vCPUs`	`56+ GB`	`100 GB`
SUM			`56 vCPUs`	`176+ GB`	`400 GB`

As of now, the offline installer supports only RHEL 7.x and CentOS 7.9 distributions, with at least @core and @base package groups installed including Mandatory and Default package sets. So, your VMs should be preinstalled with this operating system - the hypervisor and platform can be of your choosing.

We will expect from now on that you installed four VMs and they are connected to the shared network. All VMs must be reachable from install-server (below), which can be the hypervisor, infra-node or completely different host. But in either of these cases the install-server must be able to connect over ssh to all of these nodes.

Install-server

We will use distinct install-server and keep it separate from the four-node cluster. But if you wish so, you can use infra-node for this goal (if you use the default 'chroot' option of the installer), but in that case double the size of the storage requirement!

Prerequisites for the install-server:

packages described in Build Guide
extra 100 GB storage (to have space where to store these packages)
installed 'chroot' and/or 'docker' system commands
network connection to the nodes - especially functioning ssh client

Our install-server will have ip: 10.8.8.4.

NOTE: All the subsequent commands below, are executed from within this install-server.

Part 2. Configuration

All commands and setups described in this chapter MUST be run on the install-server. It’s assumed here that all commands are run as root which is of course not necessary - you can use a regular user account. The ssh/ansible connection to the nodes will also expect that we are connecting as root - you need to elevate privileges to be able to install on them. Although it can be achieved by other means (sudo), we decided here to keep instructions simple.

Installer packages

At this point you should have the installer packages already prepared (see Build Guide):

sw_package.tar
resources_package.tar
aux_package.tar

NOTE: 'aux_package.tar' is optional and if you don’t have use for it, you can ignore it.

Copy above packages to the /data directory on the install-server and then unpack the 'sw_package.tar' to your home directory:

$ mkdir ~/onap-offline-installer
$ tar -C ~/onap-offline-installer -xf /data/sw_package.tar

Application directory

Change the current directory to 'ansible':

$ cd ~/onap-offline-installer/ansible

You can see multiple files and directories inside - those are the offline-installer ansible playbooks.

If you created the 'sw_package.tar' package according to the Build Guide then at least the following files should be present:

application/application_configuration.yml
inventory/hosts.yml

Following paragraphs describe fine-tuning of 'inventory.yml' to reflect your VMs setup and 'application_configuration.yml' to setup the provisioner itself.

hosts.yml

We need to setup the 'hosts.yml' first, the template looks like this:

---
# This group contains hosts with all resources (binaries, packages, etc.)
# in tarball.
all:
  vars:
    # this key is supposed to be generated during setup.yml playbook execution
    # change it just when you have better one working for all nodes
    ansible_ssh_private_key_file: /root/.ssh/offline_ssh_key
    ansible_ssh_common_args: '-o StrictHostKeyChecking=no'

  children:
    resources:
      hosts:
        resource-host:
          ansible_host: 10.8.8.5

    # This is group of hosts where nexus, nginx, dns and all other required
    # services are running.
    infrastructure:
      hosts:
        infrastructure-server:
          ansible_host: 10.8.8.13
          #IP used for communication between infra and kubernetes nodes, must be specified.
          cluster_ip: 10.8.8.13

    # This is group of hosts which are/will be part of Kubernetes cluster.
    kubernetes:
      children:
        # This is a group of hosts containing kubernetes worker nodes.
        kubernetes-node:
          hosts:
            kubernetes-node-1:
              ansible_host: 10.8.8.19
              #ip of the node that it uses for communication with k8s cluster.
              cluster_ip: 10.8.8.19
              # External ip of the node, used for access from outside of the cluster.
              # Can be set to some kind of floating or public ip.
              # If not set, cluster_ip is used for this purpose.
              # external_ip: x.x.x.x

        # Group of hosts containing etcd cluster nodes.
        # Defaults to infra.
        kubernetes-etcd:
          hosts:
            infrastructure-server

        # This is a group of hosts that are to be used as kubernetes control plane nodes.
        # This means they host kubernetes api server, controller manager and scheduler.
        # This example uses infra for this purpose, however note that any
        # other host could be used including kubernetes nodes.
        # cluster_ip needs to be set for hosts used as control planes, external_ip can also be used.
        kubernetes-control-plane:
          hosts:
            infrastructure-server

    nfs-server:
      hosts:
        kubernetes-node-1

There is some ssh configuration under the 'vars' section - we will deal with ssh setup a little bit later in the SSH authentication.

First you need to set the ip addresses and add a couple of kubernetes nodes to match your four-node cluster:

Under the 'resource-host' set the 'ansible_host' address to the ip of the host where the packages are stored - it must be reachable by ssh from the install-server (for ansible to run playbooks on it) AND infra-node (to extract resource data from resource-host to infra-node over ssh). In our scenario the resource-host is the same as the install-server: '10.8.8.4'
Similarly, set the 'ansible_host' to the address of the infra-node under the 'infrastructure-server'.
Copy the whole 'kubernetes-node-1' subsection and paste it twice directly after. Change the numbers to 'kubernetes-node-2' and 'kubernetes-node-3' respectively and fix the addresses in the 'ansible_host' variables again to match kube-node1, kube-node2 and kube-node3.

As you can see, there is another 'cluster_ip' variable for each node - this serve as a designated node address in the kubernetes cluster. Make it the same as the respective 'ansible_host'.

NOTE: In our simple setup we have only one interface per node, but that does not need to be a case for some other deployments - especially if we start to deal with a production usage. Basically, an 'ansible_host' is an entry point for the install-server’s ansible (offline-installer), but the kubernetes cluster can be communicating on a separate network to which install-server has no access. That is why we have this distinctive variable, so we can tell the installer that there is a different network, where we want to run the kubernetes traffic and what address each node has on such a network.

After applying all described changes, the 'hosts.yml' should look similar to this:

---
# This group contains hosts with all resources (binaries, packages, etc.)
# in tarball.
all:
  vars:
    # this key is supposed to be generated during setup.yml playbook execution
    # change it just when you have better one working for all nodes
    ansible_ssh_private_key_file: /root/.ssh/offline_ssh_key
    ansible_ssh_common_args: '-o StrictHostKeyChecking=no'

  children:
    resources:
      hosts:
        resource-host:
          ansible_host: 10.8.8.4

    # This is group of hosts where nexus, nginx, dns and all other required
    # services are running.
    infrastructure:
      hosts:
        infrastructure-server:
          ansible_host: 10.8.8.100
          #IP used for communication between infra and kubernetes nodes, must be specified.
          cluster_ip: 10.8.8.100

    # This is group of hosts which are/will be part of Kubernetes cluster.
    kubernetes:
      children:
        # This is a group of hosts containing kubernetes worker nodes.
        kubernetes-node:
          hosts:
            kubernetes-node-1:
              ansible_host: 10.8.8.101
              #ip of the node that it uses for communication with k8s cluster.
              cluster_ip: 10.8.8.101
              # External ip of the node, used for access from outside of the cluster.
              # Can be set to some kind of floating or public ip.
              # If not set, cluster_ip is used for this purpose.
              # external_ip: x.x.x.x
            kubernetes-node-2:
              ansible_host: 10.8.8.102
              #ip of the node that it uses for communication with k8s cluster.
              cluster_ip: 10.8.8.102
            kubernetes-node-3:
              ansible_host: 10.8.8.103
              #ip of the node that it uses for communication with k8s cluster.
              cluster_ip: 10.8.8.103

        # Group of hosts containing etcd cluster nodes.
        # Defaults to infra.
        kubernetes-etcd:
          hosts:
            infrastructure-server

        # This is a group of hosts that are to be used as kubernetes control plane nodes.
        # This means they host kubernetes api server, controller manager and scheduler.
        # This example uses infra for this purpose, however note that any
        # other host could be used including kubernetes nodes.
        # cluster_ip needs to be set for hosts used as control planes, external_ip can also be used.
        kubernetes-control-plane:
          hosts:
            infrastructure-server

    nfs-server:
      hosts:
        kubernetes-node-1

application_configuration.yml

Here, we will be interested in the following variables:

resources_dir
resources_filename
aux_resources_filename
app_data_path
aux_data_path
app_name
timesync

'resource_dir', 'resources_filename' and 'aux_resources_filename' must correspond to the file paths on the resource-host ('resource-host' in hosts.yml), which in our case is the install-server host.

The 'resource_dir' should be set to '/data', 'resources_filename' to 'resources_package.tar' and 'aux_resources_filename' to 'aux_package.tar'. The values should be the same as are in the Installer packages section.

'app_data_path' is the absolute path on the infra-node to where the package 'resources_package.tar' will be extracted and similarly 'aux_data_path' is another absolute path for 'aux_package.tar'. Both paths are fully arbitrary, but they should point to the filesystem with enough disk space - the storage requirements are described in Overview table of the kubernetes cluster.

NOTE: As we mentioned in Installer packages - the auxiliary package is not mandatory and we will not utilize it in here either.

The 'app_name' variable should be short and descriptive. We will set it simply to onap.

The 'timesync' variable is optional and controls synchronisation of the system clock on hosts. It should be configured only if a custom NTP server is available and needed. Such a time authority should be on a host reachable from all installation nodes. If this setting is not provided then the default behavior is to setup NTP daemon on infra-node and sync all kube-nodes’ time with it.

If you wish to provide your own NTP servers configure their IPs as follows:

timesync:
  servers:
   - <ip address of NTP_1>
   - <...>
   - <ip address of NTP_N>

Another time adjustment related variables are 'timesync.slewclock' and 'timesync.timezone' . First one can have value of 'true' or 'false' (default). It controls whether (in case of big time difference compared to server) time should be adjusted gradually by slowing down or speeding up the clock as required ('true') or in one step ('false'):

timesync:
  slewclock: true

Second one controls time zone setting on host. It’s value should be time zone name according to tz database names with 'Universal' being the default one:

timesync.
  timezone: UTC

'timesync.servers', 'timesync.slewclock' and 'timesync.timezone' settings can be used independently.

Final configuration can resemble the following:

resources_dir: /data
resources_filename: resources_package.tar
app_data_path: /opt/onap
app_name: onap
timesync:
  servers:
    - 192.168.0.1
    - 192.168.0.2
  slewclock: true
  timezone: UTC

Helm chart values overrides

OOM charts are coming with all ONAP components disabled, this setting is also prepackaged within our sw_package.tar. Luckily there are multiple ways supported how to override this setting. It’s also necessary for setting-up VIM specific entries and basically to configure any stuff with non default values.

First option is to use overrides key in application_configuration.yml. These settings will override helm values originally stored in values.yaml files in helm chart directories.

For example, the following lines could be appended to application_configuration.yml to set up managed openstack credentials for onap’s so component:

overrides:
  so:
    config:
      openStackUserName: "os_user"
      openStackRegion: "region_name"
      openStackKeyStoneUrl: "keystone_url"
      openStackEncryptedPasswordHere: "encrypted_password"

In addition or alternatively to that one can configure helm_override_files variable in 'application_configuration.yml' and mention all files with helm chart values there, e.g.:

helm_override_files:
  - "/path/to/values1.yaml"
  - "/path/to/values2.yaml"

SSH authentication

Finally you need to setup password-less login from install-server to the nodes.

You can use the ansible playbook 'setup.yml' for that purpose:

$ ./run_playbook.sh -i inventory/hosts.yml setup.yml -u root --ask-pass

You will be asked for password per each node and the playbook will generate a unprotected ssh key-pair '~/.ssh/offline_ssh_key', which will be distributed to the nodes.

Another option is to generate a ssh key-pair manually. We strongly advise you to protect it with a passphrase, but for simplicity we will showcase generating of a private key without any such protection:

$ ssh-keygen -N "" -f ~/.ssh/identity

The next step will be to distribute the public key to these nodes and from that point no password is needed:

$ for ip in 100 101 102 103 ; do ssh-copy-id -i ~/.ssh/identity.pub root@10.8.8.${ip} ; done

This command behaves almost identically to the 'setup.yml' playbook.

If you generated the ssh key manually then you can now run the 'setup.yml' playbook like this and achieve the same result as in the first execution:

$ ./run_playbook.sh -i inventory/hosts.yml setup.yml

This time it should not ask you for any password - of course this is very redundant, because you just distributed two ssh keys for no good reason.

We can finally edit and finish the configuration of the 'hosts.yml':

if you used the 'setup.yml' playbook then you can just leave this line as it is:
```
ansible_ssh_private_key_file: /root/.ssh/offline_ssh_key
```
if you created a ssh key manually then change it like this:
```
ansible_ssh_private_key_file: /root/.ssh/identity
```

Part 3. Installation

We should have the configuration complete and be ready to start the installation. The installation is done via ansible playbooks, which are run either inside a chroot environment (default) or from the docker container. If for some reason you want to run playbooks from the docker instead of chroot then you cannot use infra-node or any other kube-node as the install-server - otherwise you risk that installation will fail due to restarting of the docker service.

'sw_package.tar' should contain 'ansible_chroot.tgz' file inside the 'docker' directory. Detailed instructions on how to create it manually and to get more info about the scripts dealing with docker and chroot, go to Appendix 1. Ansible execution/bootstrap.

We will use the default chroot option so we don’t need any docker service to be running.

Commence the installation process by running following command:

$ ./run_playbook.sh -i inventory/hosts.yml -e @application/application_configuration.yml site.yml

This will take a while so be patient. The whole provisioning process is idempotent so you may safely re-run it if required.

'site.yml' playbook will run following playbooks in the given order:

resources.yml
infrastructure.yml
rke.yml
nfs.yml
kube_prometheus.yml
cert_manager.yml
application.yml

Part 4. Post-installation and troubleshooting

After all of the playbooks are run successfully the ONAP kubernetes application will be still deploying and it might take some time until all pods are up and running. You can monitor your newly created kubernetes cluster with this command:

$ ssh -i ~/.ssh/offline_ssh_key root@10.8.8.100 # tailor this command to connect to your infra-node
$ watch -d -n 5 'kubectl get pods --all-namespaces'

Alternatively you can monitor progress with helm_deployment_status.py script located in offline-installer directory. Transfer it to infra-node and run:

$ python helm_deployment_status.py -n <namespace_name> # namespace defaults to onap

To automatically verify functionality with healthchecks after deployment becomes ready or after timeout period expires, append -hp switch followed by the full path to the healthcheck script and --health-mode optional switch with appropriate mode supported by that script (health by default, --help displays available modes):

$ python helm_deployment_status.py -hp <app_data_path>/<app_name>/helm_charts/robot/ete-k8s.sh --health-mode <healthcheck mode>

It is strongly recommended to tailor helm_deployment_status.py to your needs since default values might not be what you’d expect. The defaults can be displayed with --help switch.

Final result of installation varies based on number of k8s nodes used and distribution of pods. In successful deployments all jobs should be in successful state. This can be verified with:

$ kubectl get jobs -n <namespace>

If some of the job is hanging in some wrong end-state like 'BackoffLimitExceeded' manual intervention is required to heal this and make also dependent jobs passing. More details about particular job state can be obtained using

$ kubectl describe job -n <namespace> <job_name>

If manual intervention is required, one can remove failing job and retry helm install command directly, which will not launch full deployment but rather check current state of the system and rebuild parts which are not up & running. Exact commands are as follows

$ kubectl delete job -n <namespace> <job_name>
$ helm deploy <env_name> <helm_chart_name> --namespace <namespace_name>

E.g. helm deploy dev local/onap --namespace onap

Once all pods are properly deployed and in running state, one can verify functionality e.g. by running onap healthchecks

$ cd <app_data_path>/<app_name>/helm_charts/robot
$ ./ete-k8s.sh onap health

You can install screen and jq packages to aid troubleshooting. Those can be installed from resources directory.

Screen is a terminal multiplexer and allows running multiple virtual terminal sessions as well as keep active SSH connections even when terminal is closed.

Jq can be used for editing json data format as output of kubectl. For example jq was used to troubleshoot SDNC-739 (UEB - Listener in Crashloopback)

$ kubectl -n onap get job onap-sdc-sdc-be-config-backend -o json | jq "del(.spec.selector)" | jq "del(.spec.template.metadata.labels)" | kubectl -n onap replace --force -f -

Appendix 1. Ansible execution/bootstrap

There are two ways how to easily run the installer’s ansible playbooks:

If you already have or can install a docker then you can build the provided 'Dockerfile' for the ansible and run playbooks in the docker container.
Another way to deploy ansible is via chroot environment which is bundled together within this directory.

(Re)build docker image and/or chroot archive

Inside the 'ansible/docker' directory you’ll find the 'Dockerfile' and 'build_ansible_image.sh' script. You can run 'build_ansible_image.sh' script on some machine with the internet connectivity and it will download all required packages needed for building the ansible docker image and for exporting it into a flat chroot environment.

Built image is exported into 'ansible_chroot.tgz' archive in the same ('ansible/docker') directory.

This script has two optional arguments:

ansible version
docker image name

Note: if optional arguments are not used, docker image name will be set to 'ansible' by default.

Launching ansible playbook using chroot environment

This is the default and preferred way of running ansible playbooks in an offline environment as there is no dependency on docker to be installed on the system. Chroot environment is already provided by included archive 'ansible_chroot.tgz'.

It should be available in the 'ansible/docker' directory as the end-result of the packaging script or after manual run of the 'build_ansible_image.sh' script referenced above.

All playbooks can be executed via './run_playbook.sh' wrapper script.

To get more info about the way how the './run_playbook.sh' wrapper script should be used, run:

$ ./run_playbook.sh

The main purpose of this wrapper script is to provide the ansible framework to a machine where it was bootstrapped without need of installing additional packages. The user can run this to display 'ansible-playbook' command help:

$ ./run_playbook.sh --help

Developers notes

There are two scripts which work in tandem for creating and running chroot
First one can convert docker image into chroot directory
Second script will automate chrooting (necessary steps for chroot to work and cleanup)

Both of them have help - just run:

$ cd ansible/docker
$ ./create_docker_chroot.sh help
$ ./run_chroot.sh help

Example usage:

$ sudo su
$ ansible/docker/create_docker_chroot.sh convert some_docker_image ./new_name_for_chroot
$ cat ./new_name_for_chroot/README.md
$ ansible/docker/run_chroot.sh execute ./new_name_for_chroot cat /etc/os-release 2>/dev/null

Launching ansible playbook using docker container (ALTERNATIVE APPROACH)

This option is here just to keep support for the older method which relies on a running docker service. For the offline deployment use the chroot option as indicated above.

You will not need 'ansible_chroot.tgz' archive anymore, but the new requirement is a prebuilt docker image of ansible (based on the provided 'Dockerfile'). It should be available in your local docker repository (otherwise the default name 'ansible' may fetch unwanted image from default registry!).

To trigger this functionality and to run 'ansible-playbook' inside a docker container instead of the chroot environment, you must first set the ANSIBLE_DOCKER_IMAGE variable. The value must be a name of the built ansible docker image.

Usage is basically the same as with the default chroot way - the only difference is the existence of the environment variable:

$ ANSIBLE_DOCKER_IMAGE=ansible ./run_playbook.sh --help

Appendix 2. Running Kubernetes Dashboard

Kubernetes Dashboard is a web-based, general purpose user interface for managing a k8s cluster.

Some of its capabilities are:

workloads/services management (troubleshooting, scaling, editing, restarting pods)
deploying new workloads/applications to the cluster
managing the cluster itself

Dashboard also provides information on the state of the cluster resources and on any errors that may have occurred.

Kubernetes Dashboard itself is a kubernetes application. For user convenience the Offline platform has it already pre-installed:

$ kubectl -n kubernetes-dashboard get deployment
NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
dashboard-metrics-scraper   1/1     1            1           76m
kubernetes-dashboard        1/1     1            1           76m

Accessing the dashboard

There are multiple ways to access the application’s web UI. Here we’ll assume usage of local port forwarding on a box where you have access to a browser since the dashboard in Offline platform is exposed via a node port by default.

First get the node port number that the dashboard service is exposed on:

$ kubectl -n kubernetes-dashboard get svc kubernetes-dashboard -o custom-columns=PORTS:.spec.ports[].nodePort
PORTS
30825

Now establish an ssh session to the infra node from your box from which you’ll be accessing the dashboard:

$ ssh -L 8080:127.0.0.1:30825 root@<infra host ip>

Point your browser at https://localhost:8080/ and you should see the login page:

Here, we’ll leverage the Bearer Token to log in. Offline platform comes with dashboard admin user already created, we just need to extract its token. On the infra node issue following command:

$ kubectl -n kubernetes-dashboard get secret $(kubectl -n kubernetes-dashboard get sa/admin-user -o jsonpath="{.secrets[0].name}") -o go-template="{{.data.token | base64decode}}"

It will return the token string on stdout. Copy-paste it into the sign-in form, selecting the “Token” option first. Upon successful login you’ll be presented the cluster resources from default namespace. In the drop down box at the top select the namespace into which you installed the Onap application (namespace name equals the value of app_name variable from offline-installer setup) and you should see the cluster resources for Onap:

For additional information concerning the Kubernetes Dashboard please refer to the official documentation.

Appendix 3. Running kube-prometheus stack

Kube-prometheus stack is a collection of Kubernetes manifests, Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator.

The Stack is not deployed by default in Offline ONAP Platform, but all artifacts which it requires are downloaded by relevant scripts in the package build phase (see Build Guide).

Setup (optional)

Kube-prometheus stack itself is a Kubernetes native application provisioned using Helm Charts. As such it can be configured using Helm values. Offline Installer provides a handy way for passing those values to the helm installation process.

Any values for the Stack should be defined as subkeys of kube_prometheus_helm_values variable in application_configuration.yml. For instance, in order to override the default Grafana password, insert below structure into application_configuration.yml:

kube_prometheus_helm_values:
  grafana:
    adminPassword: <password>

Another example - to set custom storage size for Prometheus tsdb:

kube_prometheus_helm_values:
  prometheus:
    prometheusSpec:
      storageSpec:
        volumeClaimTemplate:
          spec:
            resources:
              requests:
                storage: 6Gi

A comprehensive list of Helm values for the Stack can be obtained on the Kube-prometheus stack project site, in the values.yaml file. Additional values for the Grafana can be checked on the Grafana project site in the charts/grafana/values.yaml file.

Installation

In order to actually install this tool it’s required to set the following variable in application_configuration.yml:

kube_prometheus_stack_enabled: true

After the Offline Platform installation process is complete, the Stack will be deployed into its own kubernetes and helm namespace kube-prometheus.

ONAP Services Monitoring

Some ONAP services export application metrics which can be scraped by Prometheus by leveraging the ServiceMonitor objects. Offline Platform provides a curated set of Grafana panels for monitoring ONAP’s mariadb-galera chart. To enable mariadb-galera monitoring provide the following helm values in application_configuration.yml:

overrides:
  mariadb-galera:
    metrics:
      serviceMonitor:
        enabled: true
        basicAuth:
          enabled: false

To access the Galera/MariaDB dashboard navigate to Dashboards -> Manage -> ONAP -> Galera/MariaDB in Grafana UI.

Accessing Grafana dashboard

The most straightforward way to access the Grafana UI is by leveraging the port-forward k8s facility. Issue following command on the Infra host:

kubectl -n kube-prometheus port-forward --address 0.0.0.0 svc/kube-prometheus-stack-grafana 8081:80

Then navigate to http://<infra IP>:8081 to access the UI:

Default username is admin and the default password is grafana.

In the left pane navigate to Dashboards -> Manage to see the various pre-defined dashboards that come bundled with kube-prometheus stack. There is also the Custom folder which holds few additional dashes defined by the Offline Installer authors:

Alternative way of accessing the UI is by leveraging the NodePort type service which exposes Grafana UI on the Infra host public port directly. To do so get the port number first:

kubectl -n kube-prometheus get service/kube-prometheus-stack-grafana -o custom-columns=PORTS:.spec.ports[].nodePort

Then navigate to http://<infra IP>:<nodePort> to access the UI.

Offline Installer Testing Guide

This testing guide describes how offline installer can be tested in local development environment (laptop) without the need for actual servers.

Documentation refers to files/directories in ansible directory of this repository.

Introduction

Offline installer uses Molecule for testing all roles.

Molecule is tool for ansible roles development and testing. In this project Molecule is used for integration type of testing for both roles and playbooks. Role code is tested against simulated host.

Molecule is designed to test single Ansible role in isolation. Offline installer however has many small roles that are dependent on each other and also execution order for roles is meaningful. In that respect Molecule’s design does not offer sufficient level of testing as it’s lacking playbook level of scenario testing by default. Luckily Molecule is highly configurable and it is possible to achieve a higher level of testing scenarios for the offline installer.

Testing with Molecule is divided to two levels of testing:

role level testing (as per Molecule design)
playbook level testing (offline installer own setup)

Purpose

The purpose of using testing framework like Molecule is to make possible for developer to verify ansible code changes locally in own laptop without the need for big resources.

Developer is also expected to do development of the Ansible code and the Molecule test code at the same time. Offline installer does not have unittest level of testing for the ansible code.

Any commit made to ansible code base needs to first pass Molecule tests before it’s merged.

Test levels

To cover both testing levels (role and playbook) with maximum benefit and minimum copy-pasting, the testing code should be written in reusable way.

Reusable test code can be achieved by writing all prepare/cleanup and other helping code as a roles into main test directory. Also testinfra test code can be shared between different roles and between different scenarios of one role.

Testing of role and one scenario (one execution run of molecule) is fully defined by molecule.yml file.

molecule.yml file is always located in directory:

<tested-role>/molecule/<scenario>/molecule.yml

i.e. one role can have multiple scenarios (different configuration, OS etc. whatever user wants) to execute tests for same role. Each scenario has own molecule.yml file and own testinfra tests.

Molecule.yml file is the only file that cannot be re-used (except with symbolic links) but all other resources can be reused by referencing those in molecule.yml file or/and indirectly from resources molecule.yml is pointing to.

tested-role is clear in case of normal role level testing, but in playbook level testing the tested-role is just an invented role name and directory with molecule directory inside but no actual ansible role code.

Role level testing

The target is to test single role in isolation just like Molecule is designed. Role level testing is supposed to cover:

Syntax checking (Yamllint, Ansible lint, flake8)
Ansible code testing
Idempotence testing
Verifying role results from target hosts (testinfra tests)

Ansible code testing can/should also cover all different options how this role can be run (scenario). Different molecule runs can be implemented as own scenarios (in addition to default scenario) or default scenario playbook can be extended to run role tests multiple times just adjusting configuration between.

Single scenario example with nexus role

├── infrastructure.yml
├── roles
│   ├── nexus
│   │   ├── defaults
│   │   ├── files
│   │   ├── molecule
│   │   │   └── default
│   │   │       ├── molecule.yml
│   │   │       ├── playbook.yml
│   │   │       ├── prepare.yml
│   │   │       └── tests
│   │   ├── tasks
│   │   └── vars

Multiple scenario example with chrony role

roles/chrony
├── defaults
│   └── main.yml
├── handlers
│   └── main.yml
├── molecule
│   ├── default
│   │   ├── molecule.yml
│   │   ├── playbook.yml
│   │   └── prepare.yml
│   └── ubuntu
│       └── molecule.yml
├── tasks
│   └── main.yml
└── templates
    └── chrony.conf.j2

By default molecule runs just default scenario. To run specific one -s <scenario name> option must be used. The only subcommands supporting --all switch for playing with all scenarios are test and destroy. If using other ones -s must be used.

The cross-scenario code reuse paradigm should be rather implemented inside particular scenario’s molecule.yml file than by using filesystem symlinks. All provisioner playbooks should be located in default scenarios directory then and referenced in alternative scenarios as follows

provisioner:
  name: ansible
  lint:
    name: ansible-lint
  env:
    ANSIBLE_ROLES_PATH: ../../../../test/roles
  playbooks:
    prepare: ../default/prepare.yml
    converge: ../default/playbook.yml

Playbook level testing

Playbook level testing is this project’s (offline installer) own setup and way of using Molecule. The target is to raise testing level from single role testing up to single playbook testing.

Playbook level testing can be used also to run multiple playbooks and/or playbooks multiple times with different configuration.

The aim is to verify multiple roles working together i.e. higher level of integration testing.

Practically the tested-role is just a wrapper directory to conform molecule required directory structure and provide a name for the test. Directory itself does not contain any ansible role code, but just molecule files configured to run multiple other roles.

Playbook level test directories should be named consistently according to tested playbook and prefix string play and with optional description if there are multiple scenarios for single playbook:

play-<playbookname>[-<description>]

E.g.

play-infrastructure
play-resources

As role’s are tested with own molecule tests in isolation, playbook level tests should focus to integration of the roles and should avoid of repeating same tests as done already for individual roles.

Playbook level testing is supposed to cover:

Ansible code testing

Basically it’s easier to highlight what is supposed to be avoided in playbook level testing for the reason not to repeat the same that is done already in role level testing.

Syntax checking is left out already by default as molecule does linting only for the role code where molecule is run, and in this case tested-role is empty.
Idempotence can be tested, but should be disabled (by default) in molecule.yml because it takes too much time and was tested already for individual roles.
Verifying target hosts with testinfra tests can be done but then something else should be tested as in role based tests. And if those 2 would overlap it’s better to leave them out.

Example with infrastructure playbook level test files

├── infrastructure.yml
└── test
    ├── play-infrastructure
    │   └── molecule
    │       └── default
    │           ├── molecule.yml
    │           ├── playbook.yml
    │           ├── prepare.yml
    │           └── tests

Test code reuse and naming

As both testing levels test the same Ansible roles, there are a need to share common code for both of them.

Testinfra Python code should be shared when also playbook level tests verify target hosts. However sharing is not limited only for the 2 test levels but also between different roles.

Individual role have testinfra tests on directory:

roles/<role>/molecule/<scenario>/tests

and any commonly usable testinfra Python code should be placed to directory:

test/testinfra

Ansible role testing uses several resources defined by provisioner section of molecule.yml https://molecule.readthedocs.io/en/latest/configuration.html#provisioner

Most common resources that are written for role testing are:

playbook.yml (mandatory but can include specific code)
prepare.yml
cleanup.yml
create.yml
destroy.yml

all of which can be just placed to scenario directory together with playbook.yml (without editing molecule.yml when in default directory) and all of which can include ansible code to do something e.g. prepare role for testing.

Example molecule files:

Role level tests for nexus role:

roles/nexus/molecule/default/molecule.yml
roles/nexus/molecule/default/playbook.yml
roles/nexus/molecule/default/prepare.yml

playbook level tests for infrastructure playbook:

test/play-infrastructure/molecule/default/molecule.yml
test/play-infrastructure/molecule/default/playbook.yml
test/play-infrastructure/molecule/default/prepare.yml

Sharing all test code should be done by writing them in the form of ansible roles and placing commonly usable roles into:

test/roles/<testrole>

Test roles should be named consistently according to action it’s needed and role for it’s for together with optional description:

<action>-<role>[-<description>]

Examples of commonly used test roles

├── infrastructure.yml
└── test
    ├── play-infrastructure
    └── roles
        ├── post-certificates
        ├── prepare-common
        ├── prepare-dns
        ├── prepare-docker
        ├── prepare-nexus
        └── prepare-nginx

Molecule platform images

Molecule can build images of the tested hosts on the fly with default Dockerfile template (docker driver) or from a Dockerfile provided by user. In case of Vagrant driver used box image can be also fully customized by user.

To speed up testing and lessen the footprint of code for image preparation it’s preferred to use unmodified images from Docker Registry whenever possible (can be pulled prior to running Molecule) or pre-build images created from Dockerfiles listed below. Most significant feature of those is support for Systemd, so they should be used in cases where ansible’s ‘systemd’ module is used.

Used Dockerfiles/Box definitions are kept in following directory structure

└── test
    └── images
        ├── docker
        │   ├── build-all.sh
        │   ├── centos7
        │   │   ├── build.sh
        │   │   └── Dockerfile
        │   └── ubuntu
        │       ├── build.sh
        │       └── Dockerfile
        └── vagrant

Build-all.sh is a script for building all images, build.sh scripts in particular platforms subdirs are for building just specific images. Keep in mind that while images from Docker Registry will be downloaded automatically at run time, the above ones must be built manually prior to launching Molecule.

Build images

Build all platforms images before running Molecule tests. Building can be done with the following single command:

test/images/docker/build-all.sh

Install

Molecule can be installed in multiple ways and in this guide 2 different ways is covered.

Install Molecule with pip in virtual environment
Use Molecule provided docker container to run Molecule

Install with pip

This is a OS dependent and some prerequisites needs to be installed, but after prerequisites are installed installing Molecule can be done by calling following script:

source test/bin/install-molecule.sh

As for the required OS packages, see example for Ubuntu in the install-molecule.sh script’s comments or from Molecule pages.

Note that sourcing the script is not needed to get Molecule installed but it leaves you already into virtual environment and ready to run Molecule.

To get out from virtual environment issue:

deactivate

And next time to activate virtual environment again before running Molecule, issue:

source ~/molecule_venv/bin/activate

And here the directory ~/molecule_venv is just the default virtual environment path that install-molecule.sh script is using and can be overridden with VENV_PATH environment variable.

Use Molecule docker container

Molecule provides docker containers images via quay.io where Molecule, Ansible and all needed dependencies are build to the image.

In this way of using Molecule, no installation is needed and only docker is the prerequisite for running Molecule.

For using provided image to test offline-installer roles, following scripts are provided:

Build container image:: test/molecule-docker/build.sh

This will build image named molecule-dev with strict version tag.

Set molecule into the PATH:: source test/bin/set_molecule_paths.sh

That will add the actual Molecule run wrapper script test/bin/molecule.sh to path usable from everywhere similarly than molecule with pip and virtual environment.

Run Molecule wrapper script:: test/bin/molecule.sh

For running Molecule. Using molecule-dev image and the exact version defined by test/docker/build.sh script.

Usage

Basic usage of molecule tests. See more detailed instructions from Molecule

Run complete testing for a role or a playbook:

cd roles/<role> or cd test/play-<playbook-name>
molecule test

Develop a role code and run testing during the coding:

cd roles/<role>
Edit ansible code and molecule test code when needed
molecule converge
Repeat steps 2 and 3 until code is ready and molecule tests are passing
molecule test

vFWCL on Ealto ONAP offline platform 

This document is collecting notes we have from running vFirewall demo on offline Elalto platform installed by ONAP offline installer tool.

Overall it’s slightly more complicated than in Dublin mainly due to POLICY-2191 issue.

Some of the most relevant materials are available on following links:

Table of Contents

vFWCL on Ealto ONAP offline platform

Step 1. Preconditions - before ONAP deployment 

Understanding of the underlying OpenStack deployment is required from anyone applying these instructions.

In addition, installation-specific location of the helm charts on the infra node must be known. In this document it is referred to as <helm_charts_dir>

Snippets below are describing areas we need to configure for successfull vFWCL demo.

Pay attention to them and configure it (ideally before deployment) accordingly.

Note

We are using standard OOM kubernetes/onap/resources/overrides/onap-all.yaml override to enable all components, however looks that better tailored one onap-vfw.yaml exists in the same folder. In following description we would be focusing on just other override values specific for lab environment.

1) Override for Update APPC / Robot and SO parts:

appc:
      enabled: true
      config:
        openStackType: "OpenStackProvider"
        openStackName: "OpenStack"
        openStackKeyStoneUrl: "http://10.20.30.40:5000/v2.0"
        openStackServiceTenantName: "service"
        openStackDomain: "default"
        openStackUserName: "onap-tieto"
        openStackEncryptedPassword: "31ECA9F2BA98EF34C9EC3412D071E31185F6D9522808867894FF566E6118983AD5E6F794B8034558"
    robot:
      enabled: true
      appcUsername: "appc@appc.onap.org"
      appcPassword: "demo123456!"
      openStackKeyStoneUrl: "http://10.20.30.40:5000"
      openStackPublicNetId: "9403ceea-0738-4908-a826-316c8541e4bb"
      openStackTenantId: "b1ce7742d956463999923ceaed71786e"
      openStackUserName: "onap-tieto"
      ubuntu14Image: "trusty"
      openStackPrivateNetId: "3c7aa2bd-ba14-40ce-8070-6a0d6a617175"
      openStackPrivateSubnetId: "2bcb9938-9c94-4049-b580-550a44dc63b3"
      openStackPrivateNetCidr: "10.0.0.0/16"
      openStackSecurityGroup: "onap_sg"
      openStackOamNetworkCidrPrefix: "10.0"
      openStackPublicNetworkName: "rc3-offline-network"
      vnfPrivateKey: '/var/opt/ONAP/onap-dev.pem'
      vnfPubKey: "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPwF2bYm2QuqZpjuAcZDJTcFdUkKv4Hbd/3qqbxf6g5ZgfQarCi+mYnKe9G9Px3CgFLPdgkBBnMSYaAzMjdIYOEdPKFTMQ9lIF0+i5KsrXvszWraGKwHjAflECfpTAWkPq2UJUvwkV/g7NS5lJN3fKa9LaqlXdtdQyeSBZAUJ6QeCE5vFUplk3X6QFbMXOHbZh2ziqu8mMtP+cWjHNBB47zHQ3RmNl81Rjv+QemD5zpdbK/h6AahDncOY3cfN88/HPWrENiSSxLC020sgZNYgERqfw+1YhHrclhf3jrSwCpZikjl7rqKroua2LBI/yeWEta3amTVvUnR2Y7gM8kHyh Generated-by-Nova"
      demoArtifactsVersion: "1.4.0"
      demoArtifactsRepoUrl: "https://nexus.onap.org/content/repositories/releases"
      scriptVersion: "1.4.0"
      config:
        # openStackEncryptedPasswordHere should match the encrypted string used in SO and APPC and overridden per environment
        openStackEncryptedPasswordHere: "f7920677e15e2678b0f33736189e8965"
    so:
      enabled: true
      config:
        openStackUserName: "onap-tieto"
        openStackRegion: "RegionOne"
        openStackKeyStoneUrl: "http://10.20.30.40:5000"
        openStackServiceTenantName: "services"
        openStackEncryptedPasswordHere: "31ECA9F2BA98EF34C9EC3412D071E31185F6D9522808867894FF566E6118983AD5E6F794B8034558"
      so-catalog-db-adapter:
        config:
          openStackUserName: "onap-tieto"
          openStackKeyStoneUrl: "http://10.20.30.40:5000/v2.0"
          openStackEncryptedPasswordHere: "31ECA9F2BA98EF34C9EC3412D071E31185F6D9522808867894FF566E6118983AD5E6F794B8034558"

Step 2. Preconditions - after ONAP deployment 

Run HealthChecks after successful deployment, all of them must pass

Relevant robot scripts are under <helm_charts_dir>/oom/kubernetes/robot

[root@tomas-infra robot]# ./ete-k8s.sh onap health

61 critical tests, 61 passed, 0 failed
61 tests total, 61 passed, 0 failed

very useful page describing commands for manual checking of HC’s

Unfortunatelly some patching is still required to get vFWCL working on ONAP plaform. Therefore we provided a bunch of files and put them into ./patches folder within this repo.

After installation is finished and all healthchecks are green it is still required to patch few things. Those will be described in following part.

Step 3. Patching 

In order to get vFWCL working in our lab on offline platform, we need to ensure 3 things except healthecks prior proceeding with official instructions.

robot a) private key for robot has to be configured properly and contain key present on robot pod

# open configmap for robot and check GLOBAL_INJECTED_PRIVATE_KEY param
kubectl edit configmap onap-robot-robot-eteshare-configmap
# it should contain stuff like
# GLOBAL_INJECTED_PRIVATE_KEY = '/var/opt/ONAP/onap-dev.pem'

we need to put some private key for that and that key must match with public key distributed to vFWCL VMs which is coming from vnfPubKey parameter in robot

b) in our lab there is some issue with cloud-init and vFW VMs are getting default route set quite randomly, which is an issue as in our lab we specified following dedicated network for vFW VMs public connectivity.

Note

same network has to be reachable from k8s host where robot container is

id	name	tenant_id	subnets
9403ceea-0738-4908-a826-316c8541e4bb	rc3-offline-network	b1ce7742d956463999923ceaed71786e	1782c82c-cd92-4fb6-a292-5e396afe63ec 10.8.8.0/24

for this reason we are patching base_vfw.yaml for all vFW VMs with following code

# nasty hack to bypass cloud-init issues
sed  -i '1i nameserver 8.8.8.8' /etc/resolv.conf
iface_correct=`ip a | grep <network_prefix> | awk {'print $7'}`
route add default gw <network_prefix>.1 ${iface_correct}

Network prefix variable is in our case “10.8.8”.

Lets treat it as an example of how these two problems can be fixed. Feel free to adjust private/public key and skip cloud-init problem if you don’t have it. Our helping script with above setting is fixing both issues (a) and (b) for us.

# copy offline-installer repo into infra node and run following script from patches folder
./update_robot.sh <namespace> <network_prefix>

drools c) usecases controller is not working - POLICY-2191

There are couple of pom files required in order to get usecases controller in drools pod instantiated properly. One can fix it by running following script.

# copy offline-installer repo into infra node and run following script from patches folder
./update_policy.sh <namespace>

Note

This script is also restarting policy, there is some small chance that drools will be marked as sick during interval its being restarted and redeployed. If it happens, just try again.

At this moment one can check that usecases controller is build properly via:

# on infra node
kubectl  exec -it onap-policy-drools-0 bash
bash-4.4$ telemetry
Version: 1.0.0
https://localhost:9696/policy/pdp/engine> get controllers
HTTP/1.1 200 OK
Content-Length: 24
Content-Type: application/json
Date: Mon, 04 Nov 2019 06:31:09 GMT
Server: Jetty(9.4.20.v20190813)

[
    "amsterdam",
    "usecases"
]

Now we can proceed with same steps as on online platform.

Step 4. robot init - demo services distribution 

Run following robot script to execute both init_customer + distribute

#  demo-k8s.sh <namespace> init

[root@tomas-infra robot]# ./demo-k8s.sh onap init

Step 5. robot instantiateVFW 

Following tag is used for whole vFWCL testcase. It will deploy single heat stack with 3 VMs and set policies and APPC mount point for vFWCL to happen.

# demo-k8s.sh <namespace> instantiateVFW

root@tomas-infra robot]# ./demo-k8s.sh onap instantiateVFW

Step 6. verify vFW 

Verify VFWCL. This step is just to verify CL functionality, which can be also verified by checking DarkStat GUI on vSINK VM <sink_ip:667>

# demo-k8s.sh <namespace> vfwclosedloop <pgn-ip-address>
# e.g. where 10.8.8.5 is IP from public network dedicated to vPKG VM
root@tomas-infra robot]# ./demo-k8s.sh onap vfwclosedloop 10.8.8.5

CHANGELOG

2022-04-19

RKE upgrade to 1.3.8
Kubernetes version deployed by RKE has been upgraded to v1.22.7
Added provisioning of the Kubernetes Strimzi Kafka Operator

2022-02-18

Add support for ONAP services monitoring (leverages Prometheus ServiceMonitor objects)

2022-01-21

Add support in download script for leveraging docker mirroring registry for images from private repositories

2021-11-25

Dropped support for obsoleted Helm v2

2021-11-16

Added support for provisioning the cert-manager (https://cert-manager.io/)
Added cmctl CLI management utility for cert-manager

2021-10-27

Upgraded Helm release to 3.6.3

2021-09-30

Add support in cicdansible for docker_storage_size heat stack parameter for setting custom docker storage volume size on nodes

2021-09-24

Upgrade supported OS to RedHat/CentOS 7.9

2021-09-17

Added provisioning of a config file containing resources dir full path

2021-09-16

A custom Grafana Home dashboard is added

2021-09-15

Default Grafana’s password was changed to “grafana”

2021-09-13

Kubernetes Dashboard has been upgraded to v2.3.1

2021-09-09

Kube Prometheus Stack has been upgraded to 18.0.4

2021-09-07

Upgraded Helm release to 3.5.2

2021-09-02

RKE upgrade to 1.3.0
Kubernetes version deployed by RKE has been upgraded to v1.19.14
kubectl upgrade to 1.19.14

2021-06-29

Added an option to turn on Helm verbose output in helm-healer script

2021-06-24

Helm v2 specific code was abandoned
Added support for provisioning Kube Prometheus Stack

OOM offline-installer

Offline Installer Package Build Guide

Part 1. Prerequisites

Part 2. Download artifacts for offline installer

Part 3. Populate local nexus

Part 4. Packages preparation

Offline Installer - Installation Guide

Part 1. Prerequisites

Kubernetes cluster

Kubernetes cluster overview

Install-server

Part 2. Configuration

Installer packages

Application directory

hosts.yml

application_configuration.yml

Helm chart values overrides

SSH authentication

Part 3. Installation

Part 4. Post-installation and troubleshooting

Appendix 1. Ansible execution/bootstrap

(Re)build docker image and/or chroot archive

Launching ansible playbook using chroot environment

Developers notes

Launching ansible playbook using docker container (ALTERNATIVE APPROACH)

Appendix 2. Running Kubernetes Dashboard

Accessing the dashboard

Appendix 3. Running kube-prometheus stack

Setup (optional)

Installation

ONAP Services Monitoring

Accessing Grafana dashboard

Offline Installer Testing Guide

Introduction

Purpose

Test levels

Role level testing

Playbook level testing

Test code reuse and naming

Molecule platform images

Build images

Install

Install with pip

Use Molecule docker container

Usage

vFWCL on Ealto ONAP offline platform

Step 1. Preconditions - before ONAP deployment

Step 2. Preconditions - after ONAP deployment

Step 3. Patching

Step 4. robot init - demo services distribution

Step 5. robot instantiateVFW

Step 6. verify vFW

CHANGELOG

2022-04-19

2022-02-18

2022-01-21

2021-11-25

2021-11-16

2021-10-27

2021-09-30

2021-09-24

2021-09-17

2021-09-16

2021-09-15

2021-09-13

2021-09-09

2021-09-07

2021-09-02

2021-06-29

2021-06-24

vFWCL on Ealto ONAP offline platform 

Step 1. Preconditions - before ONAP deployment 

Step 2. Preconditions - after ONAP deployment 

Step 3. Patching 

Step 4. robot init - demo services distribution 

Step 5. robot instantiateVFW 

Step 6. verify vFW 