OOM Base Platform

As part of the initial base setup of the host Kubernetes cluster, the following mandatory installation and configuration steps must be completed.

For additional platform add-ons, see the OOM Optional Addons section.

Install & configure kubectl 

The Kubernetes command line interface used to manage a Kubernetes cluster needs to be installed and configured to run as non root.

For additional information regarding kubectl installation and configuration see the kubectl installation guide

To install kubectl, execute the following, replacing the <recommended-kubectl-version> with the version defined in the OOM Software Requirements (base) table:

> curl -LO https://dl.k8s.io/release/v<recommended-kubectl-version>/bin/linux/amd64/kubectl

> chmod +x ./kubectl

> sudo mv ./kubectl /usr/local/bin/kubectl

> mkdir ~/.kube

> cp kube_config_cluster.yml ~/.kube/config.onap

> export KUBECONFIG=~/.kube/config.onap

> kubectl config use-context onap

Validate the installation:

> kubectl get nodes

NAME             STATUS   ROLES               AGE     VERSION
onap-control-1   Ready    controlplane,etcd   3h53m   v1.27.5
onap-control-2   Ready    controlplane,etcd   3h53m   v1.27.5
onap-k8s-1       Ready    worker              3h53m   v1.27.5
onap-k8s-2       Ready    worker              3h53m   v1.27.5
onap-k8s-3       Ready    worker              3h53m   v1.27.5
onap-k8s-4       Ready    worker              3h53m   v1.27.5
onap-k8s-5       Ready    worker              3h53m   v1.27.5
onap-k8s-6       Ready    worker              3h53m   v1.27.5

Install & configure helm 

Helm is used for package and configuration management of the relevant helm charts. For additional information, see the helm installation guide

To install helm, execute the following, replacing the <recommended-helm-version> with the version defined in the OOM Software Requirements (base) table:

> wget https://get.helm.sh/helm-v<recommended-helm-version>-linux-amd64.tar.gz

> tar -zxvf helm-v<recommended-helm-version>-linux-amd64.tar.gz

> sudo mv linux-amd64/helm /usr/local/bin/helm

Verify the helm version with:

> helm version

Helm’s default CNCF provided Curated applications for Kubernetes repository called stable can be removed to avoid confusion:

> helm repo remove stable

Install the additional OOM plugins required to un/deploy the OOM helm charts:

> git clone http://gerrit.onap.org/r/oom

> helm plugin install ~/oom/kubernetes/helm/plugins/deploy

> helm plugin install ~/oom/kubernetes/helm/plugins/undeploy

Verify the plugins are installed:

> helm plugin ls

NAME        VERSION   DESCRIPTION
deploy      1.0.0     install (upgrade if release exists) parent chart and all subcharts as separate but related releases
undeploy    1.0.0     delete parent chart and subcharts that were deployed as separate releases

Set the default StorageClass 

In some ONAP components it is important to have a default storageClass defined (e.g. cassandra), if you don’t want to explicitly set it during the deployment via helm overrides.

Therefor you should set the default storageClass (if not done during the K8S cluster setup) via the command:

> kubectl patch storageclass <storageclass> -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'

see DefaultStorageClass

Install the Strimzi Kafka Operator 

Strimzi Apache Kafka provides a way to run an Apache Kafka cluster on Kubernetes in various deployment configurations by using kubernetes operators. Operators are a method of packaging, deploying, and managing Kubernetes applications.

Strimzi Operators extend the Kubernetes functionality, automating common and complex tasks related to a Kafka deployment. By implementing knowledge of Kafka operations in code, the Kafka administration tasks are simplified and require less manual intervention.

The Strimzi cluster operator is deployed using helm to install the parent chart containing all of the required custom resource definitions. This should be done by a kubernetes administrator to allow for deployment of custom resources in to any kubernetes namespace within the cluster.

Full installation instructions can be found in the Strimzi Apache Kafka Operator helm Installation documentation.

To add the required helm repository, execute the following:

> helm repo add strimzi https://strimzi.io/charts/

To install the strimzi kafka operator, execute the following, replacing the <recommended-strimzi-version> with the version defined in the OOM Software Requirements (base) table:

> helm install strimzi-kafka-operator strimzi/strimzi-kafka-operator --namespace strimzi-system --version <recommended-strimzi-version> --set watchAnyNamespace=true --create-namespace

Verify the installation:

> kubectl get po -n strimzi-system

NAME                                        READY   STATUS    RESTARTS       AGE
strimzi-cluster-operator-7f7d6b46cf-mnpjr   1/1     Running   0              2m

Install Cert-Manager 

Cert-Manager is a native Kubernetes certificate management controller. It can help with issuing certificates from a variety of sources, such as Let’s Encrypt, HashiCorp Vault, Venafi, a simple signing key pair, self signed or external issuers. It ensures certificates are valid and up to date, and attempt to renew certificates at a configured time before expiry.

Cert-Manager is deployed using regular YAML manifests which include all the needed resources (the CustomResourceDefinitions, cert-manager, namespace, and the webhook component).

Full installation instructions, including details on how to configure extra functionality in Cert-Manager can be found in the Cert-Manager Installation documentation.

There is also a kubectl plugin (kubectl cert-manager) that can help you to manage cert-manager resources inside your cluster. For installation steps, please refer to Cert-Manager kubectl plugin documentation.

To install cert-manager, execute the following, replacing the <recommended-cm-version> with the version defined in the OOM Software Requirements (base) table:

> kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v<recommended-cm-version>/cert-manager.yaml

Verify the installation:

> kubectl get po -n cert-manager

NAME                                       READY   STATUS    RESTARTS      AGE
cert-manager-776c4cfcb6-vgnpw              1/1     Running   0             2m
cert-manager-cainjector-7d9668978d-hdxf7   1/1     Running   0             2m
cert-manager-webhook-66c8f6c75-dxmtz       1/1     Running   0             2m

Istio Service Mesh 

Note

The ONAP deployment supports the ONAP Next Generation Security & Logging Structure

ONAP is currenty supporting Istio as default ServiceMesh platform. Therefor the following instructions describe the setup of Istio and required tools. Used Istio setup guide

Istio Platform Installation

Install Istio Basic Platform

Configure the Helm repository:

> helm repo add istio https://istio-release.storage.googleapis.com/charts

> helm repo update

Create a namespace for “mesh-level” configurations:
```
> kubectl create namespace istio-config
```
Create a namespace istio-system for Istio components:
```
> kubectl create namespace istio-system
```
Install the Istio Base chart which contains cluster-wide resources used by the Istio control plane, replacing the <recommended-istio-version> with the version defined in the OOM Software Requirements (base) table:
```
> helm upgrade -i istio-base istio/base -n istio-system --version <recommended-istio-version>
```

Create an override for istiod (e.g. istiod.yaml) to add the oauth2-proxy as external authentication provider and apply some specific config settings

istiod.yaml

global:
  proxy:
    # Controls if sidecar is injected at the front of the container list and blocks the start of the other containers until the proxy is ready
    holdApplicationUntilProxyStarts: true
  #logging:
  #  level: "default:debug"
meshConfig:
  rootNamespace: istio-config
  extensionProviders:
  - name: oauth2-proxy
    envoyExtAuthzHttp:
      service: oauth2-proxy.default.svc.cluster.local
      port: 80
      timeout: 1.5s
      includeHeadersInCheck: ["authorization", "cookie"]
      headersToUpstreamOnAllow: ["x-forwarded-access-token", "authorization", "path", "x-auth-request-user", "x-auth-request-email", "x-auth-request-access-token"]
      headersToDownstreamOnDeny: ["content-type", "set-cookie"]
pilot:
  env:
    PILOT_HTTP10: true

Install the Istio Base Istio Discovery chart which deploys the istiod service, replacing the <recommended-istio-version> with the version defined in the OOM Software Requirements (base) table:
```
> helm upgrade -i istiod istio/istiod -n istio-system --version <recommended-istio-version>
--wait -f ./istiod.yaml
```

Add an EnvoyFilter for HTTP header case

When handling HTTP/1.1, Envoy will normalize the header keys to be all lowercase. While this is compliant with the HTTP/1.1 spec, in practice this can result in issues when migrating existing systems that might rely on specific header casing. In our case a problem was detected in the SDC client implementation, which relies on uppercase header values. To solve this problem in general we add a EnvoyFilter to keep the uppercase header in the istio-config namespace to apply for all namespaces, but set the context to SIDECAR_INBOUND to avoid problems in the connection between Istio-Gateway and Services

Create a EnvoyFilter file (e.g. envoyfilter-case.yaml)

envoyfilter-case.yaml

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: header-casing
  namespace: istio-config
spec:
  configPatches:
  - applyTo: CLUSTER
    match:
      context: SIDECAR_INBOUND
    patch:
      operation: MERGE
      value:
        typed_extension_protocol_options:
          envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
            '@type': type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
            use_downstream_protocol_config:
              http_protocol_options:
                header_key_format:
                  stateful_formatter:
                    name: preserve_case
                    typed_config:
                      '@type': type.googleapis.com/envoy.extensions.http.header_formatters.preserve_case.v3.PreserveCaseFormatterConfig
  - applyTo: NETWORK_FILTER
    match:
      listener:
        filterChain:
          filter:
            name: envoy.filters.network.http_connection_manager
    patch:
      operation: MERGE
      value:
        typed_config:
          '@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          http_protocol_options:
            header_key_format:
              stateful_formatter:
                name: preserve_case
                typed_config:
                  '@type': type.googleapis.com/envoy.extensions.http.header_formatters.preserve_case.v3.PreserveCaseFormatterConfig
---
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: header-casing-outbound
  namespace: istio-config
  #annotations:
  #  argocd.argoproj.io/hook: PostSync
spec:
  configPatches:
  - applyTo: CLUSTER
    match:
      context: SIDECAR_OUTBOUND
    patch:
      operation: MERGE
      value:
        typed_extension_protocol_options:
          envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
            '@type': type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
            use_downstream_protocol_config:
              http_protocol_options:
                header_key_format:
                  stateful_formatter:
                    name: preserve_case
                    typed_config:
                      '@type': type.googleapis.com/envoy.extensions.http.header_formatters.preserve_case.v3.PreserveCaseFormatterConfig
  - applyTo: NETWORK_FILTER
    match:
      listener:
        filterChain:
          filter:
            name: envoy.filters.network.http_connection_manager
    patch:
      operation: MERGE
      value:
        typed_config:
          '@type': type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          http_protocol_options:
            header_key_format:
              stateful_formatter:
                name: preserve_case
                typed_config:
                  '@type': type.googleapis.com/envoy.extensions.http.header_formatters.preserve_case.v3.PreserveCaseFormatterConfig

Apply the change to Istio:

> kubectl apply -f envoyfilter-case.yaml

Ingress Controller Installation 

In the production setup 2 different Ingress setups are supported.

Gateway API Gateway-API (recommended)
Istio Gateway Istio-Gateway (alternative, but in the future deprecated)

Depending on the solution, the ONAP helm values.yaml has to be configured. See the OOM customized deployment section for more details.

Gateway-API (recommended)

Install the Gateway-API CRDs replacing the <recommended-gwapi-version> with the version defined in the OOM Software Requirements (base) table:

> kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/download/<recommended-gwapi-version>/experimental-install.yaml

Create a common Gateway instance named “common-gateway” The following example uses provides listeners for HTTP(s), UDP and TCP

common-gateway.yaml

apiVersion: gateway.networking.k8s.io/v1beta1
kind: Gateway
metadata:
  name: common-gateway
  namespace: istio-ingress
spec:
  gatewayClassName: istio
  listeners:
    - name: http-80
      hostname: "*.{{ onap_baseurl }}"
      port: 80
      protocol: HTTP
      allowedRoutes:
        namespaces:
          from: All
    - name: https-443
      hostname: "*.{{ onap_baseurl }}"
      port: 443
      protocol: HTTPS
      allowedRoutes:
        namespaces:
          from: All
      tls:
        mode: Terminate
        certificateRefs:
          - kind: Secret
            group: ""
            name: ingress-tls-secret
    - name: udp-162
      protocol: UDP
      port: 162
      allowedRoutes:
        kinds:
          - kind: UDPRoute
        namespaces:
          from: All
    - name: tcp-4334
      protocol: TCP
      port: 4334
      allowedRoutes:
        kinds:
          - kind: TCPRoute
        namespaces:
          from: All
    - name: tcp-9000
      allowedRoutes:
        namespaces:
          from: All
      hostname: "kafka-api{{ onap_postaddr }}.{{ onap_baseurl }}"
      port: 9000
      protocol: TLS
      tls:
        certificateRefs:
          - group: ""
            kind: Secret
            name: ingress-tls-secret
        mode: Terminate
    - name: tcp-9001
      allowedRoutes:
        namespaces:
          from: All
      hostname: "kafka-api{{ onap_postaddr }}.{{ onap_baseurl }}"
      port: 9001
      protocol: TLS
      tls:
        certificateRefs:
          - group: ""
            kind: Secret
            name: ingress-tls-secret
        mode: Terminate
    - name: tcp-9002
      allowedRoutes:
        namespaces:
          from: All
      hostname: "kafka-api{{ onap_postaddr }}.{{ onap_baseurl }}"
      port: 9002
      protocol: TLS
      tls:
        certificateRefs:
          - group: ""
            kind: Secret
            name: ingress-tls-secret
        mode: Terminate
    - name: tcp-9010
      allowedRoutes:
        namespaces:
          from: All
      hostname: "kafka-bootstrap-api{{ onap_postaddr }}.{{ onap_baseurl }}"
      port: 9010
      protocol: TLS
      tls:
        certificateRefs:
          - group: ""
            kind: Secret
            name: ingress-tls-secret
        mode: Terminate

Apply the change:
```
> kubectl apply -f common-gateway.yaml
```

Istio Gateway (alternative)

Create a namespace istio-ingress for the Istio Ingress gateway and enable istio-injection:

> kubectl create namespace istio-ingress

> kubectl label namespace istio-ingress istio-injection=enabled

To expose additional ports besides HTTP/S (e.g. for external Kafka access, SDNC-callhome) create an override file (e.g. istio-ingress.yaml)

istio-ingress.yaml

service:
  # Type of service. Set to "None" to disable the service entirely
  type: LoadBalancer
  ports:
  - name: status-port
    port: 15021
    protocol: TCP
    targetPort: 15021
  - name: http2
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  - name: kafka-bootstrap
    port: 9010
    targetPort: 9010
    protocol: TCP
  - name: kafka-0
    port: 9000
    targetPort: 9000
    protocol: TCP
  - name: kafka-1
    port: 9001
    targetPort: 9001
    protocol: TCP
  - name: kafka-2
    port: 9002
    targetPort: 9002
    protocol: TCP
  - name: sdnc-callhome
    port: 4334
    targetPort: 4334
    protocol: TCP

Install the Istio Gateway chart using the override file, replacing the <recommended-istio-version> with the version defined in the OOM Software Requirements (base) table:
```
> helm upgrade -i istio-ingress istio/gateway -n istio-ingress
--version <recommended-istio-version> -f ingress-istio.yaml --wait
```

Keycloak Installation 

Add helm repositories:
```
> helm repo add bitnami https://charts.bitnami.com/bitnami
```
> helm repo add codecentric https://codecentric.github.io/helm-charts

> helm repo update

create keycloak namespace:

> kubectl create namespace keycloak
> kubectl label namespace keycloak istio-injection=disabled

Install Keycloak-Database

To configure the Postgres DB create an override file (e.g. keycloak-db-values.yaml)

keycloak-db-values.yaml

# See https://github.com/bitnami/charts/tree/master/bitnami/postgresql
global:
  postgresql:
    auth:
      username: dbusername
      password: dbpassword
      database: keycloak

Install the Postgres DB:

> helm -n keycloak upgrade -i keycloak-db bitnami/postgresql --values ./keycloak-db-values.yaml

Configure Keycloak

To configure the Keycloak instance create an override file (e.g. keycloak-server-values.yaml)

keycloak-server-values.yaml

---
command:
  - "/opt/keycloak/bin/kc.sh"
  - "--verbose"
  - "start"
  - "--http-enabled=true"
  - "--http-port=8080"
  - "--hostname-strict=false"
  - "--hostname-strict-https=false"
  - "--spi-events-listener-jboss-logging-success-level=info"
  - "--spi-events-listener-jboss-logging-error-level=warn"

extraEnv: |
  - name: KEYCLOAK_ADMIN
    valueFrom:
      secretKeyRef:
        name: {{ include "keycloak.fullname" . }}-admin-creds
        key: user
  - name: KEYCLOAK_ADMIN_PASSWORD
    valueFrom:
      secretKeyRef:
        name: {{ include "keycloak.fullname" . }}-admin-creds
        key: password
  - name: JAVA_OPTS_APPEND
    value: >-
      -XX:+UseContainerSupport
      -XX:MaxRAMPercentage=50.0
      -Djava.awt.headless=true
      -Djgroups.dns.query={{ include "keycloak.fullname" . }}-headless
  - name: PROXY_ADDRESS_FORWARDING
    value: "true"

dbchecker:
  enabled: true

database:
  vendor: postgres
  hostname: keycloak-db-postgresql
  port: 5432
  username: dbusername
  password: dbpassword
  database: keycloak

secrets:
  admin-creds:
    stringData:
      user: admin
      password: secret

Install keycloak:

> helm -n keycloak upgrade -i keycloak codecentric/keycloakx --values ./keycloak-server-values.yaml

The required Ingress entry and REALM will be provided by the ONAP “Platform” component.

Create Ingress gateway entry for the keycloak web interface using the configured Ingress <base-url> (here “simpledemo.onap.org”) as described in OOM Custom Overrides

keycloak-ingress.yaml

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  labels:
    app.kubernetes.io/managed-by: Helm
  name: keycloak-ui-http-route
  namespace: keycloak
spec:
  hostnames:
  - keycloak-ui.simpledemo.onap.org
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: common-gateway
    namespace: istio-ingress
    sectionName: https-80
  rules:
    Filters:
      Request Redirect:
        Port:         443
        Scheme:       https
        Status Code:  301
      Type:           RequestRedirect
    Matches:
      Path:
        Type:   PathPrefix
        Value:  /auth
---
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  labels:
    app.kubernetes.io/managed-by: Helm
  name: keycloak-ui-http-route
  namespace: keycloak
spec:
  hostnames:
  - keycloak-ui.simpledemo.onap.org
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: common-gateway
    namespace: istio-ingress
    sectionName: https-443
  rules:
  - backendRefs:
    - group: ""
      kind: Service
      name: keycloak-keycloakx-http
      port: 80
      weight: 1
    matches:
    - path:
        type: PathPrefix
        value: /auth

Add the Ingress entry for Keycloak:

> kubectl -n keycloak apply -f keycloak-ingress.yaml