MSB(Microservices Bus)

Microservices Bus(MSB) provides a reliable, resilient and scalable communication and governance infrastructure to support ONAP Microservice Architecture including service registration/discovery, external API gateway, internal API gateway, client SDK, Swagger SDK, etc. It’s a pluggable architecture, plugins can be added to MSB to provide whatever functionalities you need, such as an auth plugin can be used to provide centralized authentication & authorization. MSB also provides a service portal to manage the REST APIs.

MSB is platform independent, while it is integrated with Kubernetes(OOM) to provide transparent service registration for ONAP microservices, MSB also supports OpenStack(Heat) and bare metal deployment.

This document is aimed to give the users some brief introductions on MSB from different aspects. The users could find useful information, e.g. the architecuture, the APIs and the installation steps, etc. as well as the release notes of MSB in this document.

Platform

Microservices Bus provide key infrastructure functionalities to support ONAP microservice architecture including service registration/discovery, service gateway, service load balancer and service governance. It’s a pluggable architecture so it can be extended with plugins to provide value added services such as centralized authentication for APIs. Microservices Platform also provides a GUI portal for service management.

Architecture

Microservices Bus(MSB) provides a reliable, resilient and scalable communication and governance infrastructure to support ONAP Microservice Architecture including service registration/discovery, external API gateway, internal API gateway, client SDK, Swagger SDK, etc. It’s a pluggable architecture, plugins can be added to MSB to provide whatever functionalities you need, such as an auth plugin can be used to provide centralized authentication & authorization. MSB also provides a service portal to manage the REST APIs.

MSB is platform independent, while it is integrated with Kubernetes(OOM) to provide transparent service registration for ONAP microservices, MSB also supports OpenStack(Heat) and bare metal deployment.

MSB is mainly comprised of three parts: Discovery, API Gateway and MSB Client SDK. MSB also provide a swagger SDK which could generate swagger REST service description files..

Registry is the storage of service information, MSB leverage Consul as the service registry.

MSB Discovery provides REST APIs for service discovery and registration.

API Gateway provides service request routing, load balancing and service governance. It can be deployed as external Gateway or Internal Gateway.

MSB SDK is a client-side Java SDK which can be used for point to point communication of microservices.

Offered APIs

The MSB provides the following APIs.

• Service Registration.

• Service Unregistration.

• Service Discovery.

• MSB Java SDK.

• Swagger SDK.

Discover more about `MSB API & User Guide<https://wiki.onap.org/pages/viewpage.action?pageId=20873883>`_.

Consumed APIs

In the Amsterdam release, MSB does not consume any APIs provided by other ONAP components.

Delivery

Describe how functions are packaged into run-time components. For some components a block diagram may be useful.

Docker

• MSB Discovery

• MSB API Gateway

Docker containers can be downloaded from ONAP docker repository https://nexus3.onap.org/.

jar

• MSB Client SDK

• Swagger SDK

Jars can be downloaded from ONAP docker repository https://nexus.onap.org/.

Logging & Diagnostic Information

Description of how to interact with and diagnose problems with the components as delivered.

Where to Access Information

You can use docker command such as docker exec or docker logs to interact with and diagnose problems of MSB.

Error / Warning Messages

Installation

MSB can run as docker, it’s very handy to try it at your laptop. For production, MSB supports to be deployed as a cluster to provide a scalable microservice communication infrastructure with kubernetes.

Run MSB on the a single host using host network

This deployment approach is only for testing. MSB is consist of multiple microservices, for testing, the easiest way is to run them in the same host by using host network.

• Run the Consul dockers

sudo docker run -d --net=host --name msb_consul consul:0.9.3

• Run the MSB dockers

Login the ONAP docker registry first:

docker login -u docker -p docker nexus3.onap.org:10001

Run MSB dockers

docker run -d --net=host --name msb_discovery nexus3.onap.org:10001/onap/msb/msb_discovery
docker run -d --net=host -e "ROUTE_LABELS=visualRange:1" --name msb_internal_apigateway nexus3.onap.org:10001/onap/msb/msb_apigateway

• Register your REST service to MSB via curl

For testing, we can register the services via curl.

curl -X POST \
-H "Content-Type: application/json" \
-d '{"serviceName": "aai", "version": "v8", "url": "/aai/v8","protocol": "REST", "path": "/aai/v8", "nodes": [ {"ip": "10.74.215.65","port": "8443"}]}' \
"http://127.0.0.1:10081/api/microservices/v1/services”

• Access the rest service via api gateway

curl http://127.0.0.1/api/aai/v8/cloud-infrastructure/cloud-regions

• Browse the registered services

  Open MSB Web GUI portal in your browser: http://127.0.0.1/msb, you can see all the registered services. If the registered service support swagger, you can see the REST API documentation and test the registered services via the swagger UI integrated in MSB.

Run MSB using default docker network

In case that you don’t want to use host network to set up the MSB dockers, you need to export the port and specify the environment variables to let msb components know each other.

• Run the Consul dockers

sudo docker run -d -p 8500:8500  --name msb_consul consul:0.9.3
CONSUL_IP=`sudo docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' msb_consul`

• Run the MSB dockers

Login the ONAP docker registry first:

docker login -u docker -p docker nexus3.onap.org:10001

Run MSB dockers

sudo docker run -d  -p 10081:10081  -e CONSUL_IP=$CONSUL_IP --name msb_discovery nexus3.onap.org:10001/onap/msb/msb_discovery
DISCOVERY_IP=`sudo docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' msb_discovery`
sudo docker run -d -p 80:80 -e CONSUL_IP=$CONSUL_IP -e SDCLIENT_IP=$DISCOVERY_IP -e "ROUTE_LABELS=visualRange:1" --name msb_internal_apigateway nexus3.onap.org:10001/onap/msb/msb_apigateway

Deploy MSB with kubernetes

The chart for MSB running in kubernetes cluster is available here:

https://gerrit.onap.org/r/gitweb?p=oom.git;a=tree;f=kubernetes/msb;h=17f8fd89791b81f1e981716dcffdb3e2e90299ae;hb=refs/heads/master

Configuration

No machanism for customized configurtions is provided in this release. Such functionalities will be provided in the future if necessary.

Human Interfaces

MSB provides a Web GUI Portal for users. Users can register services and browse all the registered services. If the registered service support swagger, user can see the REST API documentation and test the registered services via the swagger UI integrated in MSB.

Target Users

ONAP Administrator.

Interface Type

Web GUI.

Access

Http://MSB_API_GATEWAY_IP:80/msb

MSB User-guide

Overview

Micro-Service Bus (MSB) provides a facility to register and expose ONAP Rest API.

It is particulary usefull to expose some API endpoints that are not published by default when installing ONAP with OOM installer.

There are two methods to register an API in MSB:

• via MSB Portal/GUI

• via MSB Rest API

Using MSB portal

MSB Portal/GUI is available on the following URL (adapt to your ONAP installation)

http://msb.api.discovery.simpledemo.onap.org:30282/iui/microservices/default.html

The following screen should appears

click on the “Service Discover” icon to get the list of registred API (it will take several seconds to get the next screen)

Note: majority of Registred API have been declared using MSB API and they can not be modified/deleted via MSB Portal/GUI.

click on “view” icon in “Control” column to have a more detailed description of the registred API

click on “Service Register” icon to start the register process for a new API

“Service Name” will be the new API name in MSB and will be used to build the URL that will be exposed.

“Url” is the url of the existing ONAP API that we are registering

“Protocol” must be selected in the list (usualy REST)

“Version” will be part of the exposed URL

“Visual Range” must be selected (usualy inSystem)

click on “Add Host” to provide information about IP address and port of the API that we are registering.

Tips: from an ONAP platform installed via OOM, to know about the IP/port, use Kubctl commands

example: to know about IP addresses used by ONAP Policy components

kubectl get svc -n onap | grep policy

Here after an example of Registered API for ONAP SO:

Using MSB API

Get the list of registred API in MSB

curl -X GET \
http://msb.api.discovery.simpledemo.onap.org:30280/api/microservices/v1/services \
-H 'Content-Type: application/json' \
-H 'cache-control: no-cache'

response (small extract only):

    ],
    "metadata": [],
    "labels": [],
    "status": "1",
    "is_manual": false
},
{
    "serviceName": "aai-business",
    "version": "v16",
    "url": "/aai/v16/business",
    "protocol": "REST",
    "visualRange": "1",
    "lb_policy": "ip_hash",
    "publish_port": "",
    "namespace": "",
    "network_plane_type": "",
    "host": "",
    "path": "",
    "enable_ssl": true,
    "nodes": [
        {
            "ip": "10.233.69.5",
            "port": "8447",
            "checkType": "",
            "checkUrl": "",
            "tls_skip_verify": true,
            "ha_role": "",
            "nodeId": "_v16_aai-business_10.233.69.5_8447",
            "status": "passing"
        }
    ],
    "metadata": [],
    "labels": [],
    "status": "1",
    "is_manual": false
},

Register a new API (example for ONAP policy-pap API)

“Service Name” will be the new API name in MSB and will be used to build the URL that will be exposed.

“Url” is the url of the existing ONAP API that we are registering

“Protocol” must be selected in the list (usualy REST)

“Version” will be part of the exposed URL

“Visual Range” must be selected (usualy inSystem)

“enable_ssl” must be set to “true” is using https between MSB and API service that we are registering.

“nodes” is a list of IP addresses and port of the API that we are registering.

Tips: from an ONAP platform installed via OOM, to know about the IP/port, use Kubctl commands “enable_ssl” must be set to “true” is using https between MSB and API service that we are registering.

curl -X POST \
http://msb.api.discovery.simpledemo.onap.org:30280/api/microservices/v1/services \
-H 'Content-Type: application/json' \
-H 'cache-control: no-cache' \
-d '{
"serviceName": "policy-api",
"version": "v1",
"url": "/policy/api/v1",
"protocol": "REST",
"visualRange": "1",
"enable_ssl": true,
"nodes": [

    {
    "ip": "10.233.35.125",
    "port": "6969",
    "ttl": 0
    }
]
}
'

Delete an API from MSB (=unregister an API):

curl -X DELETE \
http://msb.api.discovery.simpledemo.onap.org:30280/api/microservices/v1/services/policy-pap/version/v1/nodes/10.233.15.213/6969 \
-H 'Content-Type: application/json' \
-H 'cache-control: no-cache' \
-d '{
"serviceName": "policy-pdp-legacy",
"version": "v1",
"url": "/pdp/api",
"protocol": "REST",
"visualRange": "1",
"enable_ssl": true,
"nodes": [

    {
    "ip": "10.233.77.14",
    "port": "8081",
    "ttl": 0
    }
]
}
'

Using a registred API

Once registered, the API can then be accessible using the following URL/Port

http://msb.api.discovery.simpledemo.onap.org:30280/api/{{Service Name}}/{{Version}}/{{resource}}

where {{Service Name}} and {{Version}} are mapped to what have been registered

{{resource}} is to be replace by the object that the API is able to manage

some examples:

to get the service model list from SDC via MSB (this API is pre-registred by default in ONAP):

curl -X GET \
http://msb.api.discovery.simpledemo.onap.org:30280/api/sdc/v1/catalog/services \
-H 'Accept: application/json' \
-H 'Authorization: Basic YWFpOktwOGJKNFNYc3pNMFdYbGhhazNlSGxjc2UyZ0F3ODR2YW9HR21KdlV5MlU=' \
-H 'Content-Type: application/json' \
-H 'USER_ID: cs0008' \
-H 'X-FromAppId: ONAP-Test' \
-H 'X-TransactionId: ONAP-Test' \
-H 'cache-control: no-cache' \
-H 'x-ecomp-instanceid: ONAP-Test'

to get the customer list from AAI via MSB (this API is pre-registred by default in ONAP):

curl -X GET \
http://msb.api.discovery.simpledemo.onap.org:30280/api/aai-business/v16/customers \
-H 'Accept: application/json' \
-H 'Authorization: Basic QUFJOkFBSQ==' \
-H 'Content-Type: application/json' \
-H 'X-FromAppId: AAI' \
-H 'X-TransactionId: 808b54e3-e563-4144-a1b9-e24e2ed93d4f' \
-H 'cache-control: no-cache'

to get the list of policy models from Policy via MSB (this policy API needs to be registred):

curl -X GET \
http://msb.api.discovery.simpledemo.onap.org:30280/api/policy-api/v1/policytypes \
-H 'Authorization: Basic aGVhbHRoY2hlY2s6emIhWHp0RzM0' \
-H 'X-ONAP-RequestID: 9ac7ce8e-a867-4269-bc52-c8236b7fdad6' \
-H 'cache-control: no-cache'

Microservices Bus(MSB) provides a reliable, resilient and scalable communication and governance infrastructure to support ONAP Microservice Architecture including service registration/discovery, external API gateway, internal API gateway, client SDK, Swagger SDK, etc. It’s a pluggable architecture, plugins can be added to MSB to provide whatever functionalities you need, such as an auth plugin can be used to provide centralized authentication & authorization. MSB also provides a service portal to manage the REST APIs.

MSB is platform independent, while it is integrated with Kubernetes(OOM) to provide transparent service registration for ONAP microservices, MSB also supports OpenStack(Heat) and bare metal deployment.

Release Notes

Version: 1.2.7 Guilin Release

Release Date

2020-11-19

New Features

Bug Fixes

• MSB-539 Fix nginx say 10000 worker_connections are not enough

• MSB-540 nginx in discovery deletes useless log printing

• MSB-541 Discovery supports blocking monitoring time exceeding 10min

Fixed Security Issues

Known Issues - MSB-519 MSB has python 2.7 pods - MSB-520 MSB has java 8 pods - MSB-521 MSB certificates are too long and have bad owner - MSB-529 MSB dockers contain GPLv3

Security Notes

The remaining issues are supposed to be fixed in the next release.

Quick Links:

• MSB project page

• Passing Badge information for MSB

• Project Vulnerability Review Table for MSB

Upgrade Notes

N/A

Deprecation Notes

Plain HTTP APIs have been removed from Frankfurt release, please use HTTPS instead.

N/A

Other

N/A

Version: 1.2.6 Frankfurt Release

Release Date

2020-May-07

New Features

Bug Fixes

• MSB-470 Pairwise testing fails from Portal to MSB

• MSB-468 MSB verification job is unstable

• MSB-467 Certificate in MSB is set to use *.zte.com.cn

• MSB-451 Code coverage data can’t show at sonarcloud

Fixed Security Issues

• MSB-465 Update alpine to the latest version

• MSB-462 Pods still run as root

• MSB-418 Solve MSB vulnerability onap-msb-discovery-jackson-databind

• MSB-417 Solve MSB vulnerability onap-msb-apigateway-jackson-databind

• MSB-410 Solve MSB vulnerability onap-msb-java-sdk-jackson-databind

• MSB-408 Solve MSB vulnerability onap-msb-java-sdk-commons-codec

• MSB-407 Solve MSB vulnerability onap-msb-java-sdk-okhttp

• MSB-398 Remove plain HTTP Rest APIs

• MSB-386 General sonar fixes

Known Issues

Security Notes

MSB code has been formally scanned during build time using NexusIQ and all Critical vulnerabilities have been addressed. The remaining issues are identified as false positive and their risk assessment have been documented as part of the project.

Quick Links:

• MSB project page

• Passing Badge information for MSB

• Project Vulnerability Review Table for MSB

Upgrade Notes

N/A

Deprecation Notes

Plain HTTP APIs have been removed from Frankfurt release, please use HTTPS instead.

N/A

Other

N/A

Version: 1.2.5 EI-Alto Release

Release Date

2019-09-05

New Features

This release only contains a few bug fixes and small features.

New Features

• MSB-332 Delete custom services via the MSB management UI

• MSB-151 Support registering HTTPS service on the MSB management UI

Bug Fixes

• MSB-372 Empty pages that looks like they should be deleted

• MSB-374 MSB stripe of the headers with underscore and it blocks SDC API functionality

Known Issues

Security Notes

MSB code has been formally scanned during build time using NexusIQ and all Critical vulnerabilities have been addressed. The remaining issues are identified as false positive and their risk assessment have been documented as part of the project.

Quick Links:

• MSB project page

• Passing Badge information for MSB

• Project Vulnerability Review Table for MSB

Upgrade Notes

N/A

Deprecation Notes

N/A

Other

N/A

Version: 1.2.4 Dublin Release

Release Date

2019-06-10

New Features

This release only contains some security improvements and a few fixes.

Security improvements

• MSB MSB-295 Nexus IQ Issue: bootstrap

• MSB MSB-320 Run API Gateway as a non-root user

• MSBMSB-321 Run Discovery as a non-root user

• MSB MSB-322 Run Kube2msb as a non-root user

• MSB MSB-328 Security issue reported by Nexus-iq : jetty-http

• MSB MSB-329 Security issue reported by Nexus-iq : spring-core

• MSB MSB-330 Security issue reported by Nexus-iq : commons-beanutils

• MSB MSB-331 Jackson datatype security issue

Bug Fixes

• MSB-98 No information is available to select proper node IP from registered services

• MSB-281 improve CLM for swagger sdk

• MSB-325 consul container is outdated

• MSB-326 non STAGING version on master

Known Issues

Security Notes

MSB code has been formally scanned during build time using NexusIQ and all Critical vulnerabilities have been addressed. The remaining issues are identified as false positive and their risk assessment have been documented as part of the project.

Quick Links:

• MSB project page

• Passing Badge information for MSB

• Project Vulnerability Review Table for MSB

Upgrade Notes

N/A

Deprecation Notes

N/A

Other

N/A

Version: 1.2.3

Release Date

2018-11-30

New Features

In Casablanca Release, MSB mainly focuses on the integration of Istio service mesh with ONAP to enhance OMSA, while keeping the Istio integration compatible with the existing MSB API Gateway approaches.

How to manage ONAP microservices with Istio service mesh:

• https://wiki.onap.org/display/DW/Manage+ONAP+Microservices+with+Istio+Service+Mesh

• https://wiki.onap.org/display/DW/Manage+ONAP+Microservices+with+Istio+Service+Mesh-Mutual+TLS+Authentication+Enabled

Bug Fixes

• MSB-196 IUI displays raw placeholder texts when failed to load translation

• MSB-291 Incomplete Apache-2.0 header

• MSB-293 Portal to MSB pairwise test failing in WindRiver with OOM deployment

• MSB-294 Nexus IQ Issue: okhttp3

• MSB-296 Nexus IQ Issue: guava

• MSB-297 MSB CSIT failed

• MSB-298 Release MSB artifact version 1.2.0

• MSB-300 Incomplete Apache-2.0 header

• MSB-301 Can’t access aai resource http url via msb api gateway

Known Issues

• MSB-295 Nexus IQ Issue: bootstrap

• MSB-198 MSB GUI can not register a service mapped to an HTTPS endpoint

Security Notes

MSB code has been formally scanned during build time using NexusIQ and all Critical vulnerabilities have been addressed, items that remain open have been assessed for risk and actions to be taken in future release. The MSB open Critical security vulnerabilities and their risk assessment have been documented as part of the project.

Quick Links:

• MSB project page

• Passing Badge information for MSB

• Project Vulnerability Review Table for MSB

Upgrade Notes

N/A

Deprecation Notes

N/A

Other

N/A

Version: 1.1.0

Release Date

2018-06-07

New Features In Beijing release, MSB project mainly focused on the Platform Maturity requirements of ONAP, including the scalability and security. Some new features which were requested when integrated with other projects, such as websocket support, service registration at K8S Pod level, multiple versions of services, etc. have also been added to this release.

• MSB-117 Support horizontal scaling

• MSB-140 Providing HTTPS endpoint at API gateway

• MSB-146 Support service registration at K8s Pod level

• MSB-152 MSB JAVA SDK supports HTTPS service registration

• MSB-156 Support websocket request forwarding

• MSB-178 Support registering multiple versions under a service name

• MSB-179 Integration MSB GUI to Portal project

Bug Fixes

• MSB-92 Microservice delete is reporting 500, though it deleted the service

• MSB-102 The msb client has heavy dependencies

• MSB-150 Kube2msb doesn’t unregister service

• MSB-153 MSB kube2msb registrator does not register LoadBalancer type service

• MSB-187 MSB discovery API in swagger is not published

• MSB-195 HTTP protocol used over HTTPS port

Known Issues

N/A

Security Notes

MSB code has been formally scanned during build time using NexusIQ and all Critical vulnerabilities have been addressed, items that remain open have been assessed for risk and determined to be false positive. The MSB open Critical security vulnerabilities and their risk assessment have been documented as part of the project.

Quick Links:

• MSB project page

• Passing Badge information for MSB

• Project Vulnerability Review Table for MSB

Upgrade Notes

N/A

Deprecation Notes

N/A

Other

N/A

Version: 1.0.0

Release Date

2017-11-16

New Features Initial release of Microservices Bus (MSB) for Open Network Automation Platform (ONAP). MSB provides core functionalities to support ONAP microservices architecture, including SDK for rapid microservie development, infrastructure for service communication and tools for service governance.

The current release of MSB is mainly composed of the following components:

msb/apigateway

Provides client request routing, client request load balancing, transformation, such as https to http, authentication & authorization for service request with plugin of auth service provider, service request logging, service request rate-limiting, service monitoring, request result cache, solve cross-domain issue for web application and other functionalities with the pluggable architecture capability.

msb/discovery

Provides service registration and discovery for ONAP microservices, which leverage Consul and build an abstract layer on top of it to make it agnostic to the registration provider and add needed extension.

msb/java-sdk

Provides a JAVA SDK for rapid microservices development, including service registration, service discovery, request routing, load balancing, retry, etc.

msb/swagger-sdk

Swagger sdk helps to generate swagger.json and java client sdk during the build time, it also helps to provide the swagger.json at the given URI in the run time.

In the future release, MSB plans to provide service mesh for ONAP.

Bug Fixes

• MSB-94 Vendor name(ZTE) on the MSB Portal tiltle

• MSB-91 Duplicate class variable in service sub-classes

• MSB-88 The path parameter has been lost when register services in demo project

• MSB-87 MSB JAVA SDK dosen’t release stage binary

• MSB-85 API Gateway UT coverage doesn’t show up in Sonar

• MSB-74 Jenkins Integration Test job failed

• MSB-73 Can’t register service by using MSB Api gateway 80 port

• MSB-72 Unit test coverage data is incorrect

• MSB-71 API Gateway service Registration and discovery api causes confusion

• MSB-70 Swagger SDK site job build failed

• MSB-69 Discovery checkstyle issue

• MSB-68 Discovery daily build jenkins job failed

• MSB-67 API Gateway check style warnning

• MSB-66 API Gateway daily build failed

• MSB-60 API gateway test coverage data not in snoar

• MSB-59 Swagger SDK build failed

• MSB-58 MSB Java SDK Jenkins merge job failed

• MSB-57 Discovery site jenkins job failed

• MSB-55 Discovery site jenkins job failed

• MSB-54 API Gateway site jenkins job failed

• MSB-21 Merge and daily jenkins job failed

• MSB-17 Release version java daily job failed

Known Issues

• MSB-92 Microservice delete is reporting 500, though it deleted the service

Security Issues

None

Upgrade Notes

This is an initial release

Deprecation Notes

N/A

Other

N/A

End of Release Notes