HV-VES (High Volume VES)

Overview

Component description can be found under HV-VES Collector.

TCP Endpoint

HV-VES is exposed as NodePort service on Kubernetes cluster on port 30222/tcp. By default, as of the Frankfurt release, all TCP communications are secured using SSL/TLS. Plain, insecure TCP connections without socket data encryption can be enabled if needed. (see ref:ssl_tls_authorization).

Without TLS, client authentication/authorization is not possible. Connections are stream-based (as opposed to request-based) and long-running.

Communication is wrapped with thin Wire Transfer Protocol, which mainly provides delimitation.

-- Wire Transfer Protocol (binary, defined using ASN.1 notation)
-- Encoding: use "direct encoding" to the number of octets indicated in the comment [n], using network byte order.

WTP DEFINITIONS ::= BEGIN

-- Used to sent data from the data provider
WtpData ::= SEQUENCE {
    magic           INTEGER (0..255),           -- [1] always 0xAA
    versionMajor    INTEGER (0..255),           -- [1] major interface version, forward incompatible with previous major version, current value: 1
    versionMinor    INTEGER (0..255),           -- [1] minor interface version, forward compatible with previous minor version, current value: 0
    reserved        OCTET STRING (SIZE (3)),    -- [3] reserved for future use (ignored, but use 0)
    payloadId       INTEGER (0..65535),         -- [2] payload type: 0x0000=undefined, 0x0001=ONAP VesEvent (protobuf)
    payloadLength   INTEGER (0..4294967295).    -- [4] payload length in octets
    payload         OCTET STRING                -- [length as per payloadLength]
}

END

Payload is binary-encoded, using Google Protocol Buffers (GPB) representation of the VES Event.

/*
 * ============LICENSE_START=======================================================
 * dcaegen2-collectors-veshv
 * ================================================================================
 * Copyright (C) 2018 NOKIA
 * ================================================================================
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *      http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 * ============LICENSE_END=========================================================
 */
syntax = "proto3";
package org.onap.ves;

message VesEvent                            // top-level message, currently the maximum event size supported by the HV-VES Collector is 1 MiB
{
    CommonEventHeader commonEventHeader=1;  // required

    bytes eventFields=2;                    // required, payload
        // this field contains a domain-specific GPB message
        // the field being opaque (bytes), the decoding of the payload occurs in a separate step
        // the name of the GPB message for domain XYZ is XyzFields
        // e.g. for domain==perf3gpp, the GPB message is Perf3gppFields
}

// VES CommonEventHeader adapted to GPB (Google Protocol Buffers)

message CommonEventHeader
{
    string version = 1;                     // required, "version of the gpb common event header", current value "1.0"
    string domain = 2;                      // required, "the eventing domain associated with the event", allowed values:
                                            // fault, heartbeat, measurement, mobileFlow, other, pnfRegistration, sipSignaling,
                                            // stateChange, syslog, thresholdCrossingAlert, voiceQuality, perf3gpp

    uint32 sequence = 3;                    // required, "ordering of events communicated by an event source instance or 0 if not needed"

    enum Priority
    {
        PRIORITY_NOT_PROVIDED = 0;
        HIGH = 1;
        MEDIUM = 2;
        NORMAL = 3;
        LOW = 4;
    }
    Priority priority = 4;                  // required, "processing priority"

    string eventId = 5;                     // required, "event key that is unique to the event source"
    string eventName = 6;                   // required, "unique event name"
    string eventType = 7;                   // "for example - guest05,  platform"

    uint64 lastEpochMicrosec = 8;           // required, "the latest unix time aka epoch time associated with the event from any component--as microseconds elapsed since 1 Jan 1970 not including leap seconds"
    uint64 startEpochMicrosec = 9;          // required, "the earliest unix time aka epoch time associated with the event from any component--as microseconds elapsed since 1 Jan 1970 not including leap seconds"

    string nfNamingCode = 10;               // "4 character network function type, aligned with vnf naming standards"
    string nfcNamingCode = 11;              // "3 character network function component type, aligned with vfc naming standards"
    string nfVendorName = 12;               // " Vendor Name providing the nf "

    bytes reportingEntityId = 13;           // "UUID identifying the entity reporting the event, for example an OAM VM; must be populated by the ATT enrichment process"
    string reportingEntityName = 14;        // required, "name of the entity reporting the event, for example, an EMS name; may be the same as sourceName should match A&AI entry"
    bytes sourceId = 15;                    // "UUID identifying the entity experiencing the event issue; must be populated by the ATT enrichment process"
    string sourceName = 16;                 // required, "name of the entity experiencing the event issued use A&AI entry"
    string timeZoneOffset = 17;             // "Offset to GMT to indicate local time zone for the device"
    string vesEventListenerVersion = 18;    // required, "Version of the VesEvent Listener", current value "7.2"

    reserved "InternalHeaderFields";        // "enrichment fields for internal VES Event Listener service use only, not supplied by event sources"
    reserved 100;
}

HV-VES makes routing decisions based on the content of the domain field or stndDefinedNamespace field in case of stndDefined events.

The PROTO file, which contains the VES CommonEventHeader, comes with a binary-type Payload (eventFields) parameter, where domain-specific data should be placed. Domain-specific data are encoded as well with GPB. A domain-specific PROTO file is required to decode the data.

API towards DMaaP

HV-VES Collector forwards incoming messages to a particular DMaaP Kafka topic based on the domain (or stndDefinedNamespace) and configuration. Every Kafka record is comprised of a key and a value. In case of HV-VES:

  • Kafka record key is a GPB-encoded CommonEventHeader.

  • Kafka record value is a GPB-encoded VesEvent (CommonEventHeader and domain-specific eventFields).

In both cases raw bytes might be extracted using org.apache.kafka.common.serialization.ByteArrayDeserializer. The resulting bytes might be further passed to parseFrom methods included in classes generated from GPB definitions. WTP is not used here - it is only used in communication between PNF/VNF and the collector.

In case of Helm deployment routing is defined in values.yaml file in HV-VES Helm Chart.

Supported domains

Domains that are currently supported by HV-VES:

  • perf3gpp - basic domain to Kafka topic mapping

  • stndDefined - specific routing, when event has this domain, then stndDefinedNamespace field value is mapped to Kafka topic

For domains descriptions, see Domains supported by HV-VES

HV-VES behaviors

Connections with HV-VES are stream-based (as opposed to request-based) and long-running. In case of interrupted or closed connection, the collector logs such event but does not try to reconnect to client. Communication is wrapped with thin Wire Transfer Protocol, which mainly provides delimitation. Wire Transfer Protocol Frame:

  • is dropped after decoding and validating and only GPB is used in further processing.

  • has to start with MARKER_BYTE, as defined in protocol specification (see TCP Endpoint). If MARKER_BYTE is invalid, HV-VES disconnects from client.

HV-VES decodes only CommonEventHeader from GPB message received. Collector does not decode or validate the rest of the GPB message and publishes it to Kafka topic intact. Kafka topic for publishing events with specific domain can be configured through Consul service as described in Run-Time configuration. In case of Kafka service unavailability, the collector drops currently handled messages and disconnects the client.

Messages handling:

  • HV-VES Collector skips messages with unknown/invalid GPB CommonEventHeader format.

  • HV-VES Collector skips messages with unsupported domain. Domain is unsupported if there is no route for it in configuration (see Run-Time configuration).

  • HV-VES Collector skips messages with invalid Wire Frame format, unsupported WTP version or inconsistencies of data in the frame (other than invalid MARKER_BYTE).

  • HV-VES Collector interrupts connection when it encounters a message with too big GPB payload. Default maximum size and ways to change it are described in Deployment.

Note

xNF (VNF/PNF) can split messages bigger than 1 MiB and set sequence field in CommonEventHeader accordingly. It is advised to use smaller than 1 MiB messages for GPBs encoding/decoding efficiency.

  • Skipped messages (for any of the above reasons) might not leave any trace in HV-VES logs.