Ingestors

An ingestor defines how we interpret inbound data from a device. Either a device or device type can have an ingestor with which it is associated. The ingestor is comprised of a listener, a handler, and an associated translator ID.

Cloud vs Edge Ingestors

Ingestors can be run either in the cloud or on a downstream device such as a gateway device. If the cloud_ingestor parameter is true, the ingestor will be installed in the EdgeIQ Cloud ingest engine. If it is false, the translator will passed to the associated gateway device.

Listeners

The listener accepts messages with the specified transport and protocol.

TCP Server - Creates a TCP server on the specified address:port
Listener type: tcp
Listener object:

{
  "address": "0.0.0.0:4567"
}

TCP Client - Creates a TCP client to connect to the specified address:port
Listener type: tcp_client
Listener object:

{
  "address": "0.0.0.0:4567"
}

HTTP Server - Creates an HTTP server on the specified port. Compatible with router handler type.
Listener type: http
Listener object:

{
  "port": 4567
}

HTTP Client - Starts a polling timer with the specified interval that will issue the specified HTTP request
Listener type: http_client
Listener object:

{ 
  "url": "http://myurl.com/get_data",
  "method": "GET",
  "headers": {
    "Accept": "application/json"
  },
  "timeout": 5,
  "poll_interval": 30
}

Linux Device - Opens a connection to the specified Linux device port.
Listener type: dev
Listener object:

{ 
  "device": "/dev/ttys001",
  "baud": 115200
}

Shell Polling - Starts a polling timer with the specified interval that will execute the specified shell command.
Listener type: shell_polling
Listener object:

{ 
  "command": "cat /opt/myfile",
  "timeout": 5,
  "poll_interval": 30
}

TCP Modbus - Starts a polling timer with the specified interval that will issue a TCP Modbus command. "value" and "values" fields are used for write single and multiple request types only, respectively. "request_type" values: read_coils, read_inputs, read_input_registers, read_holding_register, read_fifo_queue, write_single_coil, write_multiple_coils, write_single_register, write_multiple_registers
Listener type: tcp_modbus
Listener object:

{ 
  "host": "192.168.10.10", 
  "port": 502, 
  "slave_id": 0, 
  "timeout": 30, 
  "keepalive": 300, 
  "poll_interval": 6000, 
  "params": { 
    "request_type": "[request_type]", 
    "address": 1, 
    "quantity": 1, 
    "value": 1, 
    "values": [1,2,3]
  } 
}

Handlers

The handler is what reads reports from the connection created by the listener. The handler must be configured such that it only reads complete messages, which are then passed along to the translator. In most cases, reports can be handled with one of three different configurable "out of the box" handlers:

Handler TypeDescription
csvA handler for reading comma-separated value reports (following RFC-4180). This handler is very configurable to handle a variety of options such as the delimiter character (e.g., to use "|" instead of comma), the number of fields per record, whether to trim spaces from fields, etc.
fixedA handler for reading reports of a fixed size. For example, configuring the message size at 42 bytes ensures that only complete 42-byte messages will be read.
delimitedA handler for reading reports that always end with a specific character or sequence of characters. These reports may be of varying length, but must always end with the specified character sequence.
passthroughA handler used when the listener is not connection based (not TCP, UDP, or DEV). It that will pass the data straight from the listener to the translator.

Translators

Each ingestor will be associated with a specific inbound translator. The translator's purpose is to convert the data received via the listener and handler into EdgeIQ report JSON. EdgeIQ supports several types of inbound translators (see Translators API Docs):

Translator TypeDescription
ankoTranslators may be written in Anko, a "scriptable interpreter written in Go". The syntax is very similar to Go, but the behavior is more like a dynamic programming language. Anko can be extended to make any Go package accessible within the scripting environment. Examples of anko translators are available. Anko support is optional in an Edge build, although included by default.
javascriptTranslators may also be written in a pure Go implementation of EcmaScript 5.1. Javascript support is optional in an Edge build, although included by default. Some customers may find that it is not desirable to include it on gateways with limited storage, as the interpreter increases the size of the resulting Edge binary.
csvThis translator is meant to be paired with the CSV handler described above. The JSON translator configuration describes each of the fields within a CSV report — including the field name, type (string, int, float, boolean), or whether to ignore the field as something we don't care about.
templateA Golang template can be defined to build the Report JSON. The output from the listener/handler can be accessed in the template definition using {{ .output }}

Translator Variables

Inside the translator, for translators of type template, certain variables are available based on the type of listener configured for the ingestor.

All Polling Listeners

NameDescription
gateway_unique_idUnique ID of Gateway device where Ingestor is running
device_unique_idUnique ID of device that Ingestor is associated with. If device is attached to a gateway, than gateway_unique_id and device_unique_id will be different. They will be identical when the Ingestor is associated with the gateway.
unique_idalias for device_unique_id
deviceThe Device resource that is associated with the Ingestor. All data fields in Device are available
device_typeThe Device resource that is associated with the Ingestor. All data fields in DeviceType are available

HTTP Client Listener

NameDescription
outputThe HTTP response body. If the Content Type is application/json, then the output variable will support JSON querypaths, i.e. output.foo for { "foo": "bar" }
bodyalias for output
status_codeThe HTTP Response status code value as an integer.

ICMP Client Listener

NameDescription
outputsuccess/fail
statusalias for output
errorif status is fail, then the error value as a string. if status is success, than an empty string.
time_in_mslatency in milliseconds

SNMP Client Listener

NameDescription
outputA JSON map where key is the SNMP Object ID and its value as either a string or integer
errorError information as a string

Example

The following example shows an ingestor for a CSV message. The handler for CSV messages is used to specify information about the overall message format (which is expected to conform to RFC 4180). Details such as the names and data types of individual fields within the message are specified in the Translators.

A CSV handler contains a single csv_msg_options JSON object, which in turn contains the following fields:

FieldTypeRequired?Description
delimstringNThe single-character delimiter. Defaults to comma if not specified.
commentstringNThe comment character. If specified, lines starting with this character are ignored.
fields_per_recordintNThe number of expected fields per record. If positive, each record must have the given number of fields. If 0, the number of fields is set on reading the first record, and future records must have the same field count. If negative, no check is made and records can have a variable number of fields.
lazy_quotesboolNIf true, a quote may appear in an unquoted field and a non-doubled quote may appear in a quoted field.
trim_leading_spacestringNIf true, leading white space in a field is ignored (even if the field delimiter is white space).

The ingestor JSON:

    {
      "name": "CSV over TCP Ingestor",
      "cloud_ingestor": false,
      "listener_type": "TCP",
      "listener": {
          "address": ":4532"    
        },
      "handler_type": "csv",
      "handler": {
        "csv_read_options": {
          "delim": ",",
          "comment": "#",
          "fields_per_record": -1,
        }
      },
      "translator_id": "559306d71d41c8b846000551"
    }

In this example, an edge ingestor will be created, listening on port 4532 for incoming CSV messages conforming to the csv_read_options. Incoming messages will be passed to the translator specified by translator_id.