An ingestor defines how we interpret inbound data from a device. Either a device or device type can have an ingestor with which it is associated. The ingestor is comprised of a listener, a handler, and an associated translator ID.

Cloud vs Edge Ingestors

Ingestors can be run either in the cloud or on a downstream device such as a gateway device. If the cloud_ingestor parameter is true, the ingestor will be installed in the EdgeIQ Cloud ingest engine. If it is false, the translator will passed to the associated gateway device.

Listeners

The listener accepts messages with the specified transport and protocol.

TCP Server - Creates a TCP server on the specified address:port Listener type: tcp Listener object:

{
  "address": "0.0.0.0:4567"
}

TCP Client - Creates a TCP client to connect to the specified address:port Listener type: tcp_client Listener object:

{
  "address": "0.0.0.0:4567"
}

HTTP Server - Creates an HTTP server on the specified port. Compatible with router handler type. Listener type: http Listener object:

{
  "port": 4567
}

HTTP Client - Starts a polling timer with the specified interval that will issue the specified HTTP request Listener type: http_client Listener object:

{ 
  "url": "http://myurl.com/get_data",
  "method": "GET",
  "headers": {
    "Accept": "application/json"
  },
  "timeout": 5,
  "poll_interval": 30
}

Linux Device - Opens a connection to the specified Linux device port. Listener type: dev Listener object:

{ 
  "device": "/dev/ttys001",
  "baud": 115200
}

Shell Polling - Starts a polling timer with the specified interval that will execute the specified shell command. Listener type: shell_polling Listener object:

{ 
  "command": "cat /opt/myfile",
  "timeout": 5,
  "poll_interval": 30
}

TCP Modbus - Starts a polling timer with the specified interval that will issue a TCP Modbus command. "value" and "values" fields are used for write single and multiple request types only, respectively. "request_type" values: read_coils, read_inputs, read_input_registers, read_holding_register, read_fifo_queue, write_single_coil, write_multiple_coils, write_single_register, write_multiple_registers Listener type: tcp_modbus Listener object:

{ 
  "host": "192.168.10.10", 
  "port": 502, 
  "slave_id": 0, 
  "timeout": 30, 
  "keepalive": 300, 
  "poll_interval": 6000, 
  "params": { 
    "request_type": "[request_type]", 
    "address": 1, 
    "quantity": 1, 
    "value": 1, 
    "values": [1,2,3]
  } 
}

Handlers

The handler is what reads reports from the connection created by the listener. The handler must be configured such that it only reads complete messages, which are then passed along to the translator. In most cases, reports can be handled with one of three different configurable "out of the box" handlers:

Handler Type	Description
`csv`	A handler for reading comma-separated value reports (following RFC-4180). This handler is very configurable to handle a variety of options such as the delimiter character (e.g., to use "\|" instead of comma), the number of fields per record, whether to trim spaces from fields, etc.
`fixed`	A handler for reading reports of a fixed size. For example, configuring the message size at 42 bytes ensures that only complete 42-byte messages will be read.
`delimited`	A handler for reading reports that always end with a specific character or sequence of characters. These reports may be of varying length, but must always end with the specified character sequence.
`passthrough`	A handler used when the listener is not connection based (not TCP, UDP, or DEV). It that will pass the data straight from the listener to the translator.

Translators

Each ingestor will be associated with a specific inbound translator. The translator's purpose is to convert the data received via the listener and handler into EdgeIQ report JSON. EdgeIQ supports several types of inbound translators (see Translators API Docs):

Translator Type	Description
`anko`	Translators may be written in Anko, a "scriptable interpreter written in Go". The syntax is very similar to Go, but the behavior is more like a dynamic programming language. Anko can be extended to make any Go package accessible within the scripting environment. Examples of anko translators are available. Anko support is optional in an Edge build, although included by default.
`javascript`	Translators may also be written in a pure Go implementation of EcmaScript 5.1. Javascript support is optional in an Edge build, although included by default. Some customers may find that it is not desirable to include it on gateways with limited storage, as the interpreter increases the size of the resulting Edge binary.
`csv`	This translator is meant to be paired with the CSV handler described above. The JSON translator configuration describes each of the fields within a CSV report — including the field name, type (string, int, float, boolean), or whether to ignore the field as something we don't care about.
`template`	A Golang template can be defined to build the Report JSON. The output from the listener/handler can be accessed in the template definition using `{{ .output }}`

Translator Variables

Inside the translator, for translators of type template, certain variables are available based on the type of listener configured for the ingestor.

All Polling Listeners

Name	Description
gateway_unique_id	Unique ID of Gateway device where Ingestor is running
device_unique_id	Unique ID of device that Ingestor is associated with. If device is attached to a gateway, than `gateway_unique_id` and `device_unique_id` will be different. They will be identical when the Ingestor is associated with the gateway.
unique_id	alias for `device_unique_id`
device	The Device resource that is associated with the Ingestor. All data fields in Device are available
device_type	The Device resource that is associated with the Ingestor. All data fields in DeviceType are available

HTTP Client Listener

Name	Description
output	The HTTP response body. If the Content Type is `application/json`, then the `output` variable will support JSON querypaths, i.e. `output.foo` for `{ "foo": "bar" }`
body	alias for output
status_code	The HTTP Response status code value as an integer.

ICMP Client Listener

Name	Description
output	`success/fail`
status	alias for output
error	if status is `fail`, then the error value as a string. if status is `success`, than an empty string.
time_in_ms	latency in milliseconds

SNMP Client Listener

Name	Description
output	A JSON map where key is the SNMP Object ID and its value as either a string or integer
error	Error information as a string

Example

The following example shows an ingestor for a CSV message. The handler for CSV messages is used to specify information about the overall message format (which is expected to conform to RFC 4180). Details such as the names and data types of individual fields within the message are specified in the Translators.

A CSV handler contains a single csv_msg_options JSON object, which in turn contains the following fields:

Field	Type	Required?	Description
`delim`	string	N	The single-character delimiter. Defaults to comma if not specified.
`comment`	string	N	The comment character. If specified, lines starting with this character are ignored.
`fields_per_record`	int	N	The number of expected fields per record. If positive, each record must have the given number of fields. If 0, the number of fields is set on reading the first record, and future records must have the same field count. If negative, no check is made and records can have a variable number of fields.
`lazy_quotes`	bool	N	If true, a quote may appear in an unquoted field and a non-doubled quote may appear in a quoted field.
`trim_leading_space`	string	N	If true, leading white space in a field is ignored (even if the field delimiter is white space).

The ingestor JSON:

    {
      "name": "CSV over TCP Ingestor",
      "cloud_ingestor": false,
      "listener_type": "TCP",
      "listener": {
          "address": ":4532"    
        },
      "handler_type": "csv",
      "handler": {
        "csv_read_options": {
          "delim": ",",
          "comment": "#",
          "fields_per_record": -1,
        }
      },
      "translator_id": "559306d71d41c8b846000551"
    }

In this example, an edge ingestor will be created, listening on port 4532 for incoming CSV messages conforming to the csv_read_options. Incoming messages will be passed to the translator specified by translator_id.