Table of Contents

Introduction

This document describes the Farsight Streaming API Framing (SAF) protocol, a data transfer protocol. It is initially used by DNSDB to transfer pre-standard IETF COF format objects.

Audience

This document is intended for programmers who want to write applications that can interact with the RESTful DNSDB API using JSON and HTTP.

Format

The data is transmitted as newline delimited JSON lines, aka JSONL or NDJSON.

All string values are UTF-8. All attribute names are US-ASCII.

Whitespace within the JSONL structure should be ignored.

Each JSONL object contains the optional attributes:

  1. “cond” is a condition. It is a scalar enum of string values:

    • “begin”
    • “ongoing”
    • “succeeded”
    • “limited”
    • “failed”
  2. “msg” is a human readable message. It is a scalar string which ought to be reported to the end-user or in the system log.

  3. “obj” contains data appropriate for the protocol using SAF encapsulation. It is a JSON object. For the DNSDB API, it contains a pre-standard IETF COF object.

Other attributes, if present, would be the subject of some future protocol revision.

The condition “begin” MUST be in the first, initiating JSONL object. This object MUST contain a “cond” and MAY contain a “msg”.

The conditions “succeeded”, “limited”, and “failed” are terminating conditions and MUST be in the last JSONL object.

JSONL objects after the first and before the last MAY contain an “obj” attribute, MAY contain a “msg” attribute, and MAY either specify “cond” to be “ongoing” OR omit the “cond”. The default value for “cond” is “ongoing”.

If a terminating condition is missing (i.e. the last object in the data stream does not contain one) then the client should interpret this as truncation and generate an appropriate error message to report to the end-user or in the system log. The previously received data objects may be discarded or used subject to the knowledge they may be incomplete. This is also the case if the data stream terminates with a TCP error. A client is free to retry a truncated operation.

If any JSONL object cannot be parsed as valid JSON then both that JSONL object and all remaining JSONL objects must be discarded and some warning must be issued to the user or system log.

If the terminating condition is “succeeded” or “limited” then the data objects are valid.

If the terminating condition is “failed” then the data objects may be discarded or used subject to the knowledge they may be incomplete.

If a JSONL object after the first and before the last, either having no “cond” attribute or having a “cond” value of “ongoing”, contains a “msg” attribute then the “msg” describes an informational or warning condition which has no decisive bearing on the validity of the objects in the response stream. Other values of “msg” SHOULD be reported to the end-user or in the system log.

If a JSONL object after the first and before the last, either having no “cond” attribute or having a “cond” value of “ongoing” contains neither a “obj” nor a “msg”, then this is a “keep alive” message, designed to keep the connection open. This is useful for queries that take a long time to process before reporting anything to the user.

Keep alives are defined in this protocol but are not currently implemented.

State Machine

The state machine grammar is:

initiating JSONL object :=
  { "cond": "begin" [, "msg": "$message"]}

ongoing JSONL object :=
  { [ "cond": "ongoing",] ["obj": { $object }] [, "msg": "$message"]}

terminating JSONL object :=
    { "cond": "succeeded" [, "msg": "$message"]}
    |
    { "cond": "limited" [, "msg": "$message"]}
    |
    { "cond": "failed" [, "msg": "$message"]}

START => initiating JSONL object => ongoing JSONL object OR terminating JSONL object

ongoing JSONL object => ongoing JSONL object OR terminating JSONL object

terminating JSONL object => END

Examples

Simple successful example

{"cond": "begin"}
{"obj":{"count":10392,"time_first":138126549...}}
{"cond": "succeeded"}

The second JSONL object implicitly has the default condition of “ongoing”.

Equivalent simple successful example

{"cond":"begin"}
{"cond":"ongoing", "obj":{"count":10392,"time_first":138126549...}}
{"cond":"succeeded"}

Long-running example with “alive” messages

This example shows a query that took a while to return the first response, and then had occasional delays in responding. The example also shows varying amounts of whitespace, as the whitespace should be ignored.

{"cond":"begin"}
{"cond":"ongoing"}
{"cond":"ongoing"}
{"cond":"ongoing", "obj":{"count":10392,"time_first":138126549...}}
...
{"cond":"ongoing"}
{"cond":"ongoing", "obj": {"count": 1234,"time_first": 238126549...}}
...
{}
...
{ "cond" : "ongoing", "obj" : {"count" :456, "time_first":338126549...} }
{"cond":"succeeded"}

Limited example

{"cond": "begin"}
{"obj":{"count":10392,"time_first":138126549...}}
{"obj":{"count":33,"time_first":19126549...}}
{"cond":"limited", "msg":"Result limit reached"}

In this case, the results are valid and SHOULD be used.

Note: The “Result limit reached” message does not imply that re-trying the query with a higher limit value will return more results, only that the limit was reached.

Failure example

{"cond": "begin"}
{"obj":{"count":33,"time_first":19126549...}}
...
{"cond": "failed", "msg": "Processing timeout; results may be incomplete"}

In this case, the results MAY be used, but it is recommended to retry the request.

Successful but empty examples

{"cond": "begin"}
{"cond": "succeeded"}
{"cond": "begin"}
{}
{}
{"cond": "succeeded"}

These are successful examples that are empty of data.

Changes

2020-08-18:

2020-07-31: