Table of Contents

Introduction

This is the technical reference guide for the Flexible Search extensions (known as the Flex API) to DNSDB APIv2. APIv2 is implemented in Farsight Security DNSDB 2.0 which provides our Standard Search capabilities. Flexible search adds ways to search DNSDB by regular expressions and globs (aka wildcarding).

Please see the Introducing DNSDB 2.0 https://www.farsightsecurity.com/blog/txt-record/DNSDB-2.0-20200804/ page for general information.

The Flex API extensions to the RESTful DNSDB APIv2 are documented in Flex API protocol.

Audience

This document is intended for those wanting a deeper understanding of the data in the Flex API and programmers who want to write applications that can interact with the Flex API extensions to the RESTful DNSDB APIv2 using JSON and HTTP.

Properties of Flexible Search thru its Flex API

  1. Sometimes you’ll run DNSDB Flexible Search and the results will be sufficient as-is.
  2. Other times, Standard Search can find the results you need, just as before, nothing more needed.
  3. On other occasions, you’ll run DNSDB Flexible Search and then pivot from those results to DNSDB Standard Search for more details.

Data representation and indexing

Trailing dot

DNS is a distributed database whereby all names are “below” the root. The root name is “.” (a single dot). DNS domain names all implicitly end in that trailing dot, even though most programs do not show it. The standard behavior for many DNS-smart libraries and programs is to add the trailing dot if it’s missing. Therefore, the Standard Search DNSDB automagically adds the trailing dot to DNS names. To illustrate this, here are some examples of programs automagically adding the trailing dot as needed, dig and dnsdbq:

$ dig +short api.dnsdb.info  CNAME
dnsdb.info.
$ dig +short api.dnsdb.info. CNAME
dnsdb.info.

$ dnsdbq -r  api.dnsdb.info -t CNAME | fgrep -v ';'
api.dnsdb.info. CNAME  dnsdb.info.

$ dnsdbq -r  api.dnsdb.info. -t CNAME | fgrep -v ';'
api.dnsdb.info. CNAME  dnsdb.info.

Similarly, RData values will usually either end in a trailing dot or a double quote. Here we show a TXT record from dig and then from dnsdbq. From the dnsdbq output, you can see that the double quotes are really part of the data (note we split the line for readability):

$ dig +short fsi.io TXT
"v=spf1 mx a a:support.farsightsecurity.com a:exch.fsi.io a:prod-mail-relay-1.iad1.fsi.io ~all"
$ dnsdbq -r fsi.io -t TXT -j -A -10000 | jq .
{
  "count": 340,
  "time_first": 1596815861,
  "time_last": 1600347647,
  "rrname": "fsi.io.",
  "rrtype": "TXT",
  "bailiwick": "fsi.io.",
  "rdata": [
    "\"v=spf1 mx a a:support.farsightsecurity.com a:exch.fsi.io a:prod-mail-relay-1.iad1.fsi.io ~all\""
  ]
}

The Flex API does not automagically adjust your queries to add the trailing dot to RRnames queries nor does the API the trailing period or double quote to RData queries. Therefore, your queries will need to match those characters. This will be illustrated below.

Filtering and Representation of RRnames and RData

To maintain a high signal-to-noise ratio and deliver an as practically-usable tool as possible, we curate what Flexible Search indexes. This means that we intentionally do not index some of the junk that gets injected into the Domain Name System:

RRTypes supported

RRtypes can be entered in either the standardized alphanumeric “symbolic” format (A, AAAA, CNAME, MX, NS, SOA, TXT, etc.) or in TYPE# format (where # is the decimal value of the RRType).

If an undefined RRType is specified (i.e. not in the DNS RFCs), then an error is returned with message “Error: RRTYPE has an unsupported value”.

The RRtype ANY is modified somewhat from its usual meaning. A search for RRtype ANY will match any Flexible Search-indexed RRtype. ANY is the default if RRTYPE is not specified.

DNSSEC records are not searchable nor returned with Flexible Search. Therefore, the following DNSSEC-related RRtypes are not permitted:

If one of those DNSSEC RRTypes is specified, then an error is returned with message “Error: RRTYPE has an unsupported DNSSEC value”.

Only the following RRTypees (plus their TYPE# equivalents) are indexed for RData queries:

If another RRType is provided for an RData query, then an error is returned with message “Error: rdata searches are not allowed for the specified RRType”.

In particular, A and AAAA records do not have their RData indexed. This means that if you’re searching for IPs, IP ranges, or CIDRs you should probably be using Standard DNSDB (any IPs that do get indexed in Flexible Search just get treated as text).

SOA records

An example full SOA record’s rdata value is "fsi.io. hostmaster.fsi.io. 2014100635 7200 3600 604800 3600". In the Flex API, SOA record values will be truncated to just mname, space, rname. The other portions of SOA records (serial number, zone timers, and minimum/error TTL) are not indexed nor available from the Flex API, but you can pivot to Standard Search for full values.

Due to the truncation of SOA records, they may not longer appear unique, and using time-fencing to match SOA records may not work as expected. For example, if you use time-fencing to select only “old” SOA records, you might find they don’t exist in the Flex API but do exist in the Standard Search API. The Flex API will have that SOA record, but across a much longer period of time.

Here’s a simplified example: dnsdbq -r fsi.io -t SOA -l 0 will return tens of thousands of rows, with an earliest time_first of 2014-10-09 20:24:59 and a latest time_last around the past hour. dnsdbflex will return just one record encompassing that entire period.

Here I show, with added line wrapping for readability:

$ dnsdbq -r fsi.io -t SOA -l 1 -j
{"count":77,"time_first":1412886299,"time_last":1412900460,"rrname":"fsi.io.",
"rrtype":"SOA","bailiwick":"fsi.io.",
"rdata":["fsi.io. hostmaster.fsi.io. 2014100635 7200 3600 604800 3600"]}
$ dnsdbq -r fsi.io -t SOA -l 0 -j  | tail -1
{"count":1,"time_first":1599750021,"time_last":1599750025,"rrname":"fsi.io.",
"rrtype":"SOA","bailiwick":"fsi.io.",
"rdata":["fsi.io. hostmaster.fsi.io. 2020090902 7200 3600 604800 3600"]}
$ dnsdbflex --glob 'fsi.io.' -j -t SOA -s rrnames
{"rrname":"fsi.io.","rrtype":"SOA"}
$ dnsdbflex --glob '*fsi.io*' -j -t SOA -s rdata
{"rdata":"fsi.io. hostmaster.fsi.io.","rrtype":"SOA",
"raw_rdata":"0366736902696F000A686F73746D61737465720366736902696F00"}
$ dnsdbq -N 0366736902696F000A686F73746D61737465720366736902696F00 -j -l 1
{"count":6,"time_first":1413228451,"time_last":1413228451,
"rrname":"local-data.fsi.io.","rrtype":"SOA",
"rdata":["fsi.io. hostmaster.fsi.io. 2014050101 7200 3600 604800 3600"]}
$ dnsdbflex --glob '*fsi.io*' -t SOA -s rdata -F
rdata/raw/0366736902696F000A686F73746D61737465720366736902696F00/SOA
# rdata/name/fsi.io. hostmaster.fsi.io./SOA
$ dnsdbflex --glob '*fsi.io*' -t SOA -s rdata -F | dnsdbq -f -j -l 1
{"count":6,"time_first":1413228451,"time_last":1413228451,
"rrname":"local-data.fsi.io.","rrtype":"SOA",
"rdata":["fsi.io. hostmaster.fsi.io. 2014050101 7200 3600 604800 3600"]}
--
$ dnsdbq -V summarize -N 0366736902696F000A686F73746D61737465720366736902696F00 -j  -l 0
{"count":36640686,"num_results":28828,"time_first":1412885673,"time_last":1599765048}

This is a regular expression search. The particulars are known as “Farsight Compatible Regular Expressions” (FCRE). For more information:

Recall that dot (aka period) is a “match any one character” regular expression operator. To match exactly a dot, use \., as you must backslash escape the dot.

The regular expressions are not implicitly anchored front and back. In domain names with standard DNSDB, a search for “fsi.io” contains two DNS labels “fsi” and “io”. With the Flex API, “fsi.io” would actually match “fsiaio”, “fsi.io”, “fsi9io”, but because it’s not anchored, it also matches “www.fsi.io” and “www.fsi-io.yourbank.com”. Use ^ to anchor at the front and $ at the end.

The Flex API does not automagically insert the trailing period. If you right-anchor a regular expression that is searching for RRnames, you need to add a character that will match the trailing period. A backslash escaped dot or not-backslash escaped dot will do, though any regular expression operator that can match a dot is valid. For RData like TXT records, the trailing character is a double quote, therefore using a not-backslash escaped dot is probably safest.

To get the same behavior as Standard Search DNSDB for “fsi.io”, you need to anchor the regular expression, backslash escape the first dot, and add a trailing, thus use ^fsi\.io\.$.

This is a glob (aka wildcarding search). For more information:

Globs, unlike regular expressions, are implicitly front and back anchored. To search for any name containing “coke”, glob *coke* works.

To get the same behavior as Standard Search DNSDB in searching for “fsi.io”, you need to add a trailing dot, fsi.io. works.

Our dnsdbflex tool will forbid an RRnames glob query without something that matches the trailing dot:

$ dnsdbflex --glob fsi.io -s rrnames
Error: a glob search argument for rrnames should end either in a period or certain glob
special characters (*, ?, or ]).
You may not get results from your search.

You can force it to issue the query, but you won’t get results:

$ dnsdbflex --glob fsi.io --force
Warning: a glob search argument for rrnames should end either in a period or certain glob
special characters (*, ?, or ]).
You may not get results from your search.
Query status: NOERROR (no results found for query.)

There are some rare RRTypes whose RData values do not end in dot or double quote, so --force can be useful for those.

For Standard Search RRset searches, the following data are returned in the output:

For Flexible Search RRnames searches, the equivalent of Standard Search RRset searches, the following data are returned in terse mode (which is the only currently supported mode):

For Standard Search RData searches, the following data are returned:

For Flexible Search RData searches, the following data are returned in terse mode (which is the only currently supported mode):

A description of each of the data fields returned by Flexible and Standard Search:

Key Description
rrname The owner name of the RRset in DNS presentation format.
For Flexible Search, see the Representation of rrnames and rdata section for details.
rrtype The resource record type of the RRset, either using the standard DNS type mnemonic, or an RFC 3597 generic type, i.e. the string TYPE immediately followed by the decimal RRtype number.
rdata An array of one or more RData values.
For Standard Search, the RData values are converted to the standard presentation format based on the rrtype value. If the encoder lacks a type-specific presentation format for the RRset’s rrtype, then the RFC 3597 generic RData encoding will be used.
For Flexible Search, see the Representation of rrnames and rdata section for details.
raw_rdata The record data value as pairs of hex digits specifying a raw octet string. This value is used for pivoting from Flexible Search into Standard Search to get more details on RData.

A description of each of the data fields only returned in Standard Search; you might need to pivot from Flexible Search to Standard Search to get these:

Key Description
bailiwick Closest enclosing zone delegated to a nameserver which served the RRset.
count The number of times the RRset was observed via passive DNS replication.
time_first, time_last UNIX epoch timestamps with second granularity indicating the first and last times the RRset was observed via passive DNS replication. Will not be present if the RRset was only observed via zone file import.
zone_time_first, zone_time_last UNIX epoch timestamps with second granularity indicating the first and last times the RRset was observed via zone file import. Will not be present if the RRset was only observed via passive DNS replication.

Comparison of dnsdbq and dnsdbflex JSON output

Here is what the JSON data output from dnsdbq and dnsdbflex looks like for RRset (in dnsdbq), RRnames (in dnsdbflex), and RData queries. Line breaks were added for readability, but the actual results were on one line.

RRset (RRnames) comparison
$ dnsdbq -r 'fsi.io' -j -l1
{"count":10392,"time_first":1381265499,"time_last":1428418529,
"rrname":"fsi.io.","rrtype":"A","bailiwick":"fsi.io.","rdata":["66.160.140.76"]}

(See above for why we need to add the trailing period on fsi.io.)

$ dnsdbflex --glob 'fsi.io.' -j -l1 -s rrnames
{"rrname":"fsi.io.","rrtype":"A"}
RData comparison
$ dnsdbq -n 'fsi.io' -j -l1
{"count":6,"time_first":1413228451,"time_last":1413228451,
"rrname":"local-data.fsi.io.","rrtype":"SOA",
"rdata":["fsi.io. hostmaster.fsi.io. 2014050101 7200 3600 604800 3600"]}

(See above for why we need to make the query wildcarded, as fsi.io. does not appear by itself in any RData values.)

$ dnsdbflex --g 'fsi.io*' -j -l1 -s rdata
{"rdata":"fsi.io. hostmaster.fsi.io.","rrtype":"SOA",
"raw_rdata":"0366736902696F000A686F73746D61737465720366736902696F00"}

Name encoding

All IDN DNS names must be encoded in Punycode (https://tools.ietf.org/rfc/rfc5891.txt). In particular, using non-basic ASCII characters may result in a 500 Internal Server error. You cannot search for a IDN name as a wildcard in its native character set; you can only search in the Punycode representation.

Quoting

If you’re using our dnsdbflex command line tools and working at the shell prompt to access DNSDB 2.0 Flexible Search, single quote marks around your query string will protect it from unwanted shell interaction. Otherwise, you’ll need to backslash escape any special characters including the backslash.

When working with dnsdbflex, DNSDB Scout, raw URLs, etc. in regular expressions you will often have to backslash escape the dot that separates DNS name labels.

Some examples of both of those:

Error messages

Flex Search will report on various errors in the regex or glob search parameters. These errors are detected after the query has started processing, i.e. after an HTTP “200 OK” was returned. DNSDB Scout and dnsdbflex will report these errors to you. If necessary, the error message will include the character index (starting at 0) where the error was noticed (like a compiler will do).

While this is not the API document, here is one detailed example for reference. The regular expression has a tail anchor followed by a head anchor.

$ curl -k -H "Accept: application/x-ndjson" -H "X-API-Key: $APIKEY" \
'https://api-dev2.dnsdb.info/dnsdb/v2/regex/rrnames/$^?limit=2'
...
< HTTP/1.1 200 OK
...
{"cond":"begin"}
{"cond":"failed","msg":"Regex syntax error: Invalid characters after '$' at 1"}

The following are the general error messages, which will be followed by more specific details, as just illustrated.

Message
Bad exclude expression:
Gateway timeout:
Glob syntax error:
Glob syntax unsupported:
Regex syntax error:
Regex syntax unsupported:
Search string unsupported:

Examples of some of those errors as reported by dnsdbflex, whereby it prints Query failed: and then specific details:

$ dnsdbflex --regex '[[:foo:]].com' -d
Query failed: Regex syntax error: Invalid character class name at 0

$ dnsdbflex --regex 'www|{0,100}'
Query failed: Regex syntax error: Invalid '{' at 4

$ dnsdbflex --regex 'missing['
Query failed: Regex syntax error: Unmatched [ at 7

$ dnsdbflex --glob '[missing.'
Query failed: Glob syntax error: Unmatched [ at 0

$ dnsdbflex --regex '*.com'
Query failed: Regex syntax error: Invalid '*' at 0

# In the previous line, the user probably meant to search for anything ending in .com,
# which as a regular expression, should be:
$ dnsdbflex --regex '.*\.com\.$' -l 2

Unlikely error messages

The following error messages will not be returned in practice, as either the error is detected at a higher-level and a “400 Bad Request” status is returned, or it indicates an internal bug. Please email support@farsightsecurity.com if you see them.

Unlikely Message
Bad return what parameter:
Bad rrtype:
Bad search method:
Bad search what parameter:
Illegal limit:
Internal error

Example client

Changes

2020-10-01:

References

What’s A Regular Expression?

What is Globbing?

Printable cheat-sheet for regex and glob

The WDNS Library, a low-level C library for dealing with wire-format DNS packets, is at:

For more information about IDNs, Homographs, and Punycode please see Mike Schiffman’s Farsight Security Global Internationalized Domain Name Homograph Report:

For more information about DNSDB Time Fencing please see Joe St Sauver’s Farsight’s DNSDB Time Fencing: A Post-Attack “Time Machine”:

Latest draft of IETF Passive DNS - Common Output Format (COF) format