AEDAT 4.0
The latest release of the AEDAT format is 4.0, introduced in July 2019 with the Dynamic Vision (DV) software platform.
It uses Google Flatbuffers to serialize data in a convenient and efficient format, with an easy path to extend existing data structures and introduce new ones. Flatbuffers also allow quick and easy support for other languages, such as Python or Java, to be added, by auto-generating the appropriate support files for them.
All Flatbuffers are size-prefixed, meaning the first four bytes represent a 32 bit little-endian integer encoding the size of the following, actual Flatbuffer data.
All timestamps inside the data are 64 bit integers, representing Unix time in microseconds.
Header
The version header line reads as follows:
#!AER-DAT4.0\r\nA size-prefixed Flatbuffer follows the IOHeader (FB Schema). The IOHeader currently contains the following information:
Field name |
Field type |
Description |
|---|---|---|
compression |
enumeration |
Compression algorithm applied to all data streams in the file. Currently supported are: NONE, LZ4, LZ4_HIGH, ZSTD and ZSTD_HIGH. The HIGH compression modes provide smaller file sizes in exchange for higher CPU usage. Scenarios where the highest performance is key should employ LZ4; for a good balance between speed and file size ZST should be used. |
dataTablePosition |
int64 |
Offset in bytes of the FileDataTable, containing all the information required to quickly interpret and jump around inside the file. ‘-1’ means no table present. |
infoNode |
string |
Dump of the XML node describing all the data streams and their information (sizes, sources, …). Each node represents a stream, with its name being an integer that is used as StreamID in the rest of the file’s data. |
After the header, the actual data streams can be found.
Data
Data packets consist of a PacketHeader (FB Schema, fixed 8 bytes struct) which identifies the stream (StreamID integer) and the size of the following content (Size integer).
The content can be compressed or not. If it is compressed (compression != NONE), the whole data block should be fed to the appropriate decompressor for either LZ4 or ZSTD’s frame format (each packet is one compressed frame). The result of decompression (or directly the content if compression == NONE) is a size-prefixed Flatbuffer. You can use ‘flatbuffers::BufferHasIdentifier()’ or similar to get the four character type identifier and parse the content using the appropriate Flatbuffers functions in your language of choice.
FileDataTable
After the data section, there can be one more Flatbuffer, the FileDataTable (FB Schema), that contains information on all the data packets written into the file previously. If the ‘dataTablePosition’ field in the IOHeader has a value of ‘-1’ this table is not present in this file, else it starts at the given position (offset in bytes). No more data is present after that offset in the file.