Intermapper Chart File Format

 

This article documents the structure of Intermapper chart files.

Pre-6.2 Chart Data File Format

Chart data files contain repeated pairs of eight-byte data:

• 32-bit UTC time stamp, in seconds since the epoch (1 Jan 1970)
• 32-bit signed data value (The value 0x80000000 is reserved to represent a discontinuity, meaning that there should be a break between the data values on either side.)

Intermapper appends new data readings to the end of the chart data file in this format. Both these values are in the proper endian order for the current processor.

6.2 format

With the release of Intermapper 6.2, chart data storage has been restructured.

Here are the major differences:

• Data is now accessible as segments of time rather than file indexes.
• Time and data values are represented in 64-bit signed or unsigned format.

Why change the format?

The original format had one major advantage. Each data point was a fixed size, with times and data in 32-bit format. This made access to data relatively easy - the number of data points could be calculated by the file size.

It also had some disadvantages:

• Dynamic purging of data was not an option.
• Data values could not be greater than 32bits. This presents issues charting high-speed links.
• The fixed size becomes very inefficient with larger charts.

This new structure permits dynamic purging of chart data.

A new limited retention policy has been added. You can now specify how long chart data will be kept.

No conversion of existing data is required. Old chart data remains untouched, and new data uses the new structures.

As a result of the new data structure, the space required to store chart data has been reduced significantly. Typical storage space reduction is around 40%.

Basic file structure

A new chart data file contains:

• File Header
• Data Segments
• Segment Index

File header

• Signature/version - 8-bytes - Provides a mechanism identify the chart version and to handle version-based structural changes.
• File start time - Absolute start time for the file
• File end time - Absolute end time - Absolute start time
• Data Segment Count- (as defined below) The maximum number of segments is 178,956,970.
• Minimum data value
• Maximum data value
• Segment start offset - specifies the start of the segment index (see below)

Segment structure

• Segment header
• Segment data
  • Array of scaled points: points that have strict time variation within [-127, 127]
  • Array of odd points: points that are plainly laid out with [relative-time, value]. The relative-time is calculated by actual time minus the segment absolute start time. The value has a byte size of 1 to 8.

The maximum number of points a segment contains is 251.

Segment header

• Discontinuity (indicates whether the segment started with a discontinuity point)
• Polling Interval
• Checksum
• Scaled Point Array Data Value Size (number of bytes used to store the data value. Can be between 1 and 8)
• Scaled Points Count
• Segment absolute start time
• Odd Point Array Data Value Size (number of bytes used to store the data value. Can be between 1 and 8)
• Odd Point Relative Time Size (number of bytes used to store the time)
• Odd Point Count

Segment body

An array of data entries containing the following:

• Array of scaled points
• Array of odd points (if any)

Segment data

To minimize the space required for a data point, Intermapper charts use a default format called a "scaled point".

Scaled points consist of:

• A signed one-byte Delta Time (-127 to 127) related to the expected time of the data (as calculated using the Scaled Point Count and the polling interval).
• A signed or unsigned Data Value, with a size specified in Scaled Point Array Data Value Size.

Odd points consist of:

• Time relative to the start of the segment
• A signed or unsigned Data Value, with a size specified in Odd Point Array Data Value Size.

For scaled points, the delta time represents the variance from the polling interval. When the data is received the time is subtracted from the expected time to calculate the variance.

Odd points are used only when the variance from the expected time exceeds +/-127.

The storage size of the data value for both scaled points and odd points depends on the type of data being stored. For example, a percentage requires only 1 byte. Storage size can be up to 8 bytes per data point.

Segment Index

The segment index is an array of fixed-size index entries, and it is always in memory when a file is loaded. It provides random access to specific segments without reading from the beginning of the file.

Each entry contains the following information:

• Segment start time (unsigned 32-bit) relative to file start time
• Segment end time (unsigned 32-bit) relative to file start time
• Segment Data Point Count (unsigned 16 -bit)
• Discontinuity count (unsigned 16-bit)
• Segment total data size (segment header + actual arrays of data) (unsigned 32-bit)
• Segment maximum data value (64-bit)

Chart Data Mechanics

When is new file created?

• When a new chart is added or a new dataset is added to a chart. (Each file contains a dataset.)
• When the number of segments reaches 4294967295
• When the difference of absolute ending time and absolute start time is exceeding 4294967295 seconds (which is about 135 years)

When is a new segment created?

A segment is created when:

• When user changes the polling interval
• When the number of data points in the segment reaches 251.

Finding the Chart Data File which Corresponds to a Specific Dataset

The Chart Data folder contains a number of subfolders whose names are “graph IDs”. A graph ID is a unique ID identifying a map, and it takes the form gXXXXXXXX, where XXXXXXXX is a hexadecimal number. Within the subfolder named for a specific map are the individual chart data files. Chart data files are named RYY-CCCC or LYY- CCCC, where YY is a hexadecimal number representing the device’s (R) or link’s (L) id within the map, and CCCC is the four-character code for the type of dataset in question. For example, L2-Util represents the “utilization” statistic for the second link on the map. (N.B. On little-endian machines, such as most Windows and Linux machines, this file will be named L2-litU.)

There are several steps for determining the graph ID for a given map is, and within that map, determining the DeviceID or LinkID, and then the four-character code for the particular dataset.

The Intermapper GUI’s “export” function exports information about devices and/or links. The “Id” field contains both the map’s GraphID (gxxxxxxxx) and the DeviceID (Rxx).

To export devices:

1. In Intermapper or Intermapper Remote Access, open the map you’re interested in.
2. Choose File->Export->Data File…
3. In the export dialog that appears, for “Table”, choose “Devices”.
4. For “Fields”, click on Id, Name, Address, Probe Type and any other information you feel may help you identify individual routers.
5. Click “Add” to move them to the “Field Export Order” box.
6. Grab the fields in the “Field Export Order” box and move them up or down to change the order.
7. Once they are in the right order, click “Export”.
8. Enter a useful filename and click “Save”.

If you look at the resulting file in Excel or a similar tool, you will see name-id pairs such as:

RT.dartware.com. g4c855b97-r74

This means that the device named “RT.dartware.com” is on the map with the graph ID “g4c855b97” and has an ID of “r74” within that map. Thus, you will find datasets pertaining to this device in the Chart Data folder named “g4c855b97”, and they will start with “R74”.

To export links:

1. In Intermapper or Intermapper Remote Access, open the map you’re interested in.
2. Choose File->Export->Data File…
3. In the export dialog that appears, for “Table”, choose “Interfaces”.
4. For “Fields”, click on DeviceId, InterfaceID, Index, Description, and any other information you feel may help you identify individual links.
5. Click “Add” to move them to the “Field Export Order” box.
6. Grab the fields in the “Field Export Order” box and move them up or down to change the order.
7. Once they are in the right order, click “Export”.
8. Enter a useful filename and click “Save”.

If you look at the resulting file in Excel or a similar tool, you will see something like:

g4c855b97-r106 g4c855b97-L49 10 Intel(R) PRO/1000 MT Network Connection�

This means that the link named “Intel(R) PRO/1000 MT Network Connection”, at index 10 on the device with the ID of r106 on the map represented by the graph ID “g4c855b97” has an ID of “49” within that map. Thus, you will find datasets pertaining to this link in the Chart Data folder named “g4c855b97”, and they will start with “L49”.

Note, too, that even if you’re interested only in link datasets, it will be helpful to have the device export, too, in order to match device IDs corresponding to the device were the link is with device names.

Identifying a dataset

The final step is to find the four-char code for a given dataset type. The table below lists the possible four-char codes. Once you’ve determined the code from the list, you can put it together with the information above to complete the filename.