Discussion of datasets, datapoints and disk space

The Future Disk Usage chart on the Overview page of the InterMapper Database web interface displays an estimate of the amount of disk space required with the current retention policy.

You can verify this manually by taking the number of datasets and multiplying by the number of polls that will happen over a specific time period. For example, to determine the amount of disk space in a 2-year period, at a 30-second poll interval:

7000 (datasets) x 2880 (polls per day) * 730 (days in 2 years) = 14.7 billion points

Each datapoint occupies roughly 35-100 bytes of space.

It's not recommended to keep raw data for longer than 2 weeks, unless the queries being run against the data specifically require it.

Keep in mind that even if the retention policy is adjusted to 2 weeks, the actual disk space will not be freed until full maintenance has a chance to run, which happens automatically at the end of each week, or manually from the Maintenance Tasks page by clicking the Database Maintenance button.

When making a dramatic change that results in the database size changing by hundreds of gigabytes, it may take hours or possibly days for full maintenance to complete.

A. In certain cases it's possible for the database to get out of sync with an InterMapper server, i.e. report that devices still exist when they have been deleted. A full sync fixes that problem by getting a full list of all devices from the server, and comparing it to the list in the database. It's run automatically every morning, and can also be run from maintenance tasks.

Daily maintenance makes it so the database can re-use space from deleted entries, but it doesn't actually give that space back to the system. So your free disk space never changes after applying policies or running daily maintenance. Only a full maintenance actually frees up the space. Full maintenance runs automatically on Sunday mornings.

Q: Can you change the day when the Full Maintenance happens? What exactly does a Full Maintenance do?

 As far as modifying the time when it activates we can’t do that since it’s hardcoded to run on Sunday nights in the software.  However, you’re able to run it on demand from the Reclaim Disk Space whenever you like.

Two tasks are run automatically to clear out data that is beyond its retention policy expiration, and to reclaim unused disk space.

• Daily task - runs at 1AM local time each day. It applies retention policies, then uses the PostgreSQL VACUUM command to mark free space for re-use by the database. This is a relatively low-impact process, and does not pause database operations. It does not release disk space for reuse by the operating system.
• Weekly task - runs at 1AM local time each Sunday. It uses the PostgreSQL CLUSTER command, and pauses database operations while it runs. This task frees up unused database space, making it available to the operating system.

After you have run one of these tasks, the Disk Usage table shows the freed disk space as available.