Q. How can I manage the amount of disk space used by IMDatabase?
A. InterMapper Database provides a detailed history of your network's activity, including data that the InterMapper Server would discard immediately (as with events) or after a day (as with auto-record datasets). This volume of data can be enormous, so it's important to install IMDatabase on a machine with plenty of free disk space.
IMDatabase 5.1 introduces a new 'Disk Usage Stats' page, with charts showing how many rows are in each of the database's major tables. It also estimates, based on the data it has collected so far, how much disk space will be needed in the future.
To manage disk space, and since it isn't necessary to keep data around forever, IMDatabase allows you to define data retention policies. This is done from the 'Retention Policies' page in the IMDatabase web interface. Retention policies are rules that determine how long data is kept in the database, and can be applied to individual devices or interfaces on your maps. The database ships with four built-in retention policies: None, Forever, 24 Hours and IM46Charts.
Devices or interfaces given the 'None' policy will not export data to the database. Any data they have already exported will be deleted the next time retention policies are applied.
The 'Forever' policy causes any data exported to the database to remain there forever. The only way to remove it is to change the retention policy, or use the 'Maintenance Tasks' page introduced in version 5.1 to delete the data manually.
The '24 Hours' policy keeps data until it is one day old; it will then be deleted the next time retention policies are applied. The 24 Hours policy is the default server-wide policy. When you create a new device, unless you have changed the server-wide policy in the Server Settings > Database Server settings, or the map-wide policy from Map Settings, that device will be using the '24 Hours' policy.
Finally, if you have used InterMapper 4.6, any devices or interfaces with charts will be assigned a retention policy called 'IM46Charts'. It starts out with the same rules as the 'Forever' policy, so users upgrading from 4.6 will find that data from existing charts is kept indefinitely. Note that this policy applies only to individual devices that had charts in 4.6. The retention policy set in the Server Settings > Database Server settings is the server-wide default for new devices. Out of the four built-in retention policies, IM46Charts is the only one whose rules you can edit. By editing the policy's rules from the 'Retention Policies' page, you can easily change the behavior from keeping data forever, to keeping it for a week or a day, without having to go through all your devices to give them new policies.
Retention policies are applied every day at 01:00 UTC time. IMDatabase will look at the datasets for each device and interface, and delete any data that is older than the time specified in the policy for that device or interface. Note that, because of the way the PostgreSQL database works, deleting the data doesn't immediately free up disk space. To save time, the database simply marks those rows as deleted; it may take a few hours or days before it actually reclaims the space. This applies mainly to IMDatabase 5.0; starting with version 5.1, disk space should be freed up within minutes after retention policies have been applied.
As a rule, install IMDatabase on a server with at least 50GB of free space. In the first few days of operation, you should monitor disk usage, as well as the estimates on the 'Disk Usage Stats' page. It should quickly become apparent whether you need to move it to a larger disk, or adjust the retention policies to delete old data sooner.
