As a communications protocol that has been around for decades, FTP has received heavy criticism for being low-security (for supporting only user name/password authentication, and for transmitting passwords in the clear) and difficult with firewalls (as the dynamic negotiation of ports requires the firewall to perform deep inspection). Contenders tried to take its place and amendments have been made (for example, adding SSL/TLS for encryption); higher layers have been added on top of it (managed file transfer solutions). In uncovering the truth about FTP we can rely on a few certainties...
FTP Is Not the Fanciest Protocol on the Block
Despite this, time and again, our customers find that FTP is near indispensable for daily operations. FTP is the behind-the-scenes worker that patiently uploads files and runs a command or two, and in that humble, unassuming manner ensures that files get from A to B. Yes, EDI solutions are big, especially in Europe; SOAP protocols work magic for web services; and “social” protocols are at the height of fashion. But when it comes to ease of use, and sheer worldwide use, the truth is that FTP is hard to beat.
Given the widespread use of FTP in IBM i shops, CCSS made FTP monitoring available as a new functionality in its message management and automation solution QMessage Monitor. FTP monitoring became available to CCSS customers in 2004 and has been popular ever since. FTP monitoring takes actions that are performed inside FTP sessions and turns them into messages. An example message text is:
FTP: User THOMASK listing directory /QSYS.LIB/QGPL.LIB
Monitored actions include FTP logon, change directory (CD), file upload (PUT), file download (GET), and execution of CL commands (QUOTE RCMD), amongst others. Messages are employed because they are the “common currency” of QMessage Monitor, allowing QMM to unleash the full arsenal of its message management functionality, especially rule-based message handling and escalations. Helping implement FTP monitoring for customers (and using it ourselves) has confirmed to us just how useful this functionality is. This use of FTP monitoring in different environments also opened our eyes to ways in which this functionality could be taken even further.
The Good
There are two main ways in which FTP is used in IBM i environments. One is the automatic execution of FTP scripts, which are typically scheduled to run every few minutes. The other is the system operator, logging on and manually moving files between systems in order to perform some housekeeping task.
Both of these are legitimate use scenarios for FTP. Its security shortcomings are these days usually compensated for by the mentioned combination with SSL/TLS, or simply by keeping all communications within the corporate network, behind a heavy-duty firewall.
Even those legitimate uses, however, may need to be logged for the purpose of auditing. If your organisation requires security audits, being able to provide logs of FTP activity is always preferable. You may be the safest admin in the world yet the auditor, whose job requires them to be skeptical, will still make demands for independent proof.
With QMessage Monitor, of course, proof is readily available through the Activity Log, the product’s log of all messages including FTP monitoring messages.
The Bad
The other reason to monitor FTP is its visibility to those scenarios you don’t want to occur:
- The outside hacker who has found a way to circumvent perimeter defences and logs onto FTP to download a few of your most confidential customer files.
- The fumbling supplier who accidentally wipes a whole directory of accounting data while trying to upload his own files.
- The malicious ex-employee who uses a forgotten user profile to log on via FTP and cause major mayhem on the system.
Here, QMessage Monitor can help in two ways:
- Real-time monitoring facilities make actions inside FTP sessions immediately visible. If you know in advance which objects, commands, or user profiles are most sensitive, you can highlight those actions in the message console and even send a warning via email or SMS.
- The Activity Log can be used to perform an initial forensic investigation.
The Beautiful
So, you have set up your FTP monitoring according to the documentation. The FTP messages:
The first thing you want to do is to check for automatic FTP processes and remove those messages from the Console. These are easy to spot: They typically occur every minute or every few minutes, and they execute the same commands. What’s more, commands are being issued at superhuman speed—faster than Commander Data in that one episode of Star Trek where… but I digress!
So, use automated responses to remove those messages from sight (while keeping a copy in Activity Log). Typical filtering criteria on these rules are the message ID, the user name, the system, and the value of message fields that reflect the names of objects or directories.
In doing this you will have effectively turned off the message fire hose and be in a position to see what else can be done.
FTP Monitoring +
FTP-related information need not only come from the FTP monitoring function. There are other message sources that also carry valuable information about FTP on your IBM i. By including them in your “field of view”, you can ensure a 360 degrees view of FTP.
The additional sources are:
- The Security Audit Journal
- The History Log
- The TCP message queue
Each of these sources will give you valuable information about FTP on your System i.
Setting up the first two is described in the product documentation, if they were not already set up for you. In essence, a conversion process takes input from either source and transforms it into, once again, messages on message queues—the MMAUDJRN message queue in one case, the MMQHST queue in the other.
The third message source is an auxiliary source of information; it is the QTCP message queue that resides in the QUSRSYS library.
Messages differ in the information they provide. FTP monitoring messages typically name the user profile, and an object path. The name of the FTP server job may also be available. This is useful if you try to match actions to a particular FTP session.
Below is a table that lists the FTP-related messages for all message sources combined, and the information available on them.
Note: The end of an FTP server job can be monitored through the MMQHST or QTCP message queues. If you are not already monitoring either, be aware that monitoring QTCP has the benefit of using less CPU than monitoring MMQHST.
This Is the End
Tracking the end of an FTP session is one thing that the FTP monitoring function itself cannot track. IBM does not make that information available in a way that the function could use to do so.
How you can work around this limitation is also a good example of how you can use other message sources to enhance FTP monitoring to get “FTP monitoring +”.
Two things can happen when an FTP session ends:
- The FTP server job performs the command QSYS/CLRLIB LIB(QTEMP).
- The FTP server job ends.
Which of these actions occur depends on the specific implementation of the FTP client. In most cases, it would be the former, but the latter action also occurs.
How Do You Track These Events?
To track execution of command QSYS/CLRLIB LIB(QTEMP), use the Security Audit Journal or MMAUDJRN. Set up the Audit Journal Filters to ensure that message UCD0002 is captured. Then set up an Automated Response to make sure the same message ID is displayed (log option “Y”). For filters on the Automated Response, use:
Specify actions for this Automated Response:
As always, if you are unsure about which message field maps to which piece of information, use the Console to display the Message Details for a sample message, and then Attributes to view the fields’ values.
The automated response above will turn this message
...into this much more useful message:
Using this technique, you can employ automated responses to highlight FTP-related messages and to transform their text.
"Are We There Yet?"
So we have now added substantially to your view of FTP activity. But is that all? To quote Mickey Rourke’s line from Sin City, “Is that the best you can do?” Well, that is not the full quote–children may be reading this article! (Who knows?)
So, the answer is no. We are not there yet. There is, for instance, the database monitoring facility in QMessage Monitor that lets you see when and how FTP jobs open up database files—and we have not even touched on how status and performance monitoring in QSystem Monitor can show you how much CPU is being consumed by FTP jobs, how much Disk I/O they are causing, how many page faults they are incurring…
And there's plenty more. See how much else—and what it can do for your system. Contact us.
