next up previous contents
Next: 8. User Interface Up: CFHT Data Acquisition Previous: 6. Session Tools   Contents


7. Infrastructure

7.1 Status

See Tom for a general overview of the Status Server and related infrastructure.

7.2 Feedback and Logging

Programs need to send text feedback messages to the user. Other messages also typically need to be recorded to a log, for later use in debugging, etc. For the developer, it is important to consider the complete set of sources of these messages:

  1. Operating system kernel
  2. Operating system processes
  3. Application software not associated with any session
  4. Application software associated with a specific session

Figure 5 illustrates the flow of messages, including the calls programs must make to log, and ways to view the messages.

Figure 5: Sources of Logging Messages
\epsfig {file=logging.eps}\end{center}\end{figure}

Ideally, items 1 and 2 all use the syslog() interface (see ``man 3 syslog''). A standard unix process called the syslogd captures the messages and routes them to the console, a file, or another machine. Programs which generate messages on stdout can be piped into a standard unix tool called logger, which redirects each line to the syslog interface. The file /etc/syslog.conf on the local host determines the behavior of the syslogd. At CFHT, we direct everything to the host called ``lawelawe''. (Some hosts may still be logging to its predecessor, ``central-services.'') On lawelawe, and on any machines which do their own logging, all messages are appended to the file /var/log/messages, while a subset including only warnings and errors is appended to /var/log/warnings. Once syslogd has received a message and appended it to one or both of these files, the original ``severity'' with which the message was logged is lost. It then becomes difficult to later pick out the important messages from /var/log/messages. This is the reason we also log the subset of more important messages to /var/log/warnings.

The above being the ideal for system software, the ideal for application software items 3 and 4 is similar. Dissociated application software uses the cfht_logv() interface - similar to unix syslog() - which relays messages to a process called ``roll'' and puts them in a circular buffer on the local machine that can be viewed with a program called ``unroll.'' To make the message format shown by ``unroll'' similar to syslog, a ``-C'' option has recently been added to this utility (C for ``Cooked''.) Messages are displayed in reverse chronological order. To see them in Forward order, use ``-F'', and ``-h'' takes an argument for the number of hours to go back. So, for example, to see the last 5 hours:

      unroll -CFh 5 | less

Other options controlling the behavior of unroll can be found in the man page.

Software developers outside of CFHT are typically writing item 4 programs, i.e., the session and instrument control software. The most basic logging option for this class of software is to use a simple printf() interface (or, in any language other than C, the appropriate calls to write lines to the program's standard-output.) The Director shell manages each of these instrument control programs (as Agents) and logs whatever they print on stdout and stderr to the cfht_log facility. Thus, output from instrument control tasks is logged together with output from dissociated tasks on the session host. It is all accessible in one place, using unroll. Even when instrument control is distributed among computers other than the session host, the director model allows all messages to be logged centrally, on the session host.

Dissociated programs which do not happen to run on the instrument's session host can be one exception. An example of such a program is the TCS server. Often, the session host and the TCS server host are not the same. To piece together messages from telescope control, one should use the unroll program on the TCS server host as well.

7.2.1 Message Types

In order to have program messages classified appropriately, the use of printf's in your program should follow some guidelines:
The alert message type is reserved for critical conditions that may require immediate attention from an engineer. This includes things like thermal overloads which could cause damage. Since these conditions could occur with no operator nearby, alert messages trigger a special ``alert'' program to be launched. The alert program is capable of calling pagers or sending e-mail notification (based on the actual message of the alert.)

Alert messages appear in red, just like error messages, but trigger the alert script which displays a timeout popup (to allow warning e-mails to be prevented.)

Displayed in red (and terminal beeps/de-iconifies). Top level command request has failed. Read warnings above for details.

Displayed in yellow. Something is not 100% normal and may cause bad results down the line. (But request itself hasn't failed yet.)

Displayed in green. Usually a confirmation that a command succeeded.

7.2.2 Additional Message Types for Agents

progress: ... (__%)
This type of message should be sent by any agent just before it begins a blocking operation (e.g., waiting for hardware to reach a given position.) If it is possible to determine how close to completion the operation is, then a percentage can be added in parentheses, just before the newline in this message. In that case, the program may wish to print a progress message at about once per second to keep the user informed of the progress.

Reserved for use by DetCom in most sessions. The most recent statusbar: line received from any agent is displayed just above the user's prompt in reverse-video or against a blue background.

The _ above is actually a digit, 1..9. Depending on the digit, this message type replaces one of the 9 optional status bars visible below the users prompt against a grey background. Currently, there is no clean mechanism to manage which agent processes should update which lines in the ``info panel.''

7.2.3 Additional Message Types for debugging

Not usually displayed except in ``verbose on'' mode. Mostly useful in tracking down problems with the instrument or elsewhere.

Not usually generated except in ``debug on'' mode. Useful to track problems with the software itself.

7.3 Director

See Sidik for a general overview of Director.

next up previous contents
Next: 8. User Interface Up: CFHT Data Acquisition Previous: 6. Session Tools   Contents
Sidik Isani