KLOGD(8)
NAME
klogd - kernel log daemon.
SYNOPSIS
klogd [ -c n ] [ -d ] [ -f fname ] [ -iI ] [ -n ] [ -o ] [
-p ] [ -s ] [ -k fname ] [ -v ]
DESCRIPTION
klogd is a system daemon which intercepts and logs Linux
kernel messages.
OPTIONS
-c n Sets the default log level of console messages to
n.
-d Enable debugging mode. This will generate LOTS of
output to stderr.
-f file
Log messages to the specified filename rather than
to the syslog facility.
-i -I Signal the currently executing klogd daemon. Both
of these switches control the loading/reloading of
symbol information. The -i switch signals the dae-
mon to reload the kernel module symbols. The -I
switch signals for a reload of both the static ker-
nel symbols and the kernel module symbols.
-n Avoid auto-backgrounding. This is needed especially
if the klogd is started and controlled by init(8).
-o Execute in 'one-shot' mode. This causes klogd to
read and log all the messages that are found in the
kernel message buffers. After a single read and
log cycle the daemon exits.
-p Enable paranoia. This option controls when klogd
loads kernel module symbol information. Setting
this switch causes klogd to load the kernel module
symbol information whenever an Oops string is
detected in the kernel message stream.
-s Force klogd to use the system call interface to the
kernel message buffers.
-k file
Use the specified file as the source of kernel sym-
bol information.
-v Print version and exit.
OVERVIEW
The functionality of klogd has been typically incorporated
into other versions of syslogd but this seems to be a poor
place for it. In the modern Linux kernel a number of ker-
nel messaging issues such as sourcing, prioritization and
resolution of kernel addresses must be addressed. Incor-
porating kernel logging into a separate process offers a
cleaner separation of services.
In Linux there are two potential sources of kernel log
information: the /proc file system and the syscall
(sys_syslog) interface, although ultimately they are one
and the same. Klogd is designed to choose whichever
source of information is the most appropriate. It does
this by first checking for the presence of a mounted /proc
file system. If this is found the /proc/kmsg file is used
as the source of kernel log information. If the proc file
system is not mounted klogd uses a system call to obtain
kernel messages. The command line switch (-s) can be used
to force klogd to use the system call interface as its
messaging source.
If kernel messages are directed through the syslogd daemon
the klogd daemon, as of version 1.1, has the ability to
properly prioritize kernel messages. Prioritization of the
kernel messages was added to it at approximately version
0.99pl13 of the kernel. The raw kernel messages are of the
form:
<[0-7]>Something said by the kernel.
The priority of the kernel message is encoded as a single
numeric digit enclosed inside the <> pair. The defini-
tions of these values is given in the kernel include file
kernel.h. When a message is received from the kernel the
klogd daemon reads this priority level and assigns the
appropriate priority level to the syslog message. If file
output (-f) is used the prioritization sequence is left
pre-pended to the kernel message.
The klogd daemon also allows the ability to alter the pre-
sentation of kernel messages to the system console. Con-
sequent with the prioritization of kernel messages was the
inclusion of default messaging levels for the kernel. In
a stock kernel the the default console log level is set to
7. Any messages with a priority level numerically lower
than 7 (higher priority) appear on the console.
Messages of priority level 7 are considered to be 'debug'
messages and will thus not appear on the console. Many
administrators, particularly in a multi-user environment,
prefer that all kernel messages be handled by klogd and
either directed to a file or to the syslogd daemon. This
prevents 'nuisance' messages such as line printer out of
paper or disk change detected from cluttering the console.
By default the klogd daemon executes a system call to
inhibit all kernel messages (except for panics) from being
displayed on the console. The -c switch can be used to
alter this behavior. The argument given to the -c switch
specifies the priority level of messages which will be
directed to the console. Note that messages of a priority
value LOWER than the indicated number will be directed to
the console.
For example, to have the kernel display all mes-
sages with a priority level of 3 (KERN_ERR) or more
severe the following command would be executed:
klogd -c 4
The definitions of the numeric values for kernel messages
are given in the file kernel.h which can be found in the
/usr/include/linux directory if the kernel sources are
installed. These values parallel the syslog priority val-
ues which are defined in the file syslog.h found in the
/usr/include/sys sub-directory.
The klogd daemon can also be used in a 'one-shot' mode for
reading the kernel message buffers. One shot mode is
selected by specifying the -o switch on the command line.
Output will be directed to either the syslogd daemon or to
an alternate file specified by the -f switch.
For example, to read all the kernel messages after
a system boot and record them in a file called
krnl.msg the following command would be given.
klogd -o -f ./krnl.msg
KERNEL ADDRESS RESOLUTION
If the kernel detects an internal error condition a gen-
eral protection fault will be triggered. As part of the
GPF handling procedure the kernel prints out a status
report indicating the state of the processor at the time
of the fault. Included in this display are the contents
of the microprocessor's registers, the contents of the
kernel stack and a tracing of what functions were being
executed at the time of the fault.
This information is EXTREMELY IMPORTANT in determining
what caused the internal error condition. The difficulty
comes when a kernel developer attempts to analyze this
information. The raw numeric information present in the
protection fault printout is of very little use to the
developers. This is due to the fact that kernels are not
identical and the addresses of variable locations or
functions will not be the same in all kernels. In order
to correctly diagnose the cause of failure a kernel devel-
oper needs to know what specific kernel functions or vari-
able locations were involved in the error.
As part of the kernel compilation process a listing is
created which specified the address locations of important
variables and function in the kernel being compiled. This
listing is saved in a file called System.map in the top of
the kernel directory source tree. Using this listing a
kernel developer can determine exactly what the kernel was
doing when the error condition occurred.
The process of resolving the numeric addresses from the
protection fault printout can be done manually or by using
the ksymoops program which is included in the kernel
sources.
As a convenience klogd will attempt to resolve kernel
numeric addresses to their symbolic forms if a kernel sym-
bol table is available at execution time. A symbol table
may be specified by using the -k switch on the command
line. If a symbol file is not explicitly specified the
following filenames will be tried:
/boot/System.map
/System.map
/usr/src/linux/System.map
Version information is supplied in the system maps as of
kernel 1.3.43. This version information is used to direct
an intelligent search of the list of symbol tables. This
feature is useful since it provides support for both pro-
duction and experimental kernels.
For example a production kernel may have its map file
stored in /boot/System.map. If an experimental or test
kernel is compiled with the sources in the 'standard'
location of /usr/src/linux the system map will be found in
/usr/src/linux/System.map. When klogd starts under the
experimental kernel the map in /boot/System.map will be
bypassed in favor of the map in /usr/src/linux/System.map.
Modern kernels as of 1.3.43 properly format important ker-
nel addresses so that they will be recognized and trans-
lated by klogd. Earlier kernels require a source code
patch be applied to the kernel sources. This patch is
supplied with the sysklogd sources.
The process of analyzing kernel protections faults works
very well with a static kernel. Additional difficulties
are encountered when attempting to diagnose errors which
occur in loadable kernel modules. Loadable kernel modules
are used to implement kernel functionality in a form which
can be loaded or unloaded at will. The use of loadable
modules is useful from a debugging standpoint and can also
be useful in decreasing the amount of memory required by a
kernel.
The difficulty with diagnosing errors in loadable modules
is due to the dynamic nature of the kernel modules. When
a module is loaded the kernel will allocate memory to hold
the module, when the module is unloaded this memory will
be returned back to the kernel. This dynamic memory allo-
cation makes it impossible to produce a map file which
details the addresses of the variable and functions in a
kernel loadable module. Without this location map it is
not possible for a kernel developer to determine what went
wrong if a protection fault involves a kernel module.
klogd has support for dealing with the problem of diagnos-
ing protection faults in kernel loadable modules. At pro-
gram start time or in response to a signal the daemon will
interrogate the kernel for a listing of all modules loaded
and the addresses in memory they are loaded at. Individ-
ual modules can also register the locations of important
functions when the module is loaded. The addresses of
these exported symbols are also determined during this
interrogation process.
When a protection fault occurs an attempt will be made to
resolve kernel addresses from the static symbol table. If
this fails the symbols from the currently loaded modules
are examined in an attempt to resolve the addresses. At
the very minimum this allows klogd to indicate which load-
able module was responsible for generating the protection
fault. Additional information may be available if the
module developer chose to export symbol information from
the module.
Proper and accurate resolution of addresses in kernel mod-
ules requires that klogd be informed whenever the kernel
module status changes. The -i and -I switches can be used
to signal the currently executing daemon that symbol
information be reloaded. Of most importance to proper
resolution of module symbols is the -i switch. Each time
a kernel module is loaded or removed from the kernel the
following command should be executed:
klogd -i
The -p switch can also be used to insure that module sym-
bol information is up to date. This switch instructs
klogd to reload the module symbol information whenever a
protection fault is detected. Caution should be used
before invoking the program in 'paranoid' mode. The sta-
bility of the kernel and the operating environment is
always under question when a protection fault occurs.
Since the klogd daemon must execute system calls in order
to read the module symbol information there is the possi-
bility that the system may be too unstable to capture use-
ful information. A much better policy is to insure that
klogd is updated whenever a module is loaded or unloaded.
Having uptodate symbol information loaded increases the
probability of properly resolving a protection fault if it
should occur.
Included in the sysklogd source distribution is a patch to
the modules-2.0.0 package which allows the insmod, rmmod
and modprobe utilities to automatically signal klogd when-
ever a module is inserted or removed from the kernel.
Using this patch will insure that the symbol information
maintained in klogd is always consistent with the current
kernel state.
SIGNAL HANDLING
The klogd will respond to eight signals: SIGHUP, SIGINT,
SIGKILL, SIGTERM, SIGTSTP, SIGUSR1, SIGUSR2 and SIGCONT.
The SIGINT, SIGKILL, SIGTERM and SIGHUP signals will cause
the daemon to close its kernel log sources and terminate
gracefully.
The SIGTSTP and SIGCONT signals are used to start and stop
kernel logging. Upon receipt of a SIGTSTP signal the dae-
mon will close its log sources and spin in an idle loop.
Subsequent receipt of a SIGCONT signal will cause the dae-
mon to go through its initialization sequence and re-
choose an input source. Using SIGSTOP and SIGCONT in com-
bination the kernel log input can be re-chosen without
stopping and restarting the daemon. For example if the
/proc file system is to be un-mounted the following com-
mand sequence should be used:
# kill -TSTP pid
# umount /proc
# kill -CONT pid
Notations will be made in the system logs with LOG_INFO
priority documenting the start/stop of logging.
The SIGUSR1 and SIGUSR2 signals are used to initiate load-
ing/reloading of kernel symbol information. Receipt of
the SIGUSR1 signal will cause the kernel module symbols to
be reloaded. Signaling the daemon with SIGUSR2 will cause
both the static kernel symbols and the kernel module sym-
bols to be reloaded.
Provided that the System.map file is placed in an appro-
priate location the signal of generally greatest useful-
ness is the SIGUSR1 signal. This signal is designed to be
used to signal the daemon when kernel modules are
loaded/unloaded. Sending this signal to the daemon after
a kernel module state change will insure that proper reso-
lution of symbols will occur if a protection fault occurs
in the address space occupied by a kernel module.
FILES
/proc/kmsg
One Source for kernel messages klogd
/var/run/klogd.pid
The file containing the process id of klogd
/boot/System.map, /System.map, /usr/src/linux/System.map
Default locations for kernel system maps.
BUGS
Probably numerous. Well formed context diffs appreciated.
AUTHOR
The klogd was originally written by Steve Lord
lord@cray.com, Greg Wettstein made major improvements.
Dr. Greg Wettstein greg@wind.rmcc.com
Enjellic Systems Development
Oncology Research Divsion Computing Facility
Roger Maris Cancer Center
Fargo, ND 58122