Kernel Services, Configuration & Modules

Kernel, init and Services

Linux Kernel

The boot loader loads both the kernel and an initial RAM–based file system (initramfs) into memory, so it can be used directly by the kernel.
When the kernel is loaded in RAM, it immediately initializes and configures the computer’s memory and also configures all the hardware attached to the system. This includes all processors, I/O subsystems, storage devices, etc. The kernel also loads some necessary user space applications.

`/sbin/init` and Services

Once the kernel has set up all its hardware and mounted the root file-system, the kernel runs /sbin/init. This then becomes the initial process, which then starts other processes to get the system running.
The /sbin/init program (usually just called init) is the first user-level process (or task) run on the system and continues to run until the system is shut down.
Most other processes on the system trace their origin ultimately to init; exceptions include the so-called kernel processes. These are started by the kernel directly, and their job is to manage internal operating system details.
Besides starting the system, init is responsible for keeping the system running and for shutting it down cleanly. One of its responsibilities is to act when necessary as a manager for all non-kernel processes; it cleans up after them upon completion, and restarts user login services as needed when users log in and out, and does the same for other background system services.
Traditionally, this process startup was done using conventions that date back to the 1980s and the System V variety of UNIX. This serial process had the system passing through a sequence of runlevels containing collections of scripts that start and stop services. Each runlevel supported a different mode of running the system. Within each runlevel, individual services could be set to run, or to be shut down if running.
However, all major distributions have moved away from this sequential runlevel method of system initialization, although they usually emulate many System V utilities for compatibility purposes.

Startup Alternatives

SysVinit viewed things as a serial process, divided into a series of sequential stages. Each stage required completion before the next could proceed. Thus, startup did not easily take advantage of the parallel processing that could be done on multiple processors or cores.
Furthermore, shutdown and reboot was seen as a relatively rare event; exactly how long it took was not considered important. This is no longer true, especially with mobile devices and embedded Linux systems. Some modern methods, such as the use of containers, can require almost instantaneous startup times. Thus, systems now require methods with faster and enhanced capabilities.
Finally, the older methods required rather complicated startup scripts, which were difficult to keep universal across distribution versions, kernel versions, architectures, and types of systems. The two main alternatives developed were:

Upstart

Developed by Ubuntu and first included in 2006
Adopted in Fedora 9 (in 2008) and in RHEL 6 and its clones

systemd

Adopted by Fedora first (in 2011)
Adopted by RHEL 7 and SUSE
Replaced Upstart in Ubuntu 16.04

While the migration to systemd was rather controversial, it has been adopted by all major distributions, and so we will not discuss the older System V method or Upstart, which has become a dead end. Regardless of how one feels about the controversies or the technical methods of systemd, almost universal adoption has made learning how to work on Linux systems simpler, as there are fewer differences among distributions.

systemd

Systems with systemd start (boot) up faster than those with earlier init methods. This is largely because it replaces a serialized set of steps with aggressive parallelization techniques, which permits multiple services to be initiated simultaneously.
Complicated startup shell scripts are replaced with simpler configuration files, which enumerate what has to be done before a service is started, how to execute service startup, and what conditions the service should indicate have been accomplished when startup is finished. One thing to note is that /sbin/init now just points to /lib/systemd/systemd; i.e. systemd takes over the init process.

The systemd system and session manager for Linux is now dominant in most major distributions. Features include the following:

Boots faster than previous init systems
Provides aggressive parallelization capabilities
Uses socket and D-Bus activation for starting services
Replaces shell scripts with programs
Offers on-demand starting of daemons
Keeps track of processes using cgroups
Maintains mount and automount points
Implements an elaborate transactional dependency-based service control logic
Provides detailed start and stop functionality, including similar abilities to cron

Instead of bash scripts, systemd uses .service files. Granular logging is provided by default using the name of the unit. Tracking and creating user processes without root privileges is another powerful ability of systemd.

systemd Configuration File Locations

systemd allows for multiple locations for its unit files, allowing for processes running under a non-root user, and inheritance from system unit files. Here are some typical locations (but may vary by distribution):

System unit files: /etc/systemd/system and /lib/systemd/system
User unit files: /etc/systemd/user and ~/.config/systemd/user

On most systems, /lib/systemd/ is a symlink to /usr/lib/systemd/.

Here is an example of a standard system unit file from Debian:

/lib/systemd/system/dbus.service
[Unit]
Description=D-Bus System Message Bus
Documentation=man:dbus-daemon(1)
Requires=dbus.socket

[Service]
ExecStart=/usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation \
  --syslog-only
ExecReload=/usr/bin/dbus-send --print-reply --system --type=method_call --dest=org.freedesktop.DBus / \
  org.freedesktop.DBus.ReloadConfig OOMScoreAdjust=-900

systemd Tools

`systemctl`	`journalctl`
Manage system state	Query systemd journal
Start/Stop/Restart/Reload
Enable/Disable
List and manage units
List and update targets

One systemd command (systemctl) is used for most basic tasks:

# Starting, stopping, restarting, & reloading a service
sudo systemctl start|stop|restart|reload httpd.service

# Enabling or disabling a system service from starting up at system boot
sudo systemctl enable|disable httpd.service

# To check the status of a service
sudo systemctl status httpd.service
# STATE        Meaning
# Active       Service Running
# Inactive     Service Stopped
# Failed       Crashed/Error/Timeout etc.

# In most cases, the .service can be omitted.
# To make changes to service unit file
systemctl edit httpd.service --full

# After making changes to service unit file, reload
systemctl daemon-reload

# To show the status of everything that systemd controls:
systemctl

# To show all available services:
systemctl list-units -t service --all

# To show only active services:
systemctl list-units -t service

# To show a summary listing what is installed:
systemctl list-unit-files

# To see the default run level (target)
systemctl get-default

# To change the default target (runlevel)
systemctl set-default multi-user.target

# To list all units systemd has loaded
systemctl list-units --all

journalctl

journalctl is used to query and view logs collected by systemd’s journal. It is invaluable for troubleshooting systemd units.

By default, logs may not be persistent across reboots. Persistence is typically enabled by creating the directory /var/log/journal.

# Print all log entries from old to new
journalctl

# See logs from current boot
journalctl -b

# See the log of a particular unit
journalctl -u UNIT

# Follow logs in real-time
journalctl -f

# View logs for previous boots
journalctl --list-boots

Creating a systemd Service

To create a custom service, create a unit file in /etc/systemd/system/:

[Unit]
Description=My Application
After=network.target

[Service]
ExecStart=/usr/bin/myapp
Restart=on-failure

[Install]
WantedBy=multi-user.target

Then enable and start it:

sudo systemctl daemon-reload
sudo systemctl enable myapp.service
sudo systemctl start myapp.service

Kernel Modules

The Linux kernel makes extensive use of modules, which are loaded and unloaded on-the-fly as needed. This is a flexible approach that keeps the kernel from being too large while still supporting a wide variety of hardware and functionality.

Module Utilities

Utility	Purpose
`lsmod`	List loaded modules
`insmod`	Insert a module directly into the kernel
`rmmod`	Remove a module from the kernel
`modprobe`	Load or unload modules, using a pre-built module database with dependency and location information
`depmod`	Rebuild the module dependency database
`modinfo`	Display information about a module

# List all currently loaded kernel modules
lsmod

# Show information about a specific module
modinfo e1000

# Load a module (handles dependencies automatically)
sudo modprobe e1000

# Remove a module
sudo modprobe -r e1000

Module Parameters

Modules can take parameters that control their behavior. You can see what parameters a module accepts:

modinfo -p e1000

Parameters can be specified when loading a module:

sudo modprobe e1000 debug=1

sysctl

sysctl is used to read and tune kernel parameters at runtime. These parameters are found in the /proc/sys directory.

# Show all current settings
sysctl -a

# Read a particular setting
sysctl net.ipv4.ip_forward

# Set a value (does not survive reboot)
sudo sysctl net.ipv4.ip_forward=1

# Set a value persistently (survives reboot)
# Edit /etc/sysctl.conf or create a file in /etc/sysctl.d/
echo "net.ipv4.ip_forward = 1" | sudo tee /etc/sysctl.d/99-custom.conf
sudo sysctl -p /etc/sysctl.d/99-custom.conf

Devices and udev

udev

Linux uses udev, an intelligent apparatus for discovering hardware and peripheral devices both during system boot, and later when they are connected to the system.
Device Nodes are created automatically and then used by applications and operating system subsystems to communicate with and transfer data to and from devices.
System administrators can control how udev operates and craft special udev rules to assure desired behavior results.

udev stands for User Device management. It dynamically discovers built-in hardware as well as peripheral devices:

during system boot
when hot-plugged at any time

udev handles loading and unloading device drivers with proper configurations. udev actions include:

Device naming
Device file and symlink creating
Setting file attributes
Taking needed actions

When devices are added or removed from the system, udev receives a message from the kernel. It then parses the rules files in the /etc/udev/rules.d directory to see if any rules are there for the device added or removed.

These rules are totally customizable and can specify device file names, device file creation, specify symlinks to be created, specify file attributes to be set for the device file (including user and group ownership), and even specify actions to be taken (programs to be executed).

Device Nodes

Character and block devices have device nodes; network devices do not. These device nodes can be used by programs to communicate with devices through nodes using normal I/O methods.

A device driver may use multiple device nodes. Device nodes are located in the /dev directory.

ls -l /dev
total 0
crw-------.  1 root root       10,   123 Jul 29 08:27 acpi_thermal_rel
crw-r--r--.  1 root root       10,   235 Jul 29 08:27 autofs
drwxr-xr-x.  2 root root             540 Jul 29 13:11 block
drwxr-xr-x.  2 root root              60 Jul 29 13:11 bsg
crw-rw----.  1 root disk       10,   234 Jul 29 08:27 btrfs-control
...

Device nodes can be created with the following command:

sudo mknod [-m mode] /dev/name <type> </major> <minor>

# example
sudo mknod -m 666 /dev/mycdrv c 254 1

Device Nodes Provide Hardware Access

In addition to providing some required (statically created outside of udev) pseudo-devices, including /dev/null and /dev/zero, the device nodes present in /dev are the primary or first interaction with the hardware on a machine.

Some common devices are:

/dev/sd* – devices appearing as hard drives
/dev/ttyS* – serial ports, often used as consoles
/dev/snd/* – various sound devices

udev Components

udev runs as a daemon (either udevd or systemd-udevd) and monitors a netlink socket.

When new devices are initialized or removed, the uevent kernel facility sends a message through the socket, which udev receives and takes appropriate action to create or remove device nodes of the right names and properties according to the rules.

The three components of udev are:

The libudev library which allows access to information about the devices.
The udevd or systemd-udevd daemon that manages the /dev directory.
The udevadm utility for control and diagnostics.

The cleanest way to use udev is to have a pure system; the /dev directory is empty upon the initial kernel boot, and then is populated with device nodes as they are needed. When used this way, one must boot using an initramfs image, which may contain a set of preliminary device nodes, as well as the udev infrastructure.

udev Rule Files

udev rules files are located under /etc/udev/rules.d/<rulename>.rules and /usr/lib/udev/rules.d/<rulename>.rules with names like:

30-usb.rules
90-mycustom.rules
70-mouse.rules
60-persistent-storage.rules

There are two separate parts defined on a single line:

The first part consists of one or more match pairs denoted by ==. These try to match a device’s attributes and/or characteristics to some value.
The second part consists of one or more assignment key-value pairs that assign a value to a name, such as a file name, assignment, even file permissions, etc.

If no matching rule is found, it uses the default device node name and other attributes.

cat /etc/udev/rules.d/99-fitbit.rules
SUBSYSTEM=="usb", ATTR{idVendor}=="2687", ATTR{idProduct}=="fb01", SYMLINK+="fitbit", MODE="0666"

cat /etc/udev/rules.d/60-vboxdrv.rules
KERNEL=="vboxdrv", NAME="vboxdrv", OWNER="root", GROUP="vboxusers", MODE="0660"
KERNEL=="vboxdrvu", NAME="vboxdrvu", OWNER="root", GROUP="root", MODE="0666"
KERNEL=="vboxnetctl", NAME="vboxnetctl", OWNER="root", GROUP="vboxusers", MODE="0660"

Creating udev Rules

Rules files can be in three places, and if they have the same name the priority order is:

/etc/udev/rules.d
/run/udev/rules.d
/usr/lib/udev/rules.d

The format for a udev rule is simple:

<match><op>value [, ...] <assignment><op>value [, ... ]

Samples:

KERNEL=="sdb", NAME="my-spare-disk"
KERNEL=="sdb", DRIVER=="usb-disk", SYMLINK+="sparedisk"
KERNEL=="sdb", RUN+="/usr/bin/my-program"
KERNEL=="sdb", MODE="0660", GROUP="mygroup"

You can monitor how these rules are applied in real-time by running:

sudo udevadm monitor

Try plugging and unplugging a USB device as a quick way to generate output from the command.