Kernel Services, Configuration & Modules

systemd

systemd is the init system and service manager for virtually all major Linux distributions. It replaced SysVinit’s sequential shell scripts with parallel, dependency-based service startup.

/sbin/init is a symlink to /lib/systemd/systemd on modern systems. systemd becomes PID 1 and starts everything else.

Why systemd is Faster

SysVinit	systemd
Runs startup scripts sequentially	Starts services in parallel
Long bootup chain	Dependency graph allows safe parallelism
Services started eagerly	Socket activation: open socket first; start service lazily when needed
No on-demand starting	D-Bus activation: start services when first called
Global cgroups	Tracks every service’s processes with per-unit cgroups

Unit Files

systemd manages resources (services, sockets, timers, mounts, etc.) as units, described by .unit files. Service units use the .service suffix.

Section	Key Fields	Purpose
[Unit]	Description, After, Requires, Wants	Metadata and dependencies
[Service]	ExecStart, ExecStop, Restart, Type, User	How to start/stop/run the service
[Install]	WantedBy	Which target enables this service

Unit File Locations

Path	Priority	Use for
/etc/systemd/system/	Highest	Admin overrides and custom units
/run/systemd/system/	Medium	Runtime-generated units
/lib/systemd/system/	Lowest	Distribution-shipped unit files (don’t edit)

Same filename in a higher-priority location overrides the lower-priority one.

Example: dbus.service

/lib/systemd/system/dbus.service

[Unit]
Description=D-Bus System Message Bus
Documentation=man:dbus-daemon(1)
Requires=dbus.socket

[Service]
ExecStart=/usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation \
  --syslog-only
OOMScoreAdjust=-900

[Install]
WantedBy=multi-user.target

Creating a Custom Service

/etc/systemd/system/myapp.service

[Unit]
Description=My Application
After=network.target

[Service]
ExecStart=/usr/bin/myapp --config /etc/myapp.conf
Restart=on-failure
User=myapp
Group=myapp
WorkingDirectory=/var/lib/myapp

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload          # reload all unit files after changes
sudo systemctl enable myapp.service   # start at boot (creates symlink in target)
sudo systemctl start myapp.service    # start now

systemctl - Control systemd

# Service lifecycle
sudo systemctl start httpd            # start
sudo systemctl stop httpd             # stop
sudo systemctl restart httpd          # stop then start
sudo systemctl reload httpd           # reload config without restarting (if supported)
sudo systemctl status httpd           # show status, last log lines, PID

# Enable/disable at boot
sudo systemctl enable httpd           # enable (start at boot)
sudo systemctl disable httpd          # disable (don't start at boot)
sudo systemctl enable --now httpd     # enable AND start immediately

# Masking (stronger disable - prevents any activation)
sudo systemctl mask httpd             # can't be started at all (even manually)
sudo systemctl unmask httpd           # restore

# Inspect units
systemctl list-units -t service       # active service units
systemctl list-units -t service --all # all service units including inactive
systemctl list-unit-files             # installed units and enabled state
systemctl list-units --failed         # only failed units

# Targets (like runlevels)
systemctl get-default                 # show default target
sudo systemctl set-default multi-user.target
sudo systemctl isolate rescue.target  # switch target immediately

# Edit unit files
sudo systemctl edit httpd.service     # create override (drop-in) - PREFERRED
sudo systemctl edit httpd.service --full  # edit full copy of unit file

systemctl Status States

State	Meaning
active (running)	Process is running
active (exited)	One-shot service completed successfully
active (waiting)	Waiting for an event
inactive (dead)	Not running; never started or stopped cleanly
failed	Process exited with error, crashed, or timed out
masked	Unit is disabled and cannot be started

journalctl - View systemd Logs

systemd’s journal collects log output from all services and the kernel into a structured binary database.

journalctl                            # all logs (oldest first)
journalctl -r                         # newest first
journalctl -b                         # logs from current boot only
journalctl -b -1                      # logs from previous boot
journalctl --list-boots               # show all boots with their IDs

journalctl -u httpd.service           # logs for a specific unit
journalctl -u httpd.service -f        # follow (tail -f equivalent)
journalctl -u httpd.service --since "1 hour ago"
journalctl -u httpd.service --since "2025-03-01" --until "2025-03-02"

journalctl -p err                     # only errors (priority filter)
journalctl -p warning                 # warnings and above
journalctl -k                         # kernel messages only (like dmesg)

journalctl --disk-usage               # check journal size
sudo journalctl --vacuum-size=500M    # trim journal to 500 MB

By default, journals may not persist across reboots. Enable persistence:

sudo mkdir -p /var/log/journal
sudo systemctl restart systemd-journald

---

Kernel Modules

The Linux kernel is designed to be minimal at its core. Device drivers and optional features are loaded as modules on demand. This keeps the kernel image small and allows new hardware to be supported without recompilation.

Module Utilities

Utility	Purpose
lsmod	List all currently loaded modules and their dependencies
modinfo <module>	Show metadata: description, author, license, parameters, filename
modprobe <module>	Load a module (resolves and loads dependencies automatically)
modprobe -r <module>	Unload a module (and unneeded dependencies)
insmod <path.ko>	Load a module directly by path (no dependency resolution)
rmmod <module>	Unload a module (no dependency handling; use modprobe -r instead)
depmod	Rebuild the module dependency database

# List all loaded modules
lsmod

# Inspect a module
modinfo e1000
# name, author, description, license, alias, depends, filename, parameters ...

# Load a module
sudo modprobe e1000

# Unload a module
sudo modprobe -r e1000

# Load with parameters
sudo modprobe e1000 debug=1 copybreak=256

# See what parameters a module accepts
modinfo -p e1000

Persistent Module Loading

To load a module at every boot:

# Create a config file in /etc/modules-load.d/
echo "e1000" | sudo tee /etc/modules-load.d/e1000.conf

# Pass module parameters
echo "options e1000 debug=1" | sudo tee /etc/modprobe.d/e1000.conf

Blacklisting a Module

To prevent a module from loading (e.g., a problematic driver):

echo "blacklist nouveau" | sudo tee /etc/modprobe.d/blacklist-nouveau.conf
sudo update-initramfs -u    # Debian/Ubuntu - rebuild initramfs
sudo dracut --force          # RHEL/Fedora - rebuild initramfs

---

sysctl - Runtime Kernel Parameters

sysctl reads and writes kernel tunable parameters exposed in /proc/sys/. Changes take effect immediately but don’t survive a reboot unless persisted.

# Show all current settings
sysctl -a

# Read one setting (two equivalent ways)
sysctl net.ipv4.ip_forward
cat /proc/sys/net/ipv4/ip_forward

# Set temporarily (lost on reboot)
sudo sysctl -w net.ipv4.ip_forward=1

# Set permanently (survives reboot)
echo "net.ipv4.ip_forward = 1" | sudo tee /etc/sysctl.d/99-ip-forward.conf
sudo sysctl -p /etc/sysctl.d/99-ip-forward.conf    # apply now

# Apply all files in /etc/sysctl.d/
sudo sysctl --system

Common sysctl Parameters

Parameter	Default	Purpose
net.ipv4.ip_forward	0	Enable IP routing (required for routers/NAT)
net.ipv4.tcp_syncookies	1	SYN flood protection
vm.swappiness	60	0=prefer RAM, 100=prefer swap aggressively
vm.dirty_ratio	20	Max % of RAM for dirty (unwritten) pages
kernel.panic	0	Seconds before reboot after kernel panic (0=never)
fs.file-max	varies	System-wide max open file descriptors
net.core.somaxconn	128	Max listen backlog for TCP connections

---

udev - Device Management

udev is the kernel’s userspace device manager. When hardware is added or removed, the kernel fires a uevent; udev receives it, parses rule files, and automatically:

• Creates/removes device nodes in /dev
• Loads appropriate driver modules
• Sets ownership and permissions on device files
• Creates persistent symlinks (e.g., /dev/disk/by-uuid/, /dev/disk/by-id/)
• Runs custom scripts

Key Directories

Path	Contents
/dev/	Device nodes (managed by udev at runtime)
/etc/udev/rules.d/	Admin custom rules (highest priority)
/run/udev/rules.d/	Runtime rules
/usr/lib/udev/rules.d/	Distro-shipped rules (don’t edit)

Rules in /etc/udev/rules.d/ override same-named files in /usr/lib/udev/rules.d/. Files are processed in alphabetical order.

udev Rule Format

MATCH_KEY=="value" [, ...] , ASSIGNMENT_KEY="value" [, ...]

Match operators	Assignment operators
== exact match	= set value
!= not match	+= append (symlinks, LABEL, etc.)
~= regex match	:= final set (cannot be changed by later rules)

Common match keys: KERNEL, SUBSYSTEM, ATTR{attribute}, DRIVER, ENV{variable} Common assignment keys: NAME, SYMLINK, MODE, OWNER, GROUP, RUN

Example Rules

/etc/udev/rules.d/99-fitbit.rules

# Give Fitbit tracker access without root
SUBSYSTEM=="usb", ATTR{idVendor}=="2687", ATTR{idProduct}=="fb01", \
  SYMLINK+="fitbit", MODE="0666"

# /etc/udev/rules.d/70-custom-disk.rules
# Give a specific disk a stable name
KERNEL=="sdb", SUBSYSTEM=="block", SYMLINK+="my-spare-disk"

# Run a script when a USB device is connected
KERNEL=="sdb", DRIVER=="usb-storage", RUN+="/usr/local/bin/backup-disk.sh"

# Set group ownership on a device
KERNEL=="vboxdrv", GROUP="vboxusers", MODE="0660"

Creating and Testing Rules

# Reload rules without reboot
sudo udevadm control --reload-rules

# Trigger rules on a device (re-apply without replug)
sudo udevadm trigger /dev/sdb

# Monitor udev events in real-time (plug/unplug USB to see events)
sudo udevadm monitor

# Inspect attributes of a device (use to write match rules)
udevadm info --attribute-walk /dev/sdb
udevadm info --query=all --name=/dev/sdb