manolismarazakis [email protected] may 10, 2019hy428/reading/lkm_pci_may10_2019.pdf · an...

An introductory tutorial on how to develop and test Linux kernel

modules

Manolis Marazakis

[email protected]

May 10, 2019

In this tutorial …

• Familiarization with Linux kernel modules

– Two examples

• Learn about ways to explore the operation of a live Linux system

• Introduction to PCI device drivers

Introduction to Linux Kernel Modules

Introduction to Linux Kernel Module Programming (aka: I can write a driver me!)

Linux Kernel module : code that can be loaded/unloaded without system reboot- Extensions of kernel functionality, in the form of a “special” object file- Loadable vs. Built-in- Device driver allows kernel to access hardware connected to the system

The kernel & its modules essentially are a single program (& single namespace)

Credit for title: “Brainiac: Science Abuse”[ https://en.wikipedia.org/wiki/Brainiac:_Science_Abuse ]


Computer systems are built on levels of abstraction

• Different perspectives on what a “machine” is– OS ISA: Instruction Set

Architecture• h/w – s/w interface

– Compiler ABI: Application Binary Interface• User ISA + OS calls• Calling conventions

– Application API: Application Programming Interface• User ISA + Library calls


ISA

ABIAPI

The OS as a resource manager

• Operating system := software that controls and manages the hardware and system resources of a computer

– Abstraction and Isolation

– Virtualization of system platform resources: processes, memory, filesystem, devices


+ Concurrency, +Persistence …

Kernel space

• The executing code has complete and unrestricted access to the underlying hardware.

– Can execute any CPU instruction and reference any memory address.

– Intended for the lowest-level, most trusted functions of the OS.

– Crashes in kernel mode are catastrophic; they will halt the entire PC.


User space

• The executing code has no ability to directlyaccess hardware or reference memory.

– Code running in user mode must delegate to system APIs to access hardware or memory.

– Due to the protection afforded by this sort of isolation, crashes in user mode are always recoverable.


Kernel vs User: enforced by CPU HWIf code executing in User modeattempts to do something outside itspurview– e.g. accessing a privilegedCPU instruction or modifyingmemory that it has no access to -- atrappable exception is thrown Instead of the entire systemcrashing, only that particularapplication crashes.


Protection rings (x86)

Linux: Kernel- vs. User- Space


system call: - A function stub - entry point for requesting OS services.

Embrace change (with kernel modules)


View list of modules: /proc/modules (also: lsmod utility)

“Hello World” example/**

* lkm0.c : "null" Linux kernel module

* FOSSCOMM 2018 @ U. Crete, Heraklion, Greece

*/

#include <linux/init.h>

#include <linux/module.h>

#include <linux/kernel.h>

MODULE_LICENSE("GPL");

MODULE_AUTHOR("FOSSCOM18");

MODULE_DESCRIPTION("A Linux kernel module for tutorial use");

MODULE_VERSION("0.00");

static int __init lkm_example_init(void) {

printk(KERN_INFO "[%s] Module initialed. \n", __FUNCTION__);

return 0;

}

static void __exit lkm_example_exit(void) {

printk(KERN_INFO "[%s] Module removed. \n", __FUNCTION__);

}

module_init(lkm_example_init);

module_exit(lkm_example_exit);Introduction to Linux Kernel Modules

Modules are built by the kbuild system

obj-m += lkm0.o

KERNEL = $(shell uname -r)

all:

make -C /lib/modules/$(KERNEL)/build M=$(PWD) modules

clean:

make -C /lib/modules/$(KERNEL)/build M=$(PWD) clean

To enable building kernel modules: apt-get install linux-headers-$(uname -r)


Exploring our first module: lkm0.ko

# sudo insmod ./lkm0.ko# cat /proc/modules# cat /proc/devices# lsmod# dmesg view kernel log# sudo rmmod lkm0


sysfs

• sysfs: a RAM-based filesystem, for exporting to user-space kernel data structures, their attributes, and the linkages between them – tied inherently to the kobject infrastructure

– mount -t sysfs sysfs /sys

• [ https://www.kernel.org/doc/Documentation/filesystems/sysfs.txt ]


A character driver (lkm1.c)#include <linux/kernel.h>

#include <linux/module.h>

#include <linux/fs.h>

#include <linux/cdev.h>

#include <asm/uaccess.h>

#include <linux/device.h>

#include <linux/sysfs.h>

#include <linux/slab.h>

….

static struct cdev *fosscomm18_cdev;

static struct device *fosscomm18_device;

static struct class *fosscomm18_class;

static dev_t fosscomm18_dev;

static char *fosscomm18_buffer; /* dynamic allocated */

static int fosscomm18_buffer_end = -1;

static atomic_t fosscomm18_buffer_start; /* may be changed by sysfs attr */


A character driver: readstatic struct file_operations fops = {

.owner = THIS_MODULE,

.read = dev_read,

.write = dev_write,

.open = dev_open,

.release = dev_release};static ssize_t dev_read(struct file *fp, char *buf, size_t len, loff_t *off) {

unsigned long rval; size_t copied; if (len > (fosscomm18_buffer_end - *off))

len = fosscomm18_buffer_end - *off; rval = raw_copy_to_user(buf, fosscomm18_buffer + *off, len); if (rval < 0) return -EFAULT; copied = len - rval; *off += copied; return copied;

}


A character driver: writestatic ssize_t dev_write(struct file *fp, const char *buf, size_t len, loff_t *off)

{

unsigned long rval; size_t copied;

printk(KERN_DEBUG DEVICE_NAME

" dev_write(fp, buf, len = %zu, off = %d\n", len, (int)*off);

if (len > fosscomm18_buffer_end - *off)

len = fosscomm18_buffer_end - *off;

rval = raw_copy_from_user(

fosscomm18_buffer + atomic_read(&fosscomm18_buffer_start), buf, len);

if (rval < 0) {

printk(KERN_DEBUG DEVICE_NAME " copy_from_user() failed\n");

return -EFAULT;

}

copied = len - rval;

*off += copied;

return copied;

}


A character driver: initialization


fosscomm18_buffer = kmalloc(PAGE_SIZE, GFP_KERNEL);

if (!fosscomm18_buffer) return -ENOMEM;

fosscomm18_buffer_end = PAGE_SIZE;

rval = alloc_chrdev_region(&fosscomm18_dev, 1, 1, DEVICE_NAME);

fosscomm18_cdev = cdev_alloc();

rval = cdev_add(fosscomm18_cdev, fosscomm18_dev, 1);

fosscomm18_class = class_create(THIS_MODULE, CLASS_NAME);

fosscomm18_device = device_create(fosscomm18_class, NULL, fosscomm18_dev, NULL, DEVICE_NAME);

rval = device_create_file(fosscomm18_device, &dev_attr_buffer_start);

rval = device_create_file(fosscomm18_device, &dev_attr_buffer_end);

A character driver: sysfsstatic ssize_t buffer_start_show(struct device *dev,

struct device_attribute *attr, char *buf)

{

return snprintf(buf, PAGE_SIZE, "%d\n“, atomic_read(&fosscomm18_buffer_start));

}

static ssize_t buffer_end_show(struct device *dev,

struct device_attribute *attr, char *buf)

{

return snprintf(buf, PAGE_SIZE, "%d\n", fosscomm18_buffer_end);

}


Exploring the character driver module


Curious enough to explore further ?• Remzi H. Arpaci-Dusseau and Andrea C. Arpaci-Dusseau, “Operating Systems: “Three

easy pieces” [ http://pages.cs.wisc.edu/~remzi/OSTEP/ ]

• Write a Real Linux Driver [ webinar by Greg Kroah-Hartman ]

– https://training.linuxfoundation.org/resources/webinars/write-a-real-linux-driver/

• Linux kernel development tutorial

– Video of Greg Kroah-Hartman’s talk at Git Merge 2016https://www.youtube.com/watch?v=vyenmLqJQjs

• Linux kernel driver interface

– https://www.kernel.org/doc/html/v4.15/process/stable-api-nonsense.html

• LDD3 book : https://lwn.net/Kernel/LDD3/

– Warning: dated (but explains core principles)

• LXR (cross-referenced source code)

– https://elixir.bootlin.com/linux/latest/source

• Tutorial on Linux PCI Drivers– http://freeelectrons.com/docs/pcidrivers


What’s going on ???

• The Linux kernel has three primary mechanisms for kernel tracing and profiling:– tracepoints – a mechanism that works over static instrumented

code– kprobes – a dynamic tracing mechanism used to interrupt a

kernel code at any point, call its own handler, and return after all of the necessary operations have been completed

– perf_events – an interface for accessing the PMU (Performance Monitoring Unit)

• cat /sys/kernel/debug/tracing/available_tracers– hwlat blk mmiotrace function_graph wakeup_dl wakeup_rt

wakeup function nop

• Front-ends for ftrace: trace-cmd, kernelshark


# trace-cmd record -e ext4 ls# trace-cmd report


Try also: # trace-cmd record -p function_graph –P <PID> (e.g. PID of sshd)# trace-cmd list –f# cat /sys/kernel/debug/tracing/available_events

ftrace (via trace-cmd)

• ftrace tracing framework

– https://www.kernel.org/doc/Documentation/trace/ftrace.txt

• trace-cmd: A front-end for Ftrace

– https://lwn.net/Articles/410200/

• Getting started with ftrace (Julia Evans’s blog)

– https://jvns.ca/blog/2017/03/19/getting-started-with-ftrace/


perf-tools


PCI Devices in Linux (1)


/sys/devices/pci0000:00/0000:00:1e.0/0000:02:01.2

PCI Devices in Linux (2)

• Configuration space (256 bytes), per device– View via: lspci –x

• Standardized format (first 64 bytes)– Offset 0: Vendor ID, Offset: 2: Device ID

– Offset 10: Class ID (e.g. NIC, video adapter, bridge)

– Offsets 16-39: BAR [0-5]• Base Address Registers

– Offset 44: Subvendor ID | Offset 46: Subdevice ID

– Rest of config. Space is up to the manufacturer


Linux PCI Drivers (1)

• Device resources (I/O addresses, IRQ lines) are assigned at boot time

• PCI drivers “simply” have to read the corresponding configurations

• PCI config. space is Little-Endian.



• Declaration of driver hooks & supported devicesstatic struct pci_device_id pci_ids[] = {

{ PCI_DEVICE(QEMU_VENDOR_ID, EDU_DEVICE_ID), },{ 0, }

};MODULE_DEVICE_TABLE(pci, pci_ids);

static struct pci_driver pci_driver = {.name = “edu_pci",.id_table = pci_ids,.probe = pci_probe, /* called by generic PCI subsystem for matching devices */.remove = pci_remove/* could also provide hooks for PM: suspend/resume */

};



static int __init myinit(void) {

if (pci_register_driver(&pci_driver) < 0) { return 1; }

return 0;

}

static void __exit myexit(void) {

pci_unregister_driver(&pci_driver);

}

module_init(myinit);

module_exit(myexit);



static int pci_probe(struct pci_dev *dev,

const struct pci_device_id *id)

static void pci_remove(struct pci_dev *dev)


Warning – things might break … but you can fix them!


• ctags, find, grep, vim/emacs, … • printk() calls (conditional), sysfs, /proc, debugfs• _many_ kernel configuration features (via

Kconfig settings)• gdb vmlinux /proc/kcore• nm, objdump, /proc/kallsyms• … and you’ll grow to love virtual machines :-D

Don’t panic, everything is under control


Thank you for attention!

manolismarazakis [email protected] may 10, 2019hy428/reading/lkm_pci_may10_2019.pdf · an...

Documents