2.7 System Memory Layout

To best use the available resources, it is important to understand the system's memory layout, and the differences between the physical address space and the kernel's virtual address space.^[4] Most importantly, many hardware peripherals are accessible within the system's physical address space, but have restricted access or are completely "invisible" in the virtual address space.

^[4] What I call here "virtual address" is known in x86 jargon as "logical address" and can have other names on other architectures.

To best illustrate the difference between virtual and physical address spaces, let's take a closer look at one component of the example system. The user interface modules, for instance, can be easily implemented on the StrongARM-based iPAQ PDA. Figure 2-6 illustrates the physical and virtual memory maps of an iPAQ running the Familiar distribution. Note that the regions illustrated are not necessarily proportional to their actual size in memory. If they were, many of them would be too small to be visible.

Figure 2-6. Physical and virtual memory maps for the Compaq iPAQ

The physical map of a system is usually available with the technical literature accompanying your hardware. In the case of the iPAQ, the StrongARM manual, the SA-1110 Developer's manual, is available from Intel's web site.

The physical map is important, because it provides you with information on how to configure the kernel and how to develop custom drivers. During the kernel's configuration, for instance, you may need to specify the location of the flash devices in your system. During development, you may also need to write a driver for a memory-mapped peripheral. You will also need to provide your bootloader with information regarding the components it has to load. For these reasons, it is good practice to take the time to establish your system's physical memory map before starting software development.

On the iPAQ, the flash storage is divided in two. The first part contains the bootloader and starts at the lowest memory address available. Given the bootloader's size, this region is rather small. The rest of the flash storage space is occupied by the system's root filesystem, which in the case of Familiar, is a JFFS2 filesystem. In this case, the kernel is actually on the root filesystem. This is possible, because the bootloader has enough understanding of JFFS2 to find the kernel on the filesystem.

Upon startup, the bootloader reads the kernel from the root filesystem into the system's RAM and jumps to the kernel's start routines. From there on, the rest of the system startup is carried out by Linux.

Once Linux is running,^[5] the programs use virtual addresses. In contrast to the physical memory map, the layout of the virtual memory map is of secondary importance for kernel configuration or device driver development. For device driver development, for instance, it is sufficient to know that some information is located in kernel space and some other information is located in user space, and that appropriate functions must be used to properly exchange data between the two.

^[5] I assume that you are using MMU-equipped hardware. This discussion does not hold if you are using a Linux variant for MMU-less processors.

The virtual memory layout is mostly important in helping you understand and debug your applications. As you can see in Figure 2-6, the kernel occupies a quarter of the virtual address space starting from address 0xC0000000. This region is also known as "kernel space." The rest of the address space is occupied by application-specific text, data, and library mappings. This is also known as "user space." Whereas the kernel is always located above the 0xC0000000 mark for all applications, applications' memory maps may differ even on the same system.

To reconstruct a process' virtual memory map, you need to look at the maps file in the process' PID entry in the /proc filesystem. For more details on how to get this information, see the Chapter 20 in Understanding the Linux Kernel (O'Reilly).