Separating user space from kernel space on ARM Cortex-M3

1 . Introduction

ARM Cortex-M processors are in SoCs of several application domains, especially in much of what we call smart devices. This publication continues the previous one, in which I had demonstrated the implementation of a minimum preemptive scheduling mechanism for an ARM Cortex-M3, taking advantage of the special hardware resources for context switching.

Other architecture features are now explored: the separation of user and kernel threads and the use of Supervisor Calls to implement system calls.

Although concepts about operating systems are addressed because they are inherent to the subject, the main goal is to explore the ARM Cortex-M3, a relatively inexpensive processor with wide applicability.

2. Special registers

There are 3 special registers on the ARM Cortex-M3. You can consult the ARM documentation to understand the role of each of them in the processor. The most important in this publication is CONTROL .

• Program Status Registers (APSR, IPSR, and EPSR)
• Interrupt Mask Registers (PRIMASK, FAULTMASK and BASEPRI)
• Control (CONTROL) .

Special registers can only be accessed through the privileged instructions MRS (ARM Read Special Register) and MSR (ARM Set Special Register):

// Load in R0 the current value contained in the special register 
MRS R0, SPECIAL  
// Load in the special register the value contained in R0) 
MSR SPECIAL, R0 

The CONTROL register has only 2 configurable bits. When an exception handler (e.g, SysTick_Handler) is running, the processor will be in privileged mode and using the main stack pointer (MSP) , and CONTROL [1] = 0, CONTROL [0] = 0 . In other routines that are not handlers, this register can assume different values ​​depending on the software implementation (Table 1).

In the small kernel shown before, the application tasks (Task1, Task2 and Task3) were also executed in privileged mode and using the main stack pointer (MSP). Thus, an application program could change the special registers of the core if it wanted to.

3. Kernel and user domains

In the last publication I highlighted the fact that register R13 is not part of the stackframe, as it stores the address of the current stack pointer. The R13 is a “banked” register, meaning that it is physically replicated, and takes a value or another depending on the state of the core.

With two stack pointers, one for application and another for the kernel, means that a user thread can not easily corrupt the kernel stack by a programming application error or malicious code. According to the ARM manuals, a robust operating system typically has the following characteristics:

• interrupt handlers use MSP (by default)
• kernel routines are activated through SysTick at regular intervals to perform task scheduling and system management in privileged mode
• user applications use PSP in non-privileged mode
• memory for kernel routines can only be accessed in privileged mode* and use MSP

* for now we will not isolate the memory spaces

4. System Calls

Putting it simple, a system call is a method in which a software requests a service from the kernel or OS on which it is running. If we intend to separate our system into privilege levels, it is inevitable that the application level needs call the kernel to have access to, for example, hardware services or whatever else we think is critical to the security and stability of our system.

A common way to implement system calls in ARM Cortex-M3 (and in other ARMv7) is to use the software interrupt Supervisor Call (SVC). The SVC acts as an entry point for a service that requires privileges to run. The only input parameter of an SVC is its number (ASM instruction: SVC #N), which we associate with a function call (callback). Unlike other exceptions triggered via software available, like PendSV (Pendable Supervisor Call), the SVC can be triggered in user mode by default.

5. Design

5.1 Using two stack pointers

To use the two available stack pointers (MSP and PSP) it is essential to understand 2 things:

• The control register manipulation: it is only possible to write or read the CONTROL register in handler mode (within an exception handler) or in privileged threads.
• The exceptions mechanism: when an interrupt takes place, the processor saves the contents of registers R0-R3, LR, PC and xPSR, as explained in the previous publication. The value of the LR when we enter an exception indicates the mode the processor was running, when the thread was interrupted. We can manipulate this value of LR together with the manipulation of the stack pointer to control the program flow.

5.1.1. One kernel stack for each user stack

Each user stack will have a correspondent kernel stack (one kernel stack per thread). Thus, each Task is associated to a kernel stack and a user stack. Another approach would be only one kernel stack for the entire the system (one kernel stack per processor). The advantage of using the first approach is that from the point of view of who implements the system, the programs that run in the kernel follow the same development pattern as the application programs. The advantage of the second approach is less memory overhead and less latency in context switching.

5.2 Kernel entry and exit mechanisms

In the previous publication, the interruption triggered by SysTick handled the context switching, i.e., it interrupted the running thread, saved its stackframe, searched for the next thread pointed to by the next field in the TCB (thread control block) structure and resumed it.

With the separation between user and supervisor spaces, we will have two mechanisms to get in and out the kernel, the system calls, explicitly called in code, and the interruption by SysTick that implements the scheduling routine. Although still using a round-robin scheme in which each task has the same time slice, the kernel threads also work cooperatively with user threads, that is: when there is nothing more to be done, the kernel can explicitly return. If the kernel thread takes longer than the time between one tick and another, it will be interrupted and rescheduled. User tasks could also use a similar mechanism, however, for simplicity of exposure, I chose to leave user tasks only in a fixed round-robin scheme, with no cooperative mechanisms.

5.2.1. Scheduler

The flowchart of the preemptive scheduler to be implemented is in Figure 4. The start-up of the kernel and user application is also shown for clarity. The kernel starts upo and voluntarily triggers the first user task. At every SysTick interruption, the thread has its state saved and the next scheduled task is resumed according to the state in which it was interrupted: kernel or user mode.

5.2.2 System Calls

System Calls are needed when the user requests access to a privileged service. In addition, I also use the same mechanism for a kernel thread to cooperatively return to the user thread.

6. Implementation

Below I explain the codes created to implement the proof of concept. Most of the kernel itself is written in assembly, except for a portion of the supervisor calls handler that is written in C with some inline assembly. In my opinion, more cumbersome and susceptible to errors than writing in assembly is to embed assembly in C code. The toolchain used is the GNU ARM.

6.1. Stacks

There is nothing special here, except that now in addition to the user stack declare another array of integers for the kernel stack. These will be associated in the Thread Control Block.

int32_t p_stacks[NTHREADS][STACK_SIZE]; // user stack
int32_t k_stacks[NTHREADS][STACK_SIZE]; // kernel stack

6.2. Task Scheduler

The main difference from this to the scheduler shown in the last publication is that we will now handle two different stack pointers: MSP and PSP. Thus, when entering an exception handler, the portion of the stackframe saved automatically depends on the stack pointer used when the exception took place. However, in the exception routine, the active stack pointer is always the MSP. Thus, in order to be able to handle a stack pointer when we are operating with another, we cannot use the PUSH and POP pseudo-instructions because they have the active stack pointer as their base address . We will have to replace them with the instructions LDMIA (load multiple and increment after) for POP, and STMDB (store multiple decrement before) for PUSH, with the writeback sign “!” at the base address [1] .

// Example of POP 
 MRS R12, PSP // reads the value of the process stack pointer in R12 
LDMIA R12!, {R4-R11} // R12 contains the base address (PSP) 
/ * the address contained in R12 now stores the value from R4; [R12] + 0x4  
 contains the value of R5, and so on until [R12] + 0x20 contains the 
 value of R11.
 the initial value of R12 is also increased by 32 bytes 
* /
MSR PSP, R12 // PSP is updated to the new value of R12

// Example of PUSH
 MSR R12, MSP 
STMDB R12!, {R4-R11}  
/ * [R12] - 0x20 contains R4, [R12] - 0x16 contains R5, ..., [R12] contains R4
 the initial value of R12 is decremented by 32 bytes * /
 MSR MSP, R12  // MSP is updated to the new value of R12 

Another difference is that the TCB structure now needs to contain a pointer to each of the stack pointers of the thread it controls, and also a flag indicating whether the task to be resumed was using MSP or PSP when it was interrupted.

// thread control block
struct tcb 
{
  int32_t*  psp; //psp saved from the last interrupted thread
  int32_t*  ksp; //ksp saved from the last interrupted kernel thread
  struct tcb    *next; //points to next tcb
  int32_t   pid; //task id
  int32_t   kernel_flag; // 0=kernel, 1=user    
};
typedef struct tcb tcb_t;
tcb_t tcb[NTHREADS]; //tcb array
tcb_t* RunPtr; 

The scheduler routines are shown below. The code was written so it is clear in its intention, without trying to save instructions. Note that in line 5 the value of LR at the entry of the exception is only compared with 0xFFFFFFFD, if false it is assumed that it is 0xFFFFFFFF9, this is because I guarantee that there will be no nested interrupts (SysTick never interrupts an SVC, for example), so the LR should never assume 0xFFFFFFF1. If other than a proof of concept, the test should be considered.

.global SysTick_Handler
.type SysTick_Handler, %function
SysTick_Handler:            
CMP LR, #0xFFFFFFFD // were we at an user thread?
BEQ SaveUserCtxt    //yes
B   SaveKernelCtxt  //no
 
SaveKernelCtxt:
PUSH	{R4-R11}  //push R4-R11
LDR		R0,=RunPtr
LDR		R1, [R0]
LDR		R2, [R1,#4]
LDR		R3, =#0  
STR		R3, [R1, #16] //kernel flag = 0
STR		SP, [R2]
B		Schedule

SaveUserCtxt:
MRS		R12, PSP
STMDB	R12!, {R4-R11}
MSR		PSP, R12
LDR		R0,=RunPtr
LDR		R1, [R0]
LDR		R3, =#1  
STR		R3, [R1, #16] //kernel flag = 1
STR		R12, [R1]		
B		Schedule
 
Schedule:
LDR R1, =RunPtr //R1 <- RunPtr
LDR R2, [R1]    
LDR R2, [R2,#8] //R2 <- RunPtr.next
STR R2, [R1]    //updates RunPtr
LDR R0, =RunPtr
LDR R1, [R0]
LDR R2, [R1,#16]
CMP R2, #1       //kernel_flag==1?
BEQ ResumeUser   //yes, resume user thread
B   ResumeKernel //no, resume kernel thread
 
ResumeUser:
LDR		R1, =RunPtr			//R1 <- RunPtr
LDR		R2, [R1]
LDR		R2, [R2]
LDMIA	R2!, {R4-R11}		//restore sw stackframe 
MSR		PSP, R2				//PSP4
MOV		LR, #0xFFFFFFFD		//LR=return to user thread
MOV		R0, #0x03
MSR		CONTROL, R0
ISB
BX		LR 
	
ResumeKernel:
LDR		R1, =RunPtr			//R1 <- RunPtr updated
LDR		R2, [R1]
LDR		R2, [R2, #4]
LDR		SP, [R2]
POP		{R4-R11}
MOV		LR, 0xFFFFFFF9
MOV		R0, #0x00
MSR		CONTROL, R0
ISB
BX		LR

6.3 System Calls

The implementation of system calls uses the SVC Handler. As stated, SVC has a unique input parameter (ARM makes it sounds like an advantage…), that is the number we associate with a callback. But then how do we pass the arguments forward to the callback, if system calls can handle only one parameter? They need to be retrieved from the stack. The AAPCS (ARM Application Procedure Call Standard) which is followed by compilers, says that when a function (caller) calls another function (callee), the callee expects its arguments to be in R0-R3. Likewise, the caller expects the callee return value to be in R0. R4-R11 must be preserved between calls. R12 is the scratch register and can be freely used.

No wonder that when an exception takes place the core saves (PUSH) the registers R0-R3, LR, PC and xPSR from the interrupted function, and when returning put them (POP) again in the core registers. It is fully prepared to get back to the same point when it was interrupted. But if we change the context, that is, after the interruption we do not return to the same point we were before, there will be a need to explicitly save the remaining stackframe so this thread can be resumed properly later. It is essential to follow the AAPCS if we want to evoke functions written in assembly from C code and vice-versa.

To system calls, I defined a macro function in C that receives the SVC code and the arguments for the callback (the syntax of inline assembly depends on the compiler used). Beware you need to tell the compiler R0 is a clobbered register so it will not store another value in it.

/*
asm ( assembler template
: output operands
: input operands
: list of clobbered registers
)*/

#define SysCall(svc_number, args) {\                                        
	asm volatile ("MOV R0, %0 " \
	: \
	: "r"  (args) \
	: "%r0");     \
	asm volatile ("svc %[immediate]" \
	: \
	:[immediate] "I" (svc_number) \
	: );  	  	  \
}

The args value is stored in R0. The SVC call is made with the immediate “svc_number”. When the SVC is triggered, R0-R3 will be automatically saved to the stack. The code was written as follows, without saving instructions, for clarity:

SVC_Handler:
MRS R12, PSP		 //saves psp
CMP LR, #0xFFFFFFFD
BEQ KernelEntry
B	 KernelExit
KernelEntry: 
MRS		R3, PSP
STMDB	R3!, {R4-R11}
MSR		PSP, R3
LDR		R1,=RunPtr
LDR		R2, [R1]
STR		R3, [R2]	
MOV		R0, R12 
B      svchandler_main

KernelExit:
// save kernel context
MRS		R12, MSP 
STMDB	R12!, {R4-R11}  //push R4-R11
MSR		MSP, R12
LDR		R0,=RunPtr
LDR		R1, [R0]
LDR		R2, [R1,#4]
STR		R12, [R2]		
// load user context
LDR		R2, [R1]
LDMIA	R2!, {R4-R11}
MOV		LR, #0xFFFFFFFD
MSR		PSP, R2
MOV		R0, #0x03
MSR		CONTROL, R0
ISB
BX		LR 

The rest of the routine for entering the kernel is written in C [2, 3]. Note that in the routine written in assembly a simple branch occurs (line 14) and therefore we have not yet returned from the exception handler .

The svc_number, in turn, is retrieved by walking two bytes (hence the cast to char) out of the address of the PC that is 6 positions above R0 in the stack [1, 2, 3]. Note that it was necessary to assign to R0 the value contained in PSP shortly after entering the interrupt, before saving the rest of the stack (lines 2 and 13 of the assembly code).

After retrieving the system call number and its arguments, the MSP is overwritten with the value stored in the TCB. Then we change the value of LR so the exception returns to the base mode. In this implementation the callback does not run within the handler. When the BX LR instruction is executed, the remaining of the stackframe is automatically activated onto the core registers.

#define SysCall_GPIO_Toggle  1 //svc number for gpio toggle
#define SysCall_Uart_PrintLn 2 //svc number for uart print line
 
void svchandler_main(uint32_t * svc_args)
{       
    uint32_t svc_number;
    uint32_t svc_arg0;
    uint32_t svc_arg1;
    svc_number = ((char *) svc_args[6])[-2]; // recupera o imediato 
    svc_arg0 = svc_args[0];
    svc_arg1 = svc_args[1]; 
  
 switch(svc_number)
 {
 case SysCall_GPIO_Toggle: 
    k_stacks[RunPtr->pid][STACK_SIZE-2] = (int32_t)SysCallGPIO_Toggle_; //PC
    k_stacks[RunPtr->pid][STACK_SIZE-8] = (int32_t)svc_arg0; //R0
    k_stacks[RunPtr->pid][STACK_SIZE-1] = (1 << 24); // T=1 (xPSR)
    __ASM volatile ("MSR MSP, %0" : : "r" (RunPtr->ksp) : );
    __ASM volatile ("POP {R4-R11}");
    __ASM volatile ("MOV LR, #0xFFFFFFF9");
	__ASM volatile ("MOV R0, #0x0"); 
	__ASM volatile ("MSR CONTROL, R0"); /*base mode*/
    __ASM volatile ("ISB");
    __ASM volatile ("BX LR"); /***returns from exception****/
    break;
 case SysCall_Uart_PrintLn: 
    k_stacks[RunPtr->pid][STACK_SIZE-2] = (int32_t)SysCallUART_PrintLn_; 
    k_stacks[RunPtr->pid][STACK_SIZE-8] = (int32_t)svc_arg0;
    k_stacks[RunPtr->pid][STACK_SIZE-1] = (1 << 24); // T=1
    __ASM volatile ("MSR MSP, %0" : : "r" (RunPtr->ksp) : );
    __ASM volatile ("POP {R4-R11}");
    __ASM volatile ("MOV LR, #0xFFFFFFF9");
	__ASM volatile ("MOV R0, #0x0"); 
	__ASM volatile ("MSR CONTROL, R0"); 
    __ASM volatile ("ISB");
    __ASM volatile ("BX LR"); /***returns from exception***/
    break;
 default:
    __ASM volatile("B SysCall_Dummy");
    break;
 break;
 }
}

A callback looks like this:

static void SysCall_CallBack_(void* args)
{
    BSP_Function((int32_t*) args); //BSP function with one argument int32
    exitKernel_(); // leaves cooperatively
}

6.4. Start-up

The start-up is a critical point. System starts in base mode. Stacks are assembled. The first task to be performed by the kernel after booting is to configure SysTick, switch to user mode and trigger the first user thread .

The assembly routines for the start–up are as follows:

.equ SYSTICK_CTRL, 0xE000E010 
.equ TIME_SLICE,    999
 
.global kStart 
.type kStart, %function
kStart:
LDR R0, =RunPtrStart
LDR R1, [R0]
LDR R2, [R1,#4]
MSR MSP, R2   // MSP <- RunPtr.ksp
POP {R4-R11}  //loads stackframe 0 at call stack
POP {R0-R3}
POP {R12}
ADD SP, SP, #4
POP {LR}     //LR <- PC = UsrAppStart
ADD SP, SP, #4
BX  LR // branches to UsrAppStart
 
//this function manages the stack to run the first user thread
.global UsrAppStart 
.type   UsrAppStart, %function
UsrAppStart:                
LDR R1, =RunPtr //R1 <- RunPtr
LDR R2, [R1]        
LDR R2, [R2]
MSR PSP, R2
BL  SysTickConf //configures systick
MOV R0, #0x3
MSR CONTROL, R0 //thread unprivileged mode
ISB         // inst set barrier: guarantees CONTROL is updated before going
POP {R4-R11}   //loads stackframe 0
POP {R0-R3}
POP {R12}
ADD SP, SP, #4
POP {LR}       //LR <- PC
ADD SP, SP, #4
BX LR
     
SysTickConf:
LDR R0, =SYSTICK_CTRL 
MOV R1, #0
STR R1, [R0]  // resets counter
LDR R1, =TIME_SLICE  
STR R1, [R0,#4] // RELOAD <- TIME_SLICE
STR R1, [R0,#8] // CURR_VALUE <- TIME_SLICE
MOV R1, #0x7   // 0b111:
            // 1: Clock source = core clock 
            // 1: Enables irq
            // 1: Enables counter
STR R1, [R0]        
BX  LR      //get back to caller

7. Test

As a small test, we will write on the PC screen via UART. The callback for the system call was written as follows:

static void SysCallUART_PrintLn_(const char* args)
{
    __disable_irq(); //guarded begin
    uart_write_line(UART, args);        
// waits until transmission is done
    while (uart_get_status(UART) != UART_SR_TXRDY); 
    __enable_irq(); // guarded end
    exitKernel_(); // exit kernel cooperatively
}

The tasks (main threads) look like this:

void Task1(void* args)
{
    const char* string = (char*)args;
    while(1)
    {
        SysCall(SysCall_Uart_PrintLn, string);
    }
}

Be careful when using multitasking to access any shared resource, since we have not yet inserted any inter-process communication mechanism. However, the operation is within a “Guarded Region”, and it will not be interrupted by SysTick. The main program is as follows:

#include <commondefs.h> //board support package, std libs, etc.
#include <kernel.h>  
#include <tasks.h>
 
int main(void)
{
  kHardwareInit(); 
  kAddThreads(Task1, (void*)"Task1\nr", Task2, (void*)"Task2\nr", Task3, (void*)"Task3\nr");
  RunPtrStart = &tcbs[0]; 
  RunPtr = &tcbs[1];
  uart_write_line(UART, "Inicializando kernel...\nr");
  kStart(); 
  while(1);
}

System running:

8. Conclusions

The use of two stack pointers, one for application and another for the kernel isolate these spaces not allowing the application to corrupt the kernel stack. Privilege levels prevent the user from overwriting special registers, keeping the kernel safe from application programming errors or malicious code.

Adding another stack pointer to the system required changes to the scheduling routine because we now manipulate two stacks in the domain of two different stack pointers, and both can be preempted. In addition, a cooperative mechanism has also been added for kernel exiting.

The one kernel stack per user stack approach makes the development of kernel or application routines to follow the same pattern from the perspective of who is writing the system. The price to pay is memory overhead and more latency when switching contexts. To mitigate the last, cooperative mechanisms can be added as shown. To mitigate the memory overhead more attention should be put when modeling the tasks (or concurrent units), so they are efficiently allocated.

The system call mechanism is used as an entry point to hardware services, or whatever else we deem critical for the security and stability of the system. This will make even more sense by separating not only the stacks at privilege levels but also the memory regions with the MPU .

Next publication

9. References

[1] http://infocenter.arm.com/help/topic/com.arm.doc.ddi0337e/DDI0337E_cortex_m3_r1p1_trm.pdf

[2] The definitive Guide to ARM Cortex M3, Joseph Yiu

[3] https://developer.arm.com/docs/dui0471/j/handling-processor-exceptions/svc-handlers-in-c-and-assembly-language

Author: Antonio Giacomelli de Oliveira

Embedded Systems Engineer View all posts by Antonio Giacomelli de Oliveira