Software Architecture is one of the last understood concepts in computer science. Architecure, as generally defined, goes mostly unquestioned on our minds and is bound to the design of buildings and other physical structures. When brought in context for hardware, Computer Architecture still goes easily on most of us. However, when the subject approaches software, the concept of Software Architecture often goes too abstract on an already abstract world.
Perhaps no one has reasoned on Operating System Software Architecture as deeply as Prof. Schröder-Preikschat. On his book The Logical Design of Parallel Operating Systems and on a series of publications afterwards, he emphatically makes the point on the separation of structural and behavioral properties of operating systems, showing that the same behavior---and indeed the same source code---can be brought about on a variety of architectures. In other words, an operating system can deliver its functionality taking the software architecture that better matches the requirements of the applications running on top of it.
The most known operating system architectures are:
Library: the least structured architecture an OS can assume. OS functionality is implemented and packed as a library to be reused on demand by applications. The software architecture imposed on the application will actually also dictate the OS architecture. The didactic version of OpenEPOS you have been using until now is a good example of this architecture.
Monolithic Kernel: a monolithic-kernel operating system implements all the functionality delivered to applications in the kernel space (i.e. running in supervisor mode, isolated from applications). The main weakness of this architecture, that is, having to handle all possible OS services inside the kernel, is usually overcome by loadable modules. In this way, the monolithic kernel can come up much like a microkernel and extend its functionality on demand by loading modules as new services are requested by applications. Many contemporary OS adhere to this architecture, including Linux and BSD. The major motivation behind this architecture is performance.
Microkernel: an ideal microkernel implements the minimum OS functionality that is necessary to orderly implement the remaining functionality outside the kernel, at user space. Typically, this includes memory management, process management, inter-process communication, and I/O exporting (I/O itself is handled at user space by specific servers, not by kernel device drivers). The imposition of a system call barrier between applications, servers, and the kernel adds considerable overhead and is responsible for the low performance associated with this architecture. In order to overcome this, many microkernels incorporate additional functionality, often making them indistinguishable from monolithic kernels. This, in turn, brought about such buzz words as nanokernel and picokernel. Good examples of this architecture are Mach, Amoeba, and L4 .
Exokernel: some times taken as the leanest microkernels, exokernels are actually quite bound to MIT's Parallel and Distributed Operating Systems group. Instead of implementing the minimal functionality needed to implement OS services at user space, exokernel are limited to safely multiplex resources to be used at user space. Since exokernels expose the bare hardware and not a higher level API, porting them to multiple architectures is very hard. Reusing OS abstractions implemented for an exokernel for a given hardware platform on another platform is also hard.
Besides these architectures, EPOS features a built-in architecture that consists basically in separating OS code from application code without installing a system call interface. It is a mostly uninteresting architecture, but it is quite useful in two development scenarios:
Application debugging: even if a system is conceived as single-task, application development is likely to incur in errors that escape from application into the OS (library code). By activating a MMU and marking OS memory as protected, one can more easily catch such bugs (since they will now cause traps that can precisely report the problem when and where it happened).
OS development: sometimes developing new OS functionality on a library-based architecture can be quite convenient in terms of compilation time.
On your path to make OpenEPOS a real microkernel, the built-in architecture will be very handy. It will enable you to isolate the OS from the application without raising a trap on each small step. The techniques explored while solving the multiple, specialized heaps exercise will also be very helpful here, since isolating application and OS heaps can be achieved using them.
Implement the built-in architecture and ensure both code and data from the operating system, even dynamically allocated data, go to the highest addresses, while application's go to the lowest. Traces will be fundamental to demonstrate such an isolation, and the final mapping of OS memory as supervisor mode will confirm it.