%This the latex file for lecture 4.  Please send corrections to
%swu@athena.mit.edu and sikora@athena.mit.edu.


%------------------------------------------------------
% to get hardcopy of this lecture, you need the following files (plus the
% normal latex and tex base directories):
%     lecture.tex:     latex source file
%     macros.tex:      figure macros
%     psfig.tex:       postscript figure (psfig) macro definition
%     sort.idraw:      first figure, created with idraw
%     security.idraw:  second figure, created with idraw
%     proof.idraw:     third figure, created with idraw
%     pocs-header.sty: Principles of Computer Systems lecture latex header
%
% run the following programs:
% latex lecture
%   -> Creates lecture.dvi, plus latex intermediate files
% latex lecture
%   -> Run a second time to get cross-references right
% dvi2ps lecture > lecture.ps
%   -> Merges the figures with the text, can also pipe to lpr
%
\documentstyle[12pt,pocs-header]{article}
\Scribes{Suwei Wu and Scott Sikora }
\Lecturer{Butler Lampson}
\LectureNumber{4}
\LectureDate{September 24, 1990}
\include{macros} % used to include figures in this document
\begin{document}
\MakeScribeTop

\section{Administrata}

\begin{table}[h]
\centering
\begin{tabular}{|c|l|} \hline
\multicolumn{1}{|c|}{Handout} &
\multicolumn{1}{c|}{Title} \\ \hline
9 & Problem Set \#2 \\ 10 & Simple File System Implementation \\ Notes
& Lecture Notes \#2 \\ \hline
\end{tabular}
\end{table}

\begin{itemize}
\item Correction for PS \#2: The handout is incorrectly labeled Problem Set
\#4, and has an incorrect due date.  The correct due date is October 10,
1990.
\item The scribe style sheet can be found on mintaka.lcs.mit.edu \\
/common/pub/6.826/pocs\_header.sty
\end{itemize}

\section{Overview}

The following specifics of a file system implementation were covered:

\begin{itemize}
\item cached and buffered disks
\item files
\item allocation
\item directories
\item crashes
\end{itemize}

Note, the reference material for this lecture can be found in Handout
10, Simple File System Implementation.

\section{Cached and Buffered Disks}

When implementing the disk specification, there are two major issues
to be considered: 
\begin{enumerate} 
\item the slowness of disk access, and 
\item the advantage of improved time with sequential operations. 
\end{enumerate}
Therefore, a cached and write buffered implementation of the disk
abstraction is utilized.

The basic caching technique is similar to the write back cache
outlined in Handout 4, Example Memory Systems.  In this case, the
cache is simply a limited map from disk block addresses to disk
blocks.  

***The write queue is a list of writes which can be performed all at
the same time.  This offers the advantages of increased performance if
the writes are sequential, and the option to (have separate processors
to perform the writes concurrently) implement the process concurrently
in the background.  The queue (is implemented as )itself is a sequence
of disk addresses, while the disk blocks exist in the cache.  Note,
the disk addresses must be kept in the queue in the same order as they
were written, in case of a failure.  This has the advantage of
allowing a stable disk state to be recovered after a crash, but does
not allow clever optimizations (such as sequential writes) that would
greatly increase disk performance.

This implementation resembles the Unix file operating system, 
which operates as a single cache and then performs all of the writes 
once everything is done.  Therefore disk operations are not atomic. 
(Except for certain special file operations such as rename.)

\section{Files}

There is one main issue in file implementation -- maintaining the file
so that it is contiguous, thereby keeping the performance of large
reads and writes as fast as possible.  Note also that the
implementation must keep track of allocation and free space.
(something about efficient mapping by taking advantage of sequential
blocks to reduce the number of seeks). This will be discussed in the
next section.

The implementation strategy uses extents: pairs composed of the disk
address and (size) an integer.  Extents act as references to
contiguous sequences of disk blocks.

Ideally a file is represented by just one extent (which requires only
one seek to read or write the file) since then can be written and read
in one large gulp.  In fact, in some older systems this was actually a
requirement.  However, there are several disadvantages:
\begin{enumerate}
\item files increase and decrease in size and
\item files are deleted and added.
\end{enumerate}

Therefore, due to fragmentation, using a scheme of single extents can
often lead to allocation problems where there exist no blocks large
enough to hold the file and yet there exist plenty of space on the
disk.

Instead, multiple extents are used to solve the allocation problems
described above. When using multiple extents, two main issues must be
considered:
\begin{enumerate}
\item How many extents should a file have?  What should be the limit?
\item How to keep track of the extents?
\end{enumerate}

\subsection{Sequences}
One simple representation scheme is to use a sequence of disk
addresses for each block on the disk.  This scheme does not put any
restrictions on how the disk blocks should be organized.  However, it
creates the problem of where to store the sequences of addresses,
since it could become larger than one disk block.  One solution is to
use another sequence of extents, gradually reducing the size
until the reference sequence fits into one disk block. 

\subsection{Chaining}
Another representation scheme involves chaining the blocks together by
keeping the disk address of the next block at the end of the previous
block.  Random access performance, however, is poor requiring reads of
each successive block until the desired block is found.

However, this has the disadvantage of having to read in all of
the blocks in order to access just one of them.  Thus, random access
performance suffers terribly.

\subsection{Trees}

Yet another implementation scheme is to use a tree of extents.
However, a tree structure in the worst case may resemble a list and
cause random access performance to suffer.  The solution is to use
balanced tree structures such as B-tree.

\section{Allocation}

Whenever a file is extended or truncated or is deleted or added, the
freespace on the disk is either increased or decreased.  Freespace is
defined as all of the blocks that are not reachable from the
directory.  

Note that there two types of invariants that can be used:
\begin{itemize}
\item the stronger invariant: free $<=>$ \~ reachable and
\item the weaker invariant: free $=>$ \~ reachable. 

\subsection{Bit Table}
One representation scheme to manage free space is to use a bit table
-- on bit for each block on the disk.  Each bit is a boolean
indicating whether the block is free or allocated.  One advantage of
this scheme is that he table is of a fixed size depending on the disk
size.  However, this strategy proves inefficient, if only a few blocks
are free since than most of the space allocated for the table is
unused.

\subsection{Extents}

Another scheme involves representing free space with extents, using
the less stringent rep invariant.  The main disadvantage is that disk
blocks can be unaccountable.  However, this can be dealt with by
rescanning the disk at appropriate times.  Also, atomic operations are
hard to perform since two updates must be made: (1) to change the
extents and (2) to possibly change the free function.  Note, when
updating the file, the free function must be modified first and then
the file.  Otherwise after a crash, blocks may be allocated to more
than one file.

The trick in this scheme is again to use B trees while keeping the
structure balanced and the structure in proper shape with atomic
updates.  The lesson to be learned here is to look carefully at the
abstraction function since it is not always necessary to maintain the
strongest invariant possible.

\section{Directories}

Directories are simply functions from pathnames to files (PN $->$ F).
One way to implement this is with {\em Encode} and {\em Decode}
functions where encode is a mapping from the internal to the external
representation (often a sequence of bytes that has to go out over a
wire), and decode is a mapping back from the external to internal
representation.  Note that t = {\em Decode}({\em Encode}(t)) for all t.

Some possible examples:
\begin{enumerate}
\item Internal: a list of real numbers \\
      External: a list containing the size of the numbers and then the
                numbers themselves.  	
\item Internal: a tree containing a node and two leaves	\\
      External: represent as an S-expression: 1(2 3) 
\item Internal: a network 		
      External: an S-expression using UID's for nodes that have
		already been defined.
\end{enumerate}

One scheme for a directory is to use an external representation
composed of a list of pairs of pathnames and disk addresses.  However,
this requires sequential scanning of possibly the entire directory to
locate a particular pair.  

One solution to the search problem is to sort the pairs and use a
binary search.  Another solution is to use B-trees which were in
originally devised for just this purpose.  However, in this case, we
must carefully examine atomicity to ensure that writes leave the
directory in a known state.

\section{Crashes}

While it is possible to carefully design a system in order to be 
consistent across crashes, it can become very difficult with large 
systems.  Therefore it is advantageous to design a general purpose 
way to handle crashes and avoid an inefficient design loop. 

The general scheme is to build large atomic operations from little
atomic operations called transactions.  The transaction scheme
requires three types: values, states, and actions.  In addition, there
are three main operations: {\em Crash}, {\em commit}, and {\em Do} and
two possible states: a stable state and a volatile state.  Basically,
the volatile state is updated with new actions.  When the sequence of
actions is complete, the {\em Commit} copies the volatile state into
the stable state, thus making the actions atomic.  If a crash occurs,
{\em Crash} is used to copy the last stable state into the volatile
state.

Transactions are implemented with the use of logs.  There are two
types of logs needed: a volatile log and a stable log.  A log is
basically a sequence of updates (atomic operations).  {\em Commit}
simply copies the volatile log to the stable log, erases the volatile
log and applies the stable log updates to the stable state.  Note, in
order for this operation to work, the logs must have the property that
s+l+l = s+l, since a crash may occur during recovery.  For example,
simple disk writes are one operation that has this property.

\end{document}