% to get hardcopy of this lecture, you need the following files (plus the
% normal latex and tex base directories):
%     lecture.tex:     latex source file
%     macros.tex:      figure macros
%     psfig.tex:       postscript figure (psfig) macro definition
%     pocs-header.sty: Principles of Computer Systems lecture latex header
%
% run the following programs:
% latex lecture
%   -> Creates lecture.dvi, plus latex intermediate files
% latex lecture
%   -> Run a second time to get cross-references right
% dvi2ps lecture |  egrep -v '^showpage$' > lecture.ps
%   -> Merges the figures with the text, can also pipe to lpr
%
\documentstyle[12pt,pocs-header]{article}
\Scribe{Henry Minsky}
\Lecturer{Butler Lampson}
\LectureNumber{11}
\LectureDate{October 21, 1991}
\include{macros} % used to include figures in this document
\begin{document}
\MakeScribeTop

\section{Handouts and Announcements}

\begin{itemize}
\item Handout 25: ``Concurrent Disk Reads'' by Butler Lampson. 

\item Transcriptions of lectures 7 and 8.
\end{itemize}

\section{Examples Of Concurrency }

This lecture covers some moderately complex examples of concurrency,
and examines some of the issues raised. Some examples are given of the
programming techniques used to manage these issues. 

Handout 25 {\it Concurrent Disk Reads} gives an example of a partial
implementation of a cached Disk module, with mechanisms to manage
multiple concurrent processes which need access to the disk.

The implementation demonstrates a basic techinque, which consists of
breaking down the problem as follows:


\begin{itemize}
\item {\it What data is shared?} --- Identify separable shared blocks
of data.

\item {\it Create one lock per data block} --- Protect each block
of data with its own lock, implemented with Mutex. A process must
acquire the lock before altering any of the data.

\item {\it Maintain an invariant outside of the lock } --- Protected
data must be left in a consistent state when a process leaves a
critical section.

\end{itemize}

Figure~\ref{tfig1} shows two threads, {\bf T1} and {\bf T2}, both
executing code which wants to access a shared data block, protected by
lock $l$. Each thread must first acquire the lock, using {\it l.acq}
before altering the data. The code between the acquisition of the lock
and its release is called a {\it critical section}. The mutual exclusion
property of the mutex ensures that only one thread at a time has access
to its critical section.


\begin{figure}
\centerline{\psfig{height=2.5in,figure=tfig1.ps}}
\caption{Two threads attempting to access the same data}
\label{tfig1}
\end{figure}


It is important to realize that in general, data which has been cached
inside a critical section (such as a copy of a piece of shared data,
assigned to a local variable) will not be valid outside of the
critical section, unless some invariant is maintained on the data.

\section{Invariants For Shared Data}

It is possible to show by induction that an invariant $I(d)$ on a
piece of shared data is preserved, if you maintain a couple of
conditions:

\begin{itemize} 

\item The invariant $I(d)$ on the  protected data is true at
initialization.

\item The invariant is maintained when its lock is released.

\end{itemize} 

As long as you leave the data obeying the invariant when you exit the
critical section, and don't touch it outside of the critical section,
it will still obey the invariant the next time a process acquires the
lock.

\subsection{Fine-Grain Concurrency and Deadlocks}

Having all the data in a program protected by one lock can create a
bottleneck in the processing. In many cases there are logically
independent regions of data, in the sense that operations can be
performed concurrently on the regions by different processes, without
interference between them.

Thus, it is often desirable to break the data up into several
independent shared blocks, each protected by its own lock.  This
introduces the possibility of {\it deadlock}.  Figure
\ref{tfig2} shows a potential problem with two threads, each trying to
hold two locks. A situation has arisen where neither process can
advance, because they are both blocking on a resource which the other
owns.

\begin{figure}
\centerline{\psfig{height=2.5in,figure=tfig2.ps}}
\caption{Two threads, each attempting to acquire locks {\it l1} and
{\it l2} }
\label{tfig2}
\end{figure}


A general solution to this problem is to put a partial order on all
locks in the system, and require that for each process, ``to acquire
lock $l$, it must be the case that all locks already held are
less-than $l$''.

\section{Concurrent Disk Example}

The {\tt ConcurrentDisk} module of Handout 25 can be asked to perform
several types of operations, ordered here by expense in terms of
latency and possibly CPU usage:

\begin{itemize} 

\item {\it cache lookup} = cheap

\item {\it block-copy} = medium

\item {\it disk access} = expensive 

\end{itemize}

What benefit would be gained here by using concurrency? Consider the
cases of cache-lookup and block-copy, and a single CPU system. If we
only care about CPU utlization, then there is no overall benefit to be
gained in making these operations concurrent. If a certain number of
CPU operations must be done for each task, then time-slicing the tasks
does not get the whole job done any faster.

If we have two or more CPUs, or a CPU and disk drives, then it is
possible to make real gains in performance, if several tasks can
actually be executed simultaneously. In general, if we have several
engines of computation which are capable of making progress
simultaneously, then concurrency will be useful.

Actually, even in the single CPU case, if what we care about is the
real-time response of the system, then it is beneficial to simulate
running operations concurrently, using a scheduler. 
The responsiveness of a system is an important factor for human users\footnote{Birrell ({\it An
Introduction to Programming with Threads}) points out that this is
particularly true in interactive applications, such as window systems.}.


A task can be made more concurrent by separating out subtasks, each of
which can have its own lock.  In both the single and multiple CPU
examples, we find that when there is a need to distribute load more
fairly, either in time or space, then it is time to look at finer
grain locking strategies. This can result in

\begin{enumerate}

\item better response time, and

\item keeping mutliple resources busy (if you have them)
\end{enumerate}


\begin{figure}
\centerline{\psfig{height=3.5in,figure=cdisk1.ps}}
\caption{A concurrent cached disk module}
\label{cdisk1}
\end{figure}

\subsection{Cached Concurrent Disk}

We want the implementation of the {\tt ConcurrentDisk} module to
support multiple threads reading and writing simultaneously. This
could be accomplished by serializing all of the operations, which
would basically be the {\tt BufferedDisk} implementation, by making
all operations atomic. But this makes threads wishing to access
independent cached disk blocks wait needlessly for slow disk
operations or block-copies to complete.

The {\tt ConcurrentDisk} implementation uses a combination of multiple
locks for data blocks in the cache, and a single monolithic lock for
the relatively brief bookeeping operations on the cache data-structure
itself.

Figure~\ref{cdisk2} shows the relation between locks and cache blocks
in the implementation. The {\tt i} field in a cache block counts
outstanding references to the block, and is protected by {\tt mc}, the
master mutex for the cache. The {\tt m} field is a mutex which
protects the the data.


\begin{figure}
\centerline{\psfig{height=3.5in,figure=cdisk2.ps}}
\caption{Locking for the ConcurrentDisk module}
\label{cdisk2}
\end{figure}

The invariant on the cache blocks is that when the mutex {\tt m} is
{\em not} held by anyone, then the data in {\tt db}, the cached disk
block, represents the true data for that block. As long as the
invariant is maintained when a thread releases the lock (
{\tt b\^.m.rel}), then it is maintained when the lock is next acquired.


The {\tt ReadBlocks} procedure is constructed to maintain these
invariants, and to manage the allocation of storage in the cache. The
purpose of the {\tt i} field in the cache block is to keep track of
how many threads are accessing the data in that block. A cache block
{\tt b} can only be flushed if that block has no readers (i.e., 
{\tt b\^.i $ = 0$}).

A call to {\tt ReadBlocks(e)} works by looping to find if the desired
data-blocks in the extent are present in the cache. For each block
which is present, the number-of-readers counter, {\tt i}, is
incremented. The global cache lock must be held during this operation,
or else incorrect values of {\tt i} could result.  For efficiency, if
some of the desired blocks are not present, the algorithm tries to
find the longest contiguous string of available Disk Addresses in the
cache to fill at a time, so as to do as few physical disk-accesses as
possible.  The global lock can be released now, since the available
cache blocks have been effectively ``wired down'' by making their {\tt
i} field non-zero.  All of the data from cache hits are then copied to
the local variable {\tt data}, and the cache replacement algorithm is
run.  This copy operation requires grabbing the lock for each data
block sequentially, but only one of these need be held at a time; a
big gain in concurrency over using a single lock for the entire cache.

The cache replacement algorithm now tries to allocate enough free
buffers to hold the data it is about to read from disk. When these
buffers have been allocated, (i.e., locked), the {\tt Disk.ReadBlocks}
operation reads the data from the disk, and copies it into the
allocated buffers.




\section{Concurrency And Filesystems}

The filesystem needs to maintain a datastructure which maps pathnames
to files. In the face of concurrency, it is possible to ensure correct
behavior, by making all filesystem operations atomic. This is a poor
solution for several reasons. If the entire filesystem must be grabbed
exclusively by a process wishing to do any filesystem operation, then
there will be a lot of processes busy waiting, even if they want to
perform totally independent operations. Also, it is hard to make some
operations, such as large writes, atomic. This is a case where finer
grained locking is a good idea.

Consider the model of a filesystem directory structure as a DAG.
Lookup of pathnames is done by traversal of the directory tree from
the root.

Imagine two processes running concurrently, one performing a {\tt
lookup(/b/z)} and another performing {\tt rename(/b)}. With no locking
at all, the lookup is liable to return a wrong result. Clearly some
sort of locking is needed.

One idea is to lock subtrees. But this doesn't help, since pathnames
are always relative to the root. We end up locking the directory tree
from the root down, which is amounts to the same thing as a global
lock on the filesystem.

One good solution is two introduce two kinds of locks; read-locks and
write-locks. A read-lock allows multiple readers for a block, whereas
a write-lock allows one writer and no readers for a block. 


Read/write locks allow concurrent operations, but the use of symbolic
links can create deadlocks, as shown in Figure~\ref{files}. It is
possible to do something akin to the partial ordering system for locks
to anticipate deadlocks. Algorithmic detection (a topological sort for
example) is possible, but in practice is probably expensive and
complex. A simpler idea is to detect deadlocks using a timeout.  A
scheme similar to the exponential backoff schemes used in Ethernet
CSMA with random timers will allow deadlocks to be broken with high
probablility..



\begin{figure}
\centerline{\psfig{height=1.5in,figure=files.ps}}
\caption{Deadlock in a directory tree with symbolic links}
\label{files}
\end{figure}


\end{document}
