% to get hardcopy of this lecture, you need the following files (plus the % normal latex and tex base directories): % lecture.tex: latex source file % macros.tex: figure macros % psfig.tex: postscript figure (psfig) macro definition % pocs-header.sty: Principles of Computer Systems lecture latex header % % run the following programs: % latex lecture % -> Creates lecture.dvi, plus latex intermediate files % latex lecture % -> Run a second time to get cross-references right % dvi2ps lecture | egrep -v '^showpage$' > lecture.ps % -> Merges the figures with the text, can also pipe to lpr % \documentstyle[12pt,pocs-header]{article} \Scribe{Henry Minsky} \Lecturer{Butler Lampson} \LectureNumber{11} \LectureDate{October 21, 1991} \include{macros} % used to include figures in this document \begin{document} \MakeScribeTop \section{Handouts and Announcements} \begin{itemize} \item Handout 25: ``Concurrent Disk Reads'' by Butler Lampson. \item Transcriptions of lectures 7 and 8. \end{itemize} \section{Examples Of Concurrency } This lecture covers some moderately complex examples of concurrency, and examines some of the issues raised. Some examples are given of the programming techniques used to manage these issues. Handout 25 {\it Concurrent Disk Reads} gives an example of a partial implementation of a cached Disk module, with mechanisms to manage multiple concurrent processes which need access to the disk. The implementation demonstrates a basic techinque, which consists of breaking down the problem as follows: \begin{itemize} \item {\it What data is shared?} --- Identify separable shared blocks of data. \item {\it Create one lock per data block} --- Protect each block of data with its own lock, implemented with Mutex. A process must acquire the lock before altering any of the data. \item {\it Maintain an invariant outside of the lock } --- Protected data must be left in a consistent state when a process leaves a critical section. \end{itemize} Figure~\ref{tfig1} shows two threads, {\bf T1} and {\bf T2}, both executing code which wants to access a shared data block, protected by lock $l$. Each thread must first acquire the lock, using {\it l.acq} before altering the data. The code between the acquisition of the lock and its release is called a {\it critical section}. The mutual exclusion property of the mutex ensures that only one thread at a time has access to its critical section. \begin{figure} \centerline{\psfig{height=2.5in,figure=tfig1.ps}} \caption{Two threads attempting to access the same data} \label{tfig1} \end{figure} It is important to realize that in general, data which has been cached inside a critical section (such as a copy of a piece of shared data, assigned to a local variable) will not be valid outside of the critical section, unless some invariant is maintained on the data. \section{Invariants For Shared Data} It is possible to show by induction that an invariant $I(d)$ on a piece of shared data is preserved, if you maintain a couple of conditions: \begin{itemize} \item The invariant $I(d)$ on the protected data is true at initialization. \item The invariant is maintained when its lock is released. \end{itemize} As long as you leave the data obeying the invariant when you exit the critical section, and don't touch it outside of the critical section, it will still obey the invariant the next time a process acquires the lock. \subsection{Fine-Grain Concurrency and Deadlocks} Having all the data in a program protected by one lock can create a bottleneck in the processing. In many cases there are logically independent regions of data, in the sense that operations can be performed concurrently on the regions by different processes, without interference between them. Thus, it is often desirable to break the data up into several independent shared blocks, each protected by its own lock. This introduces the possibility of {\it deadlock}. Figure \ref{tfig2} shows a potential problem with two threads, each trying to hold two locks. A situation has arisen where neither process can advance, because they are both blocking on a resource which the other owns. \begin{figure} \centerline{\psfig{height=2.5in,figure=tfig2.ps}} \caption{Two threads, each attempting to acquire locks {\it l1} and {\it l2} } \label{tfig2} \end{figure} A general solution to this problem is to put a partial order on all locks in the system, and require that for each process, ``to acquire lock $l$, it must be the case that all locks already held are less-than $l$''. \section{Concurrent Disk Example} The {\tt ConcurrentDisk} module of Handout 25 can be asked to perform several types of operations, ordered here by expense in terms of latency and possibly CPU usage: \begin{itemize} \item {\it cache lookup} = cheap \item {\it block-copy} = medium \item {\it disk access} = expensive \end{itemize} What benefit would be gained here by using concurrency? Consider the cases of cache-lookup and block-copy, and a single CPU system. If we only care about CPU utlization, then there is no overall benefit to be gained in making these operations concurrent. If a certain number of CPU operations must be done for each task, then time-slicing the tasks does not get the whole job done any faster. If we have two or more CPUs, or a CPU and disk drives, then it is possible to make real gains in performance, if several tasks can actually be executed simultaneously. In general, if we have several engines of computation which are capable of making progress simultaneously, then concurrency will be useful. Actually, even in the single CPU case, if what we care about is the real-time response of the system, then it is beneficial to simulate running operations concurrently, using a scheduler. The responsiveness of a system is an important factor for human users\footnote{Birrell ({\it An Introduction to Programming with Threads}) points out that this is particularly true in interactive applications, such as window systems.}. A task can be made more concurrent by separating out subtasks, each of which can have its own lock. In both the single and multiple CPU examples, we find that when there is a need to distribute load more fairly, either in time or space, then it is time to look at finer grain locking strategies. This can result in \begin{enumerate} \item better response time, and \item keeping mutliple resources busy (if you have them) \end{enumerate} \begin{figure} \centerline{\psfig{height=3.5in,figure=cdisk1.ps}} \caption{A concurrent cached disk module} \label{cdisk1} \end{figure} \subsection{Cached Concurrent Disk} We want the implementation of the {\tt ConcurrentDisk} module to support multiple threads reading and writing simultaneously. This could be accomplished by serializing all of the operations, which would basically be the {\tt BufferedDisk} implementation, by making all operations atomic. But this makes threads wishing to access independent cached disk blocks wait needlessly for slow disk operations or block-copies to complete. The {\tt ConcurrentDisk} implementation uses a combination of multiple locks for data blocks in the cache, and a single monolithic lock for the relatively brief bookeeping operations on the cache data-structure itself. Figure~\ref{cdisk2} shows the relation between locks and cache blocks in the implementation. The {\tt i} field in a cache block counts outstanding references to the block, and is protected by {\tt mc}, the master mutex for the cache. The {\tt m} field is a mutex which protects the the data. \begin{figure} \centerline{\psfig{height=3.5in,figure=cdisk2.ps}} \caption{Locking for the ConcurrentDisk module} \label{cdisk2} \end{figure} The invariant on the cache blocks is that when the mutex {\tt m} is {\em not} held by anyone, then the data in {\tt db}, the cached disk block, represents the true data for that block. As long as the invariant is maintained when a thread releases the lock ( {\tt b\^.m.rel}), then it is maintained when the lock is next acquired. The {\tt ReadBlocks} procedure is constructed to maintain these invariants, and to manage the allocation of storage in the cache. The purpose of the {\tt i} field in the cache block is to keep track of how many threads are accessing the data in that block. A cache block {\tt b} can only be flushed if that block has no readers (i.e., {\tt b\^.i $ = 0$}). A call to {\tt ReadBlocks(e)} works by looping to find if the desired data-blocks in the extent are present in the cache. For each block which is present, the number-of-readers counter, {\tt i}, is incremented. The global cache lock must be held during this operation, or else incorrect values of {\tt i} could result. For efficiency, if some of the desired blocks are not present, the algorithm tries to find the longest contiguous string of available Disk Addresses in the cache to fill at a time, so as to do as few physical disk-accesses as possible. The global lock can be released now, since the available cache blocks have been effectively ``wired down'' by making their {\tt i} field non-zero. All of the data from cache hits are then copied to the local variable {\tt data}, and the cache replacement algorithm is run. This copy operation requires grabbing the lock for each data block sequentially, but only one of these need be held at a time; a big gain in concurrency over using a single lock for the entire cache. The cache replacement algorithm now tries to allocate enough free buffers to hold the data it is about to read from disk. When these buffers have been allocated, (i.e., locked), the {\tt Disk.ReadBlocks} operation reads the data from the disk, and copies it into the allocated buffers. \section{Concurrency And Filesystems} The filesystem needs to maintain a datastructure which maps pathnames to files. In the face of concurrency, it is possible to ensure correct behavior, by making all filesystem operations atomic. This is a poor solution for several reasons. If the entire filesystem must be grabbed exclusively by a process wishing to do any filesystem operation, then there will be a lot of processes busy waiting, even if they want to perform totally independent operations. Also, it is hard to make some operations, such as large writes, atomic. This is a case where finer grained locking is a good idea. Consider the model of a filesystem directory structure as a DAG. Lookup of pathnames is done by traversal of the directory tree from the root. Imagine two processes running concurrently, one performing a {\tt lookup(/b/z)} and another performing {\tt rename(/b)}. With no locking at all, the lookup is liable to return a wrong result. Clearly some sort of locking is needed. One idea is to lock subtrees. But this doesn't help, since pathnames are always relative to the root. We end up locking the directory tree from the root down, which is amounts to the same thing as a global lock on the filesystem. One good solution is two introduce two kinds of locks; read-locks and write-locks. A read-lock allows multiple readers for a block, whereas a write-lock allows one writer and no readers for a block. Read/write locks allow concurrent operations, but the use of symbolic links can create deadlocks, as shown in Figure~\ref{files}. It is possible to do something akin to the partial ordering system for locks to anticipate deadlocks. Algorithmic detection (a topological sort for example) is possible, but in practice is probably expensive and complex. A simpler idea is to detect deadlocks using a timeout. A scheme similar to the exponential backoff schemes used in Ethernet CSMA with random timers will allow deadlocks to be broken with high probablility.. \begin{figure} \centerline{\psfig{height=1.5in,figure=files.ps}} \caption{Deadlock in a directory tree with symbolic links} \label{files} \end{figure} \end{document}