% to get hardcopy of this lecture, you need the following files (plus the
% normal latex and tex base directories):
%     lecture.tex:     latex source file
%     macros.tex:      figure macros
%     psfig.tex:       postscript figure (psfig) macro definition
%     pocs-header.sty: Principles of Computer Systems lecture latex header
%
% run the following programs:
% latex lecture
%   -> Creates lecture.dvi, plus latex intermediate files
% latex lecture
%   -> Run a second time to get cross-references right
% dvi2ps lecture |  egrep -v '^showpage$' > lecture.ps
%   -> Merges the figures with the text, can also pipe to lpr
%
\documentstyle[12pt,pocs-header]{article}
\Scribe{Henry Minsky}
\Lecturer{Bill Weihl}
\LectureNumber{25}
\LectureDate{December 11, 1991}
\include{macros} % used to include figures in this document
\begin{document}
\MakeScribeTop

\section{Handouts and Announcements}

\begin{itemize}
\item Solutions to problem set 7.
\item Transcriptions of lectures 2, 20, 19, and 24.
\end{itemize}

\section{Cache Coherence}

Some of the techniques for improving performance using cache-coherence
in shared-memory multiprocessors were covered, as described in Handout
46 {Cache Coherence Protocols: Evaluation Using a Multiprocessor
Simulation Model}.  First, the {\tt WRITE-ONCE} protocol was
mentioned. This is a protocol with four states, {\tt INVALID, VALID,
RESERVED, DIRTY}. The state diagram is shown in Section 2.1 in the
handout.

\subsection{The Illinois Protocol}

 The Illinois protocol was discussed next. The states defined for this
protocol are

\begin{itemize} 

\item INVALID

\item VALID-EXCLUSIVE {\it  (clean, only copy)}

\item SHARED {\it (clean, possibly other copies)}

\item DIRTY {\it (modified, only copy) }

\end{itemize}

This protocol has the feature that missed blocks always come from
other caches, if any copies are cached, and from memory if no cache
has a copy. The protocol assumes there is a mechanism for the
requesting cache to determine whether or not the requested block is
shared. Blocks are written back at replacement time if they are DIRTY.

The protocol scheme works as follows

\begin{enumerate}

\item Read miss: If any other cache owns the block, it supplies it on the
bus. There must be some sort of arbitration for which cache supplies
the data.  Otherwise, memory supplies the data. If the supplied block
is from a cache, and DIRTY, it is also written to main memory.

If the supplied block was shared by another cache, all caches who own
the block will notice the read request, and set their blocks to SHARED
also.  If the block was from memory, it is set to the VALID-EXCLUSIVE
state.

\item
Write hit: If the block written to is DIRTY, it can be written to with
no delay. If it is VALID-EXCLUSIVE, change it to dirty. Again, no bus
traffic is needed. If the block is SHARED however, an invalidate is
sent to other caches, then the write is performed, and the block
is marked DIRTY.

\item Write miss: Like a read miss, a copy of the block comes from some cache,
if any cache has a copy. All other caches invalidate, and the block
is loaded in state DIRTY.

\end{enumerate}

\subsection{The Dragon Protocol}

The Dragon protocol, described in Section 2.6 Handout 46, allows dirty
cache blocks to be ``shared''. That is, on a write to a cache block,
other caches are updated rather than invalidated. 

The states in the Dragon protocol are

\begin{itemize}
\item VALID-EXCLUSIVE {\it only copy in caches, not modified}
\item SHARED-DIRTY  {\it dirty, possibly shared, write-back required}
\item SHARED-CLEAN {\it clean, possibly shared}
\item DIRTY {\it modified, not shared}
\end{itemize}

The protocol scheme works as follows

\begin{enumerate}

\item Read miss: If another cache has a DIRTY or SHARED-DIRTY copy,
that cache supplies the data, and sets its own block to SHARED-DIRTY.
It also raises a Shared line to inform the caches that this data is
shared. Any caches with V-E or SHARED-CLEAN copies raise the Shared
line, and set their copies to SHARED-CLEAN. The requesting cache loads
the block as SHARED-CLEAN if the Shared line is asserted, otherwise as
V-E. 

\item
Write hit: If the block written to is DIRTY, it can be written to with
no delay. If it is VALID-EXCLUSIVE, change it to dirty. Again, no bus
traffic is needed. Otherwise, a bus write must take place. The bus
write updates all caches with a copy of the block. Each such cache
will have their block set to SHARED-CLEAN. Any caches with copies will
assert the Shared line. This lets the writing cache know if it has an
exclusive copy or not. If it is exclusive (Shared line not asserted),
the block state is set to DIRTY, else SHARED-DIRTY.

\item Write miss: As with read-miss, the block is supplied from a
cache with a copy marked DIRTY or SHARED-DIRTY, otherwise from
main-memory. The block is loaded in the DIRTY or SHARED-DIRTY state,
depending on the state of the Shared line.

\end{enumerate}


\subsection{Update vs. Invalidate }
The tradeoff in bus traffic for an update protocol vs. an invalidate
protocol will depend on the actual address trace. It was mentioned
(ref?) that better performance could be obtained with a protocol that
has a hybrid behavior; it does an update for the first write, but if a
block has been updated twice without any intermediate read of the
block, it is invalidated. [This is sort of analogous to a LRU cache
eviction strategy.]


\subsection{Protocotols For Switched Networks}

Given a general point-to-point interconnection network, rather than a
broadcast bus, what are some policies for cache coherence? Several
architectures are possible. One possible configuration has processors
one one side of the network, and memories on the other.  Another
machine architecture might consist of more uniform set of
processor/memory/cache nodes, connected to a switching network. The
caching protocol involves use of the network to perform remote reads and
writes to other memories or caches. For arguments of scalability of
bandwidth, we will disallow broadcasts in the network.

For shared-memory cache coherence, a number of {\it directory}\/
protocols have been proposed. A directory is a list, for each cache
block, of the nodes who have a copy of this block. This allows updates
or invalidate messages to be sent only to those other nodes who are
known to share a given block. There are several proposed
implementation techniques:

\begin{itemize}
\item Full-Map Directory 

For each cache block, a fixed-size vector keeps track of which nodes
have copies of the block. The space complexity for this approach will
be $O(n^2)$, if the map is replicated in each copy of the cache block.

\item Limited Directories

Instead of a full map, the directory size can be fixed. Rather than a
bit-vector, a fixed size table of node addresses could be stored along
with a cache block. This leads to $O(n\log n)$ memory usage.

\item Directory Chaining

The directory entries can be chained using pointers, in the style of a
linked list or tree. This gives the ability to have arbitrarily shared
data, as in the full map. 


\item Hardware/Software

An implementation optimization would be to handle a simple protocol in
some hardware mechanism, and handle hairy special cases using traps to
software.

\end{itemize}

The issue of concurrent writes comes up here. Since we don't have
broadcast to implement atomic writes, as we did in the bus-based
architectures, we need to make sure that some sort of invariant is
maintained when writes happen. It can take a large amount of time for
a write to invalidate or update all the caches that it affects. Some
sort of locking is called for.  



\section{Coherence For Network File Servers}

Figure~\ref{fsfig} shows a sketch of a networked file server. The
issues of cache-coherence are similar for shared memory and shared
file systems. The optimization is caching in both cases. But where
should the caching be done? 

There are three places that caching could be done:

\begin{itemize}
\item 

The file could be cached in the memory at the server, and transmitted
over the network when needed.

\item The client could cache the file in its local memory

\item The client could cache the file on its local disk. This would
make sense where the network bandwidth is very expensive, e.g. on the
order of disk access times. 

\end{itemize}

The problem we are solving with caching is not so much decreasing the
access times, but taking some of the load off of the server system. In
particular, limiting the server's disk traffic. The solution used
today is to generally have files cached in memory at the server. 

The Andrew system at CMU cached whole files at the client end. This
made large file accesses bad, however. Now they have moved to caching
file blocks. 

\subsection{Fault tolerance and coherence}

The problem with distributed cached copies of files is that it is
difficult to determine when a cached block is out of date.

If a file is in a client cache, how do we know if it is still valid?
(Assuming the client didn't put a lock on the file when it was
opened.) 

\begin{itemize}
\item Validate On Open: The client can validate the file when it is opened. This
delays each file open.

\item Callback: The server should update or invalidate all clients with cached
copies when the file is changed. The disadvantage here is the server
needing
to maintain state of what is on which clients. 

\end{itemize}

Cached files with write buffering also stand to lose some data if the
client crashes without having done a write-back. (This is true of
write-buffering on the server too, of course). 

\subsection{Semantics of concurrent access}

Having multiple distributed readers on a file introduces the
possibility of inconsistent versions of the file on different clients.
Some filesystems try to ensure atomicity for reading and writing of
files.

Possible implementation techniques for providing atomicity are: 

\begin{itemize}


\item use read/write locking on files

\item disable caching for shared files; force all accesses through the
server (slow)

\end{itemize}

A problem with distributed caching is ``false sharing''. If the
granularity of a cache block is large, then updates will cause traffic
to other caches, even if the actual data is not really shared, but
just lies in the same cache block.  Thus it is worthwhile to try to
keep at most one spinlock in a cache line.

Unlike uniprocessor caches, the cache hit rate does not approach 100\%
as the size of the cache increases.  This is because invalidation
messages force data out of the cache even though it may still be in
use.  Thus, the maximum hit rate is limited by the invalidation
traffic.

\begin{figure}
\centerline{\psfig{height=2.5in,figure=fsfig.ps}}
\caption{File system client and server.}
\label{fsfig}
\end{figure}


\end{document}