%     to get hardcopy of this lecture, you need the following files (plus 
%     the normal latex and tex base directories):
%     lecture24.tex:   latex source file
%     macros.tex:      figure macros
%     psfig.tex:       postscript figure (psfig) macro definition
%     pocs-header.sty: Principles of Computer Systems lecture latex header
%     caches.idraw:    picture for this lecture
%
% run the following programs:
% latex lecture24
%   -> Creates lecture24.dvi, plus latex intermediate files
% latex lecture24
%   -> Run a second time to get cross-references right
% dvi2ps lecture24 > lecture24.ps
%   -> Merges the figures with the text, can also pipe to lpr
%
\documentstyle[12pt,pocs-header]{article}
\Scribes{David Waitzman}
\Lecturer{Bill Weihl}
\LectureNumber{24}
\LectureDate{December 9, 1990}
\include{macros} % used to include figures in this document

\begin{document}
\MakeScribeTop

\section{Administrivia}

\begin{table}[h]
\centering
\begin{tabular}{|c|l|} \hline
\multicolumn{1}{|c|}{Handout} &
\multicolumn{1}{c|}{Title} \\ \hline
45 & {\em Memory Consistency and Event Ordering} \\
& {\em in Scalable Shared-Memory Multiprocessors} \\
46 & {\em Cache Coherence Protocols:  Evaluation} \\
& {\em Using a Multiprocessor Simulation Model} \\
47 & Problem Set \#5 Solutions \\
48 & Problem Set \#6 Solutions \\
Notes & Lecture Notes \#18 \\
Notes & Lecture Notes \#22 \\
\hline
\end{tabular}
\end{table}

%%12/9/91 Lecture
%%Bill lecturing
%%Handouts: 45, 46, 47, 48; Lecture notes 18 and 22

\section{Caching in Distributed Systems}

\subsection{Motivations}

In a uniprocessor, a cache reduces the average latency of memory
access because of temporal locality.  The reasons for using a cache
in a distributed system are:
\begin{itemize}

\item Reducing latency, as in a uniprocessor.  Access data ``closer''
to a processor reduces latency.

\item Reducing contention by introducing concurrency into data
accesses.  Reduces the beating (load) on the server and the required
network bandwidth.

\end{itemize}

The problem is how to keep caches coherent, i.e. how to make reads and
writes appear atomic.  

%% CUT? The requirements of multiprocessors and
%% distributed systems are similar, although the study of cache protocols
%% on distributed systems additionally concentrates on fault
%% tolerance.

In distributed filesystems, people have also worried about fault
tolerance issues.  One example worry is what happens when a client
crashes, and that client is holding a dirty cached block?

Simple approaches only allow one client to cache each block at one
time.  This works fine for lots of types of data (ex. local data, like
a program stack), or for unshared files (ex. files in {\tt /tmp} on a unix
machine).  This won't work for other types of data (ex. shared code,
since it will bounce from cache to cache, even though it is
read-only).  Our focus is on shared data.

So, either cache the data or don't cache it.

As an example of the don't cache case, could have dynamic multiple
readers/writers detection, with caching disabled when this occurs.

Or you can allow multiple cached copies of data.  This raises the
``cache coherence problem:'' keeping multiple copies of the data
coherent across caches.  We need to choose the semantics we want in
this case.

Details differ between the bus (ex. multi-processor) and lan (ex.
filesystem) coherence mechanisms, but it seems that the researchers in
each area have not read the papers published in the other area.

\subsection{Specification}

{\bf Strong coherence} --- informally, atomic operations.
\\
The operations supplied by the memory are:
\begin{itemize}
\item minimum of read and write operations
\item testAndSet (or variants, thereof)
\end{itemize}

These operations appear as if there is a single copy of the data.  In
actuality, several different types of coherence have appeared in the
literature: 
\begin{description}
\item[linearizability] (Herlihy and Wing at Carnegie-Mellon University)

What you would get in the SPEC language with atomicity brackets, etc.

Important property if, in a program, higher level primitives may need
linearizability to appear atomic.

\item[sequential consistency] (Leslie Lamport)

Model is multiple processes executing requests.  There exists some
sequential ordering of the operations that is serializable
sequentially for each process.  This is different than linearizability
because in linearizability the time ordering between operations
performed on different nodes is obeyed.  Sequential consistency is
``looser,'' and may allow more concurrency in the system.  Herlihy and
Wing say that it is ``non-local.''
\end{description}

A program typically wants at least sequential consistency behavior.
But, if we only have sequential consistency, then there are certain
operations that need to be coordinated across systems to get stronger
consistency. 

General conclusion: ``strong coherence'' is too vague a term.  The
differences in semantics of different definitions of strong coherence
are critical.
\\
\\
Alternative coherence schemes are ``weaker.''
{\bf Weak consistency} or {\bf Release consistency}\footnote{Explained
in handout 45.} are examples.

The basic idea behind release consistency is that it delays releasing
locks until every write (affected by the locks) is propagated
throughout the system.

For both weak and release consistency, if all data
accesses are protected via locks, then you can show sequential
consistency.  

{\bf Pragmatics:} A typical programmer on a system with ``weak''
underlying coherence mechanisms usually relies upon critical section
and producer/consumer libraries, created by a system wizard.


\subsection{Multiprocessors}

There are two classes of algorithms.

\begin{itemize}
\item Bus-based.  Supports broadcast and are serialized (e.g. atomic).

\item More general interconnection networks, such as
butterflies, hypercubes, and meshes.  To be covered in the next
lecture.
\end{itemize}


\subsubsection{Cache protocols for Bus-based Algorithms}

\begin{figure}
\centerline{\psfig{height=3.5in,figure=caches.idraw}}
\caption{A bus-based system}
\label{bbs}
\end{figure}

Bus-based systems are theoretically interesting, although they are not
universally applicable.  Figure 1 shows a bus-based system.  Each
processor (P) has a cache (C) which is attached to the bus, as are the
memory modules (M).  In a bus based system, the performance is often
limited by bus bandwidth.

The basic idea is that when a processor wants to access a piece of
data, it checks its local cache before placing the request on the
bus.  Because the caches watch the bus, this class of protocols is
called ``snoopy cache algorithms.''  As in any scheme, there is a
choice of write-through versus write-back strategies.

In a write-back policy, data is not copied back to main memory on a
write, and therefore, does not incur a bus cycle, until that block is
replaced in the cache (or possibly needed by another processor).  A
write-through policy requires that main memory and cached copies are
always identical; thus a bus cycle is required for every write.  In
practice, write-back causes less bus traffic (see the Archibald
paper).

Snoopy cache coherency protocols allow for multiple copies of the data
to reside in different caches and relies on a smart cache controller
at each processor to monitor bus transactions and determine what, if
any, actions to take.  Each cache controller is charged with the task
of maintaining the consistency of those blocks of which it has copies.
A block in a cache is assigned to some local state which can be
changed by the controller depending upon what actions are taken upon
that block by other caches.  Other caches' actions will be visible on
the bus.

This approach doesn't scale well.  Around 5-20 processors works ok.
\\
\\
The {\bf actions} that the system can take are:
\begin{itemize}
\item Read and write hits and misses. 
\item Replacements.
\item Bus transactions the cache sees (``snooped'' transactions).
\end{itemize}

{\bf Difference Among Protocols} \ \ Some differences among protocols
are:
\begin{itemize}
\item Do caches know if data is shared?
\item When a cache places a request to read a block on the bus, can
another cache respond to the request or will main memory necessarily respond?
\item When one cache writes a block, do the other caches {\em
invalidate} (throw out) that block, or do they receive an {\em
updated} copy?  Does main memory receive the update?
\item Do they allow multiple writers?
\item They also differ in the snooping hardware.
\end{itemize}

Which protocol is best depends on many factors.
\\
\\
{\bf Definitions}  \ \ Consider cache blocks.  The global state of a block
can be any of the following:

\begin{itemize}
\item Not cached
\item Clean, not shared
\item Clean, shared
\item Dirty, not shared
\item Dirty, shared
\end{itemize}

Some protocols do not allow some of the above global states.  For
example, many algorithms disallow ``dirty, shared.''  A specific
processor does not necessarily know the global state of a block.

\subsubsection{Write-Once Protocol}

The write-once protocol, an early cache coherence protocol designed by
Goodman, has the following states in a single cache:

\begin{itemize}
\item Invalid
\item Valid (clean, maybe shared)
\item Reserved (clean, not shared)
\item Dirty (dirty, not shared)
\end{itemize}

The actions are as follows:
\begin{itemize}

\item Read miss: Request data.  If another cache has the state dirty,
that cache supplies the data and writes it back.  Otherwise, main
memory provides the data.  The requesting cache doesn't know who
provides the data.  All caches that have the data set it to valid.

\item Write hit: The action depends on the state of the writing cache:
\begin{itemize}
\item  If dirty, write the data to the cache.
\item  If reserved, change to dirty, write the data to the cache.
\item  If valid, write through to main memory and change to reserved.
The other caches seeing the write on the bus will invalidate the
blocks locally.
\end{itemize}

\item Write miss: Like a read miss, but other caches go invalid.  The
writer changes its state to dirty.
\end{itemize}


\end{document}