% to get hardcopy of this lecture,you need the following files (plus the
% normal latex and tex base directories):
%     lecture.tex:     latex source file
%     macros.tex:      figure macros
%     psfig.tex:       postscript figure (psfig) macro definition
%     figI.idraw:        figures created with idraw (I = 1,2,3,4)
%     pocs-header.sty: Principles of Comp Systems lecture latex header
%
% run the following programs:
% latex lecture
%   -> Creates lecture.dvi, plus latex intermediate files
% latex lecture
%   -> Run a second time to get cross-references right
% dvi2ps lecture |  egrep -v '^showpage$' > lecture.ps
%   -> Merges the figures with the text, can also pipe to lpr
%
\documentstyle[12pt,pocs-header]{article}
\Scribe{Atul Adya}
\Lecturer{Bill Weihl}
\LectureNumber{20}
\LectureDate{November 25, 1991}
\include{macros} % used to include figures in this document
\begin{document}
\MakeScribeTop

\section*{Handouts}
	Handouts 39 ({\em Primary Copy Replication}) and 40 ({\em
Replication in the Harp File System - Barbara Liskov, Sanjay Ghemawat,
et al.}) were distributed.

\section*{Overview}
	The lecture discussed the techniques of primary copy
replication for achieving higher availability in distributed systems.
The reliability of a system can be increased by using one of the
replication schemes.

\section*{Specification}

The two specifications given in handout 39 have at-most-once semantics
for the actions. Both of them have the same externally observable
behavior, {\em i.e.}, either the {\em Do}\/ action is performed or an
exception is raised, or an exception is raised after doing the action.
After seeing a failed exception, the user has to check whether the
action was actually done or not (If the actions are idempotent then it
does not matter and he can just keep executing the operation until it
gets done.) The first spec is more natural and easy to understand but
to use an abstraction function we need additional machinery like
prophecy or non-deterministic variables (similar to history variables
except that these variables ``keep'' information about the future
rather than the past). Hence, we make use of the second spec, where in
the case of failure , it delays its decision about performing the
action or not.

\section*{Implementation}

In a primary copy replication scheme, we have one primary and a number
of backups that keep the replicated information. Whenever
a client sends a request to the primary, it performs the actions
locally and logs them to a majority of the replicas. As long as no
failures occur in the system, the functioning is smooth. In the event
of a failure, {\em i.e.}, network partitioning or node crashing, the
system has to be {\em reconfigured}. This reconfiguration, which
essentially involves electing a new primary and re-initialization of
the state, is called a {\em view change}. If the system was in the
middle of a client request it can raise a failed exception.

The first problem is to elect a primary. We have to ensure that there
exists at most one primary in the system at any given instant. Any of
the replicas can initiate a view change and if it succeeds in getting
a positive reply from a majority of the replicas then it can become
the primary \footnote {The basic protocol used in this replication
scheme is that for any action to be acceptable in the system, a
majority of the replicas must agree.}.  If $P$ fails or the network
partitions (as shown in figure 1) then this protocol can be used for a
view change. In the case of network partitioning , the views on both
sides of the partition are different. So when the network is repaired,
the rest of the replicas have to be reintegrated into the system to
bring them up-to-date with the majority. Also, while the network was
down, the new primary was handling the requests for the clients on the
right of the partition (figure 1). But the clients that were sending
requests from the left of the partition were stuck and remain stuck
until the network is repaired. So if there are two network partitions
such that there does not exist a majority in any of the partitions
then the system is blocked until further repair is done.  To handle
such a situation:

\begin{itemize}

\item As soon a replica goes down , another one can be brought
 into the system to replace it.

\item The number of the replicas (as known by each of the nodes) can
 also be reduced. We do not require any new protocols since the
majority protocol is sufficient for reaching a consensus on the number
of the replicas in the system.

\end{itemize}

\begin{figure}
\centerline{\psfig{width=2.5in,height=2in,figure=fig1.idraw}}
\caption{Network partitioning leads to a contention  for the primary status}
\label{fig1}
\end{figure}

\vspace {2 mm}
It could happen that more than one replica decides to become the
primary. This contention can be cleared up if we label each view with
a view-id (VID) that is monotonically increasing. So, the invoker
chooses a view-id larger than the largest id it is aware of and sends
messages to the other replicas. If some other replica tries to invoke
a view change then the replica with the higher VID wins the game and
becomes the new primary. (Of course, if there are lot of crashes and
view changes keep getting invoked then we might reach a livelock
situation.)


The second problem is to re-initialize the state. Each replica
maintains a log and so each one of them has a complete history. The
system starts with a particular view and then goes through a series of
views. Anyone who notices a failure can invoke a view-change and try
and get a majority of votes for himself. Since there has been a crash,
each replica might not have the same information. So the invoker (or
the view manager) also has to ensure that he is up-to-date and so are
the voters, {\em i.e.}, the logs of the participating replicas have to
be made the same. 

The view-manager sends a request to all the nodes with the new VID and
these nodes acknowledge (if they decide to join this view) by sending
their log. The view-manager keeps track of the {\em shortest} log
which it has received since it wants to keep the {\em largest} log
agreed to by the majority. When the view-manager gets a majority of
the acks it broadcasts the new state to all the nodes and polls until
it gets an accept response from a majority of the nodes. It then
becomes the primary for the new view and goes to the idle state ,
ready to accept client requests (see figure 2). So essentially, every
view has a log associated with it and it corresponds to one that the
majority agrees on. The key invariant that is being maintained is
that if an operation has been seen by a majority in one view then it
will be seen in any of the later views, {\em i.e.}, any action
committed in a view stays committed in the later views also.

\begin{figure}
\centerline{\psfig{width=3.5in,height=4in,figure=fig2.idraw}}
\caption{State transitions during a view change}
\label{fig2}
\end{figure}

\subsection*{Bugs/Changes in Handout 39}

\begin{itemize}

\item  When a replica is not idle and it receives a request from a client,
it non-deterministically sends the message to one of the replicas.
Thus, the client request might get bounced about and the client might
give up due to a timeout. The client might now send his second request,
which gets executed, and then the old request (which had been bouncing)
gets executed. This violates the condition that client requests are
executed in the order in which they are sent. This problem can be
resolved if the requests are tagged by monotonically increasing ids.

\item In the implementation given in handout 39 (page 8), after changing
its state from waiting to logging, it should send an acceptance message
along with cv to r'.

\item In the trying state, when a majority of the replicas have decided
to join the view, the invoker goes to the polling state before
actually having made the required state changes ({\em i.e.}, before
broadcasting the initial state). This is an optimization that helps in
faster view change.

\end{itemize}

\subsection*{Variations}

Primary copy schemes require flushing to the backup at the end of each
operation. The primary is not allowed to say $yes$ until the operation
has been committed in the majority of the replicas. So there might be
a long delay in the answer for the client. Brian Oki uses a
write-behind strategy to improve the response time. In Oki's scheme,
the flushing is done at the commit point ({\em i.e.}, before voting yes
in the 2-phase commit protocol). If there is a crash then the update
might be lost (if it had not been committed to the majority), but
since we are using 2PC the transaction would have anyway been aborted.
Lamport's $Paxos$ (non-blocking) protocol is essentially the same as
the protocol presented in class except that it has been described in a
much more complicated manner.  In the early '80s, 3-phase commit
protocols were designed with an extra phase to remove blocking. Skeen
gave such a protocol in which the client requests will be serviced as
long as there is at least one node functioning. It is quite similar to
the protocol discussed in class but is quite complicated. The protocol
presented in the class is essentially a non-blocking protocol as long
as the majority of the nodes are available. Of course, if several
nodes are trying to form a new view constantly then we might reach a
livelock situation. For example, if a new node (say X) initiates a
view change and overtakes the others. Now one of these nodes could
initiate a view change later and overtake node X. Thus, this process
could go on forever or at least degrade performance if replicas keep
invoking view changes quite often.


\section*{Other Schemes}

\subsection*{Voting}
This is another way of increasing availability in distributed systems.
There is a concept of {\em quorum} involved, {\em i.e.}, a set of sites
that are sufficient to perform a particular operation (say read and
write). The basic conditions for these quorums are:

\begin{itemize}

\item The write quorums must intersect so that every update must see
the previous updates.  

\item The read quorums must also intersect.

\item The read and write quorums must intersect so that when a read
operation is done, it is guaranteed to see every write.

\end{itemize} 

Two types of voting schemes were discussed:

\begin{itemize}

\item {\em Write-all, Read-one} --- While writing, all the copies are
updated whereas for reading we can read from any one of the replicas.
This makes writes very expensive and reads extremely cheap. So a high
throughput can be obtained if the operations in the system are heavily
weighted towards reads.

\item {\em Write-majority, Read-majority} --- Both reading and writing
require the action to be done on a majority of the sites. So reading
becomes more expensive than in the previous strategy but writing is considerably
cheaper.  

\end{itemize}

\subsection*{Echo File System} This file system by DEC
uses the notion of a {\em witness}. If there are 3 nodes in the
system, one acts as the primary, one as the backup and the third as
the witness. The primary logs the action only to the backup (not to
the witness) and so only 2 copies of the file system are kept. If the
primary goes down, the witness is promoted to the status of a backup
(and the backup becomes the primary). When the old primary comes up
again, the current backup is demoted to its original witness
status.\footnote{Similar issues are discussed in handout 40.}

\subsection*{Others}
Some systems cannot implement the at-most-once specification since
it's too strong, {\em e.g.} a widely-distributed name server, where if
there is a loss of copies then they cannot be updated atomically. So a
{\em diffusion strategy}\/ is used where the update is done at any
server and the effect gets propagated in the background. Hence, some
server might not see the changes for some time. The spec for such
systems is much weaker than the one we have discussed in the class.




\end{document}