% to get hardcopy of this lecture, you need the following files (plus the
% normal latex and tex base directories):
%     lecture.tex:     latex source file
%     macros.tex:      figure macros
%     psfig.tex:       postscript figure (psfig) macro definition
%     pocs-header.sty: Principles of Computer Systems lecture latex header
%
% run the following programs:
% latex lecture
%   -> Creates lecture.dvi, plus latex intermediate files
% latex lecture
%   -> Run a second time to get cross-references right
% dvi2ps lecture |  egrep -v '^showpage$' > lecture.ps
%   -> Merges the figures with the text, can also pipe to lpr
%
\documentstyle[12pt,pocs-header]{article}
\Scribe{Irene Shen}
\Lecturer{Bill Weihl}
\LectureNumber{18}
\LectureDate{November 18, 1991}
\include{macros} % used to include figures in this document
\begin{document}
\MakeScribeTop

\section{Handouts}
\begin{itemize}
\item Handout 37:  ``A Census of Tandem System Availability Between
1985 and 1990'' by Jim Gray.
\item Handout 38:  Problem Set \#7
\end{itemize}

\section{Transactions in Distributed Systems}

The specification of a distributed transaction system is fundamentally
the same as the specification of transactions in a centralized system.
The transaction system is used to guarantee the atomicity and
serializability of actions.  However, a distributed implementation of a
transaction system is spread over multiple sites, and must be able to
tolerate the failure of various combinations of the sites.  There are
several fundamental issues inherent to a distributed system that the
distributed implementation should address: 
\begin{itemize}
\item coping with partial failures,
\item coordinating sites for synchronization,
\item load distribution (avoiding bottlenecks), and
\item crash recovery to maintain reliability.
\end{itemize}

\section{Recovery from Partial Failure}

To identify the goals of a distributed system, examine a naive
implementation of a transaction system.  Appoint one site as a {\em
lock manager}, which centralizes the access to all shared objects.
When any site in the distributed system fails, reboot every site in
the entire system.  This first-cut implementation suffers from two
fundamental problems.  First, the lock manager is a performance
and reliability bottleneck.  Since all shared accesses must be
coordinated through a central site, the bandwidth of the system is
limited by the bandwidth of the lock manager.  When the lock manager
fails, the entire system fails.  Instead, a distributed transactions system
should disperse the management of transactions.  Second, the availability 
of the system as a whole is poor.  Rather than propagating the effects
of a site failure, a distributed system should attempt to isolate failure.

A model of a distributed transaction system is illustrated in
Figure~\ref{distfig}.  A transaction manager (TM) resides at each site
and maintains an independent log for the stable state at the site.
Each site maintains its own REDO and UNDO logs.  Assume (for now) that
when a site fails, it crashes completely (fail-stop).  The network may
lose or duplicate messages, but it does not corrupt messages.  A
transaction can begin at any site, and can move to other sites via a
parameter of a remote procedure call (RPC).  A transaction starts at
some site, and wanders around the distributed system through the
network.  To simplify the model, assume that a transaction commits at
the site where it started.  (This assumption about commits is not
essential.) 
\begin{figure}
\centerline{\psfig{height=2.5in,figure=dist.idraw}}
\caption{A distributed transaction system.}
\label{distfig}
\end{figure}
There are two acceptable behaviors for a transaction.
\begin{enumerate}
\item  A transaction can commit successfully at all sites and persist.
\item  A transaction can abort, and none of its modifications can affect
any transactions that commit.
\end{enumerate}
The problem in a distributed system is determining when a transaction may
commit.  Since it is unacceptable for some sites to commit a transaction
while others abort it, it is necessary to implement a communication
protocol that enforces {\em atomic commitment}, even in the presence of
site failures.

\subsection{The 2-Phase Commit Protocol}

The classic solution to the atomic commitment problem is the 2-phase
commit protocol (2PC).\footnote{The 2-phase commit protocol should not
be confused with the 2-phase locking scheme that is used to serialize
committed transactions.} For each transaction, this protocol
designates one site as a {\em coordinator}\/ to control the commit
procedure.  Assume (for now) that a transaction's coordinator is the
site where it starts and commits.  The other sites that are involved
in a transaction are called {\em participants}.

The 2-phase protocol is used when a transaction attempts to commit.
Its operation is shown in Figure~\ref{2pc}.  In the first phase of the
protocol, the {\em prepare}\/ phase, the coordinator collects
commitment votes.  A participant votes ``yes'' if it thinks that it is
okay to commit the transaction, otherwise it votes
``no.''\footnote{Implementations of the 2-phase protocol can make
provisions for transactions that are pending.  For example, a
participant may wait a certain amount of time before responding to a
vote request.  A participant might also reply with a ``try again
later'' message.} A participant sends a ``yes'' response only after
ensuring that all modifications associated with the transaction are in
stable storage and are recoverable.  Thus, a ``yes'' vote requires a
log force.  After voting ``yes,'' a participant may not change its
mind and unilaterally abort the transaction unless told to do so.

\begin{figure}
\centerline{\psfig{height=2.5in,figure=2PC.idraw}}
\caption{The 2-Phase Commit Protocol}
\label{2pc}
\end{figure}

Once the coordinator receives all of the votes, it decides whether or
not to commit the transaction.  If all of the participants vote
``yes,'' then the coordinator decides to commit, otherwise it decides
to abort the transaction.  In the second phase of the protocol, the
coordinator distributes the decision to the participants.  When a
participant receives the decision, it can release the locks and
perform the updates associated with the transaction decision.  Then,
the participant sends an ACK back to the coordinator.  The coordinator
must receive all the ACK's before it may move on to a new transaction.

The above discussion describes how the 2-phase protocol achieves the
goal of distributing transaction management.  The protocol handles
failures in the following ways:

\begin{enumerate}

\item {\bf Network failures:} If the network loses a message, the
coordinator retransmits the vote request until it receives a
response.  If a participant receives a duplicate vote request, it
resends its vote.  If a coordinator receives a duplicate response,
it ignores the redundant vote.  The same is done for a network failure
during the decision distribution phase.

%A participant can either abort
%transactions that are unprepared or assume that transactions are
%active until they are explicitly aborted.

\item {\bf Participant fails before voting:}
To ensure correct behavior after a crash, the participant aborts the
transaction when it recovers and then votes ``no'' when the
coordinator attempts to commit the transaction. 

\item {\bf Participant fails after voting:} 
When a participant in a 2-phase protocol crashes, it can not simply
abort all active transactions, as in a centralized system.  If the
participant voted to abort, then it just aborts the transaction during
recovery.  If the participant voted to commit the transaction before
failing, then after recovering, it must recover the transaction's
state prior to the failure.  To allow this type of recovery, a
participant marks a transaction as {\em prepared}\/ before sending a
``yes'' vote to the transaction's coordinator (hence the name of the
{\em prepare}\/ phase).  Marking a transaction as {\em prepared}\/
essentially forces enough volatile state to stable storage, called the
{\em prepare record}\/ so that the transaction can commit or abort.
After performing this action, the participant enters the {\em prepared
state}.  After recovery, the participant just waits for the
coordinator's decision since it is able to restore the current
transactions state.  

\item {\bf Coordinator fails:}
If the coordinator crashes, it still needs to distribute consistent
information to the participants of the current transaction.  In order
to achieve this, the coordinator is required to have a transaction's
participants recorded in stable storage.  If the coordinator fails
before making a decision, it aborts any pending transactions during
recovery.  If the coordinator fails after making a decision, the
decision must persist.  This implies that the coordinator must write
the decision to the redo log and force the log before transmitting the
decision to any participants.  The point at which the decision is
forced to stable storage is called the {\em commit point}.

There are two schemes to consider for implementing the decision force.
One case, the {\bf Presumed Abort} scheme, optimizes the abort
decision.  For this case, if the decision is to abort, the coordinator
can forget about the transaction as soon as it sends its decision to
the participants; the coordinator does not need to wait for any ACK's
before continuing on to the next transaction.  The coordinator also
does not need to force the abort decision since if it crashes, the
transaction will be aborted in the recovery procedure.  Unfortunately,
abort decisions should not occur very often, so this scheme does not
optimize the usual case. 

For optimizing the commit decision, there is a {\bf Presumed to
Commit} scheme.  This scheme assumes that a transaction will always
commit.  If the coordinator's decision is to abort, then it needs to
keep a transaction's information until all the ACK's return from the
abort decision.  The ACK's are only needed for an abort decision.
However, this scheme requires an extra log write before the
coordinator sends out a commit request.  

There is also a scheme to optimize transactions involving Read-Only
participants.  Participants who hold only read-only locks may release
all locks after the {\em prepared}\/ phase in {\bf Pre-Abort} scheme.

\end{enumerate}

Essentially, there are two types of log forces that are required during
a 2-phase commit protocol.  First, the participants must make sure that
all of the transaction's updates are logged in stable storage.  This
force is analogous to the ForceAll() in the sequential implementation
of transactions.  Second, the coordinator must record the decision in
stable storage.  This action is analogous to writing the commit record
to the stable log in the sequential implementation.

\subsection{2-Phase Commit Protocol Issues}

Two key questions remain about the 2PC Protocol:
\begin{itemize}
\item How does a participant choose to vote?
\item What are the issues of blocking?
\end{itemize}

\subsubsection{2PC Voting}

If a transaction manager is asked for a vote, it must vote no if it
has no record of the transaction for any reason.  It can not vote yes
based solely on the fact that it has a record, however.  Consider the
following scenario:
\begin{quote}
\begin{tabbing}
1. \= Transaction T visits participant X \\
2. \> T leaves X and executes at a different site \\
3. \> X crashes \\
4. \> X recovers \\
5. \> T visits X again \\
\end{tabbing}
\end{quote}

Here, the participant must have a record of the fact that it
crashed at some point after the transaction first visited it.  The
following ways of accomplishing this were suggested.

\begin{itemize}
\item The participant logs a ``T was here'' record at each visitation
(this implies a log force).
\item Each node keeps a count of the number of times it has crashed.
The transaction can then record the crash count of each node at the
time it visits it and will know to abort if the crash count of any
participant increases.
\item The participant marks itself on the transaction if it believes
that this is the first time it has been visited.  If any site
marks a transaction multiple times, then it must have crashed and the
transaction should be aborted.
\end{itemize}

\subsubsection{Blocking}

The two-phase commit protocol is termed a {\em blocking commit
protocol}, because it blocks a participant from taking action on a
transaction while the coordinator of that transaction is down.  The
time span of concern starts at the time that the coordinator gets all
the votes and ends when the coordinator finishes distributing the
decision.  If the coordinator fails during this window of vulnerability, the
participant continues to hold the locks associated with that transaction.
Participants have no autonomous method for deciding the fate of
pending transactions. 

\section{Managing Synchronization}

Consider two sites in a distributed transaction processing system, one
of which achieves serialization using two-phase locking (2PL), the
other of which uses multi-version time stamps (MVTS).  Now consider
the scenario in Figure~\ref{synchfig}.

\begin{figure}
\centerline{\psfig{height=2.5in,figure=synch.idraw}}
\caption{Effect of multiple serialization techniques.}
\label{synchfig}
\end{figure}

Locally to each site, things are fine.  But from the point of view of
the system, the site doing 2PL would have expected the value read in
T1 on site Y to be that which was written in T2 on site Y because it
considers the committed transaction to have occurred first.  Site Y,
however, is serializing based on time stamps, so it is simply assuring
that the value read while servicing T1 is {\em not}\/ affected by T2.

One solution to this is simply to assure that all the participating
transaction managers have the same impression of the serialization
being done.  This, of course, decreases the ability to use
heterogeneous systems that may be made by different vendors.

\end{document}