% to get hardcopy of this lecture, you need the following files (plus the
% normal latex and tex base directories):
%     lecture.tex:     latex source file
%     macros.tex:      figure macros
%     psfig.tex:       postscript figure (psfig) macro definition
%     pocs-header.sty: Principles of Computer Systems lecture latex header
%
% run the following programs:
% latex lecture
%   -> Creates lecture.dvi, plus latex intermediate files
% latex lecture
%   -> Run a second time to get cross-references right
% dvi2ps lecture > lecture.ps
%   -> Merges the figures with the text, can also pipe to lpr
%
\documentstyle[12pt,pocs-header]{article}
\Scribes{David Chaiken and Doug DeAngelis}
\Lecturers{Butler Lampson and Bill Weihl}
\LectureNumber{20}
\LectureDate{November 26, 1990}
\include{macros} % used to include figures in this document
\begin{document}
\MakeScribeTop

\parskip = 5pt

\begin{quote}
Well, I guess this class won't be writing any operating systems...
{\em Butler Lampson}
\end{quote}

\section{Administrative Information}

\begin{table}[h]
\centering
\begin{tabular}{|c|l|} \hline
\multicolumn{1}{|c|}{Handout} &
\multicolumn{1}{c|}{Title} \\ \hline
34 & Problem Set \#4 Solutions \\
--- & Problem Set \#4 Corrected \\ \hline
\end{tabular}
\end{table}

The failure on Problem Set \#4 was generalized and wide-spread, so
don't feel too bad if you did poorly.

\section{Last Week's Leftovers}

There are two issues regarding secure communication (discussed in Lecture~19)
that need elaboration.  The first involves the notation and semantics used to
reason about secure channels, and the second is an observation about the
context of secure channels.

\subsection{Semantics of Secure Channels}

The process of in-band authentication 
establishes a secure channel between two parties through the use
of an authentication server.   Section~6.1 of Lecture~19 examines a
scenario in which principals A and B establish a secure channel.
When reasoning about this scenario,
the statement $K_{AS} \models K_{AB} \Rightarrow B$ has the meaning:
\begin{quote}
``From the channel defined by the key $K_{AS}$, received a message
stating that the channel defined by key $K_{AB}$ speaks for
principal B.''
\end{quote}
The syntax $K_{AS} \models K_{AB} \Rightarrow B$ may
be replaced by another syntax, such as $\{K_{AB}, B\}_{K_{AS}}$.
Although the latter syntax may more concretely represent 
the actual encoding of the message in a real system, both notations
have the same meaning (semantics).  Despite the jihads that are fought over
syntax, formal reasoning is based on semantics, not on syntax.

\subsection{The Context of Secure Channels}

The communication network specifications discussed in class allow
messages to be dropped and to be received multiple times.  However in
practice, it is a good assumption that such unfavorable events are
rare.  This assumption is {\em not} valid when reasoning about secure
communication.  In the context of secure channels, it is necessary to
assume that an adversary (the ``bad guy'') will attempt to break the
communication scheme.  

In particular, an adversary can duplicate an
arbitrary number of messages and repeat them at any time.  Thus, certain
arguments that are valid for a normal network are not valid for a secure
network.  For example, it
is possible to argue that a message identifier may be reused after a
given amount of time in a normal network.  However, the activity of
the adversary makes such an argument invalid for a secure network.
While the adversary complicates the issues involving
the generation and use of unique message identifiers, many of the
concepts from the communication field carry over into the realm of security.

\begin{table}[h]
\centering
\begin{tabular}{|c|c|} \hline
\multicolumn{1}{|c|}{Communications} &
\multicolumn{1}{c|}{Security} \\ \hline
Duplicate Receives & Replay Attack \\
Unique Identifier  & Nonce \\ \hline
\end{tabular}
\end{table}

\section{Distributed Transactions}

The specification of a distributed transaction system is fundamentally
the same as the specification of transactions in a centralized system.
Both types of transaction system are used to guarantee the atomicity,
permanence, and serializability of collections of
operations~\cite{Camelot}.  However, a distributed implementation of a
transaction system is spread over multiple sites, and can tolerate the
failure of various combinations of the sites.  The two fundamental
issues that are inherent in a distributed system are
\begin{itemize}
\item coping with partial failures in the system, and
\item coordinating sites for synchronization.
\end{itemize}

\section{Recovery from Partial Failure}

To identify the goals of a distributed system, examine a naive
implementation of a transaction system.  Appoint one site as a {\em
lock manager}, which centralizes the access to all shared objects.
When any site in the distributed system fails, reboot every site in
the entire system.  This first-cut implementation suffers from two
fundamental problems.  First, the lock manager is a performance
and reliability bottleneck.  Since all shared accesses must be
coordinated through a central site, the bandwidth of the system is
limited by the bandwidth of the lock manager.  When the lock manager
fails, the entire system fails.  Instead, a distributed transactions system
should disperse the management of transactions.  Second, the availability 
of the system as a whole is poor.  Rather than propagating the effects
of a site failure, a distributed system should attempt to isolate failure.

A model of a distributed transaction system is illustrated in
Figure~\ref{distfig}.  A transaction manager (TM) resides at each site
and maintains an independent log for the stable state at the site.
Assume (for now) that when a site fails, it crashes completely
(fail-stop).  The network may lose or duplicate messages, but does not
corrupt messages.  A transaction can begin at any site, and can move
to other sites via a parameter of a remote procedure call (RPC).
A transaction starts at some site, and wanders around the
distributed system through the network.  To simplify the model, assume
that a transaction commits at the site where it started.  (This
assumption about commits is not essential.)

\begin{figure}
\centerline{\psfig{width=3.5in,figure=dist.idraw}}
\caption{A distributed transaction system.}
\label{distfig}
\end{figure}

There are two acceptable behaviors of a transaction.
\begin{enumerate}
\item  A transaction can commit successfully at all sites and persist.
\item  A transaction can abort, and none of its modifications can affect
any transactions that commit.
\end{enumerate}
The problem in a distributed system is determining when a transaction may
commit.  Since it is unacceptable for some sites to commit a transaction
while others abort it, it is necessary to implement a communication
protocol that enforces {\em atomic commitment}, even in the presence of
site failures.

\subsection{The 2-Phase Commit Protocol}

The classic solution to the atomic commitment problem is the 2-phase
commit protocol.\footnote{The 2-phase commit protocol should not be
confused with the 2-phase locking scheme that is used to serialize
committed transactions.} For each transaction, this protocol
designates one site as a {\em coordinator} to control the commit
procedure.  Assume (for now) that a transaction's coordinator is the
site where it starts and commits.  The other sites that are involved
in a transaction are called {\em participants}.

The 2-phase protocol is used when a transaction attempts to commit.
In the first ({\em prepare}) phase of the protocol, the coordinator
collects commitment votes.  A participant votes ``yes'' if it thinks
that it is okay to commit the transaction, otherwise it votes
``no.''\footnote{Implementations of the 2-phase protocol can make
provisions for transactions that are pending.  For example, a
participant may wait a certain amount of time before responding to a
vote request.  A participant might also reply with a ``try again
later'' message.} A participant sends a ``yes'' response only after
ensuring that all modifications associated with the transaction are in
stable storage and are recoverable.  Thus, a ``yes'' vote requires a
log force.  After voting ``yes,'' a participant may not change its mind
and unilaterally abort the transaction.

Once the coordinator receives all of the votes, it decides whether or
not to commit the transaction.  If all of the participants vote
``yes,'' then the coordinator decides to commit, otherwise it decides
to abort the transaction.  In the second phase of the protocol, the
coordinator distributes the decision to the participants.  When a
participant receives the decision, it can release the locks and
perform the updates associated with the transaction.

The above discussion describes how the 2-phase protocol achieves the
goal of distributing transaction management.  The protocol handles
failures in the following ways:

\begin{enumerate}

\item {\bf Network failures:} If the network loses a message, the
coordinator retransmits the vote request until it receives a
response.  If a participant receives a duplicate vote request, it
resends its vote.  If a coordinator receives a duplicate response,
it ignores the redundant vote.

\item {\bf Participant fails after voting:} 
When a participant in a 2-phase protocol crashes, it can not simply
abort all active transactions, as in a centralized system.
If the participant votes to commit a transaction before failing, 
then after recovering, it must recover the transaction's state prior
to the failure.  To allow this type of recovery, a participant marks
a transaction as {\em prepared} before sending a ``yes'' vote to
the transaction's coordinator (hence the name of the {\em prepare} phase).

The recovery is implemented by recording prepare records in the redo
log.  A participant must force a prepare record to stable storage
before sending voting to commit a transaction.  The redo procedure
restores the locks associated with each of the prepared transactions,
and leaves the transactions in an active state.  The participant must
be able to infer the locks associated with each transaction from the
stable redo log.

%A participant can either abort
%transactions that are unprepared or assume that transactions are
%active until they are explicitly aborted.

\item {\bf Participant fails before voting:}
Since it is usually impossible for a coordinator to tell the difference
between communication failure and participant failure, the coordinator must
time-out after long voting delays.  Upon a time-out, a coordinator aborts
the delayed transaction.  However, this policy is not enough to guarantee
correct behavior after participant failure.  Consider the following scenario:
\begin{quote}
\begin{tabbing}
1. \= Transaction T visits participant X \\
2. \> T leaves X and executes at a different site \\
3. \> X crashes \\
4. \> X recovers \\
5. \> T attempts to commit \\
\end{tabbing}
\end{quote}
In general, it is difficult to detect this scenario.  To ensure correct
behavior, X can abort the transaction when it recovers and then vote ``no''
when the coordinator attempts to commit T.

\item {\bf Coordinator fails before decision:} The coordinator aborts pending
transactions during recovery.  To successfully abort a transaction after
a crash, the transaction's participants must be recorded in stable storage.

\item {\bf Coordinator fails after decision:}
If the coordinator fails after making a decision, the decision must persist.
This implies that the coordinator must write the decision to the redo log
and force the log before transmitting the decision to any participants.

\end{enumerate}

There are two types of log forces that are required during
a 2-phase commit protocol.  First, the participants must make sure that
all of the transaction's updates are logged in stable storage.  This
force is analogous to the ForceAll() in the sequential implementation
of transactions.  Second, the coordinator must record the decision in
stable storage.  This action is analogous to writing the commit record
to the stable log in the sequential implementation.

\subsection{The 3-Phase Commit Protocol}

The two-phase commit protocol is termed a {\em blocking commit
protocol}, because it blocks a participant from taking action on a
transaction while the coordinator of that transaction is down.  The
time span of concern starts at the time that the coordinator gets all
the votes and ends when the coordinator finishes distributing the
decision.  If the coordinator fails during this window of vulnerability, the
participant continues to hold the locks associated with that transaction.
Participants have no autonomous method for deciding
the fate of pending transactions.

It would be quite desirable to allow participants to release the locks
associated with a transaction when the coordinator fails.  By assuming
that sites can detect the difference between communication failures
and failures of other sites, it is possible to implement a
non-blocking scheme.  Unfortunately, it is extremely difficult (if not
impossible) to differentiate between network partitions and site failures.
Thus, schemes that assume such a mechanism have more theoretical than
practical interest.

Nevertheless, the 3-phase commit protocol has been proposed to implement a
non-blocking scheme.  Camelot~\cite{Camelot}, the CMU distributed
transaction processing system, offers the 3-phase protocol as an
optional feature.  The protocol consists of the following three phases:
\begin{enumerate}
\item collect votes
\item distribute votes
\item distribute decision
\end{enumerate}

This protocol returns a degree of autonomy to the participants.  In
the typical case, the coordinator makes the commit/abort decision, and
eventually distributes the decision.  However, the transaction can be
aborted, even if all of the participants vote ``yes.''
Figure~\ref{3phase} shows such a scenario.  In the first phase, the
two participants in a transaction vote to commit.  During the second
phase, the coordinator distributes the votes to one participant.
Before the vote distribution phase is complete, both the coordinator
and the participant that received the votes crash.  The other participant
detects that all of the sites that know the votes have crashed.  (Recall
that the 3-phase protocol requires a crash detection mechanism.)  The
remaining participant must abort the transaction in order to ensure
correct behavior without blocking.

\begin{figure}
\centerline{\psfig{height=1.5in,figure=3phase.idraw}}
\caption{3-phase protocol scenario.  The transaction is aborted, even though
both participants vote ``yes.''}
\label{3phase}
\end{figure}

%Butler suggestion for a synchronous system.  With common delay time and
%maximum message delay, can make a non-blocking protocol work without
%relying on crash detection.

\subsection{Crash Counts}

If a transaction manager is asked for a vote, it must vote no if it
has no record of the transaction for any reason.  It can not vote yes
based solely on the fact that it has a record, however.  Consider the
following scenario:
\begin{quote}
\begin{tabbing}
1. \= Transaction T visits participant X \\
2. \> T leaves X and executes at a different site \\
3. \> X crashes \\
4. \> X recovers \\
5. \> T visits X again \\
\end{tabbing}
\end{quote}

Here, the participant must have a record of the fact that it
crashed at some point after the transaction first visited it.  The
following ways of accomplishing this were suggested.

\begin{itemize}
\item The participant logs a ``T was here'' record at each visitation
(this implies a log force).
\item Each node keeps a count of the number of times it has crashed.
The transaction can then record the crash count of each node at the
time it visits it and will know to abort if the crash count of any
participant increases.
\item The participant marks itself on the transaction if it believes
that this is the first time it has been visited.  If any site
marks a transaction multiple times, then it must have crashed and the
transaction should be aborted.
\end{itemize}

\subsection{Comparing Distributed and Centralized Transactions}

These are the fundamental differences between the DTP systems
described and the sequential systems discussed earlier in the term:

\begin{enumerate}
%elaborate based on whatever you said earlier
\item Two log forces are involved.
\item A node can not just simply decide to abort after a crash.
\item The coordinator (or someone) needs to remember the decision on
the fate of a transaction.
\end{enumerate}

\section{Managing Synchronization}

Consider two sites in a distributed transaction processing system, one
of which achieves serialization using two-phase locking (2PL), the
other of which uses multi-version time stamps (MVTS).  Now consider
the scenario in Figure~\ref{synchfig}.

\begin{figure}
\centerline{\psfig{width=4in,figure=synch.idraw}}
\caption{Effect of multiple serialization techniques.}
\label{synchfig}
\end{figure}

Locally to each site, things are fine.  But from the point of view of
the system, the site doing 2PL would have expected the value read in
T1 on site Y to be that which was written in T2 on site Y because it
considers the committed transaction to have occurred first.  Site Y,
however, is serializing based on time stamps, so it is simply assuring
that the value read while servicing T1 is {\em not} affected by T2.

One solution to this is simply to assure that all the participating
transaction managers have the same impression of the serialization
being done.  This, of course, decreases the ability to use
heterogeneous systems which may be made by different vendors.

\bibliographystyle{plain}
\bibliography{lecture20}

\end{document}