% to get hardcopy of this lecture, you need the following files (plus the
% normal latex and tex base directories):
%     lecture19.tex:     latex source file
%     model.ps:		 figure
%     pocs-header.sty:   Principles of Computer Systems lecture latex header
%
% run the following programs:
% latex lecture19
%   -> Creates lecture.dvi, plus latex intermediate files
% latex lecture19
%   -> Run a second time to get cross-references right
% dvi2ps lecture19 |  egrep -v '^showpage$' > lecture19.ps
%   -> Merges the figures with the text, can also pipe to lpr
%
\documentstyle[12pt,pocs-header]{article}
\Scribe{Bill Kruger}
\Lecturer{William Weihl}
\LectureNumber{19}
\LectureDate{November 20, 1991}
\include{macros} % used to include figures in this document

\begin{document}
\MakeScribeTop


\section{Concurrency Control --- Continued}

Recall from last lecture that in a distributed transaction handling
system, if at one site the local transaction manager uses, for
example, Two Phase Locking as its method of synchronization, and at
another site within the same system that transaction manager uses some
other synchronization method (Multi-Version Time Stamp, for example)
then one can encounter situations where locally all of the updates of a
particular transaction are serializable, but on a global scale they
are not.  Two possibile solutions were suggested:

\begin{itemize}
\item Enforce the same synchronization mechanism at all sites

\item Include in the system specification a specification of the correct
behavior of the synchronization mechanism
\end{itemize}

The first choice is somewhat impractical, as it severely restricts the
implementation possibilities at each site, and makes a heterogeneous
system built from different local transaction managers more difficult
(as one can only use a transaction manager package that happens to
use the synchronization mechanism of choice).  The second choice is
more practical, as it allows any type of implementation as long as the
behavior of the synchronization mechanism obeys the overall system
specification.

Our model of a distributed transaction system is shown in
Figure~\ref{model}.  The client is an entity outside the system, while
the ``system'' includes all of the transaction managers (TM).  The TMs
handle the synchronization and everything else locally by themselves.
Our specification of this system must include all of these TMs, and
specifically must address how the TMs can ``agree'' on a correct
serialization order.

\begin{figure}
\centerline{\psfig{height=1.5in,figure=model.ps}}
\caption{Distributed Transaction System Model}
\label{model}
\end{figure}

\subsection{Example System Specification}

If one examines the serialization order of the Two Phase Locking synchronization
mechanism, one finds that the serialization is {\em lock point order}:

\begin{itemize}
\item {\bf Lock Point Order}: any point between the last lock acquire operation and
the first lock release operation.
\end{itemize}

We can specify our distributed system so that any synchronization
mechanism yields a serialization order consistent with lock point
order.  One problem is immediately encountered, however, and that is
that no TM globally knows the lock point order (it only knows the
order of locks that it has locally, and has no knowledge of locks held
by other TMs involved in the transaction).  It should be possible,
however, for a TM to use its local knowledge to ``compute'' the
necessary global lock point order.

To do this, the relation {\em Precedes(x)} was introduced:

\begin{itemize}
\item {\bf Precedes(x)}: if $T_{1}$ and $T_{2}$ are transactions and X is a TM, then we say
that $T_{1}$ {\em precedes} $T_{2}$ at X if X learns of the commit of
$T_{1}$ before $T_{2}$ executes any operation at X.
\end{itemize}

Now, in our specification, we can state the necessary serialization
behavior required of all TMs:

\begin{itemize}
 \item $TM_{x}$ must ensure that committed transactions are
serializable in all orders consistent with Precedes(x).
\end{itemize}

Note that if two transactions run concurrently, then they must be
serializable in either order ($T_{1}T_{2}$ or $T_{2}T_{1}$) (i.e.,
with 2PL, $T_{1}$'s and $T_{2}$'s actions commute, so either order is
okay).  A theorem is arrived at out of this discussion:

\begin{itemize}
 \item {\bf Theorem}: There exists a total order O such that for all
X, X a TM in the system, O is consistent with Precedes(X).
\end{itemize}

Having specified the serialization order required of a synchronization
mechanism employed at each site, each TM is allowed a choice of which
synchonization mechanism to use, as there are other synchronization
mechanisms other than Two Phase Locking that also satisfy the above
requirements.

\section{Availability}

The rest of the lecture was devoted to a discussion of availability in
a distributed system.  In a distributed system, a particular site can
fail while other sites remain available: we'd like the system to
reconfigure itself to assume the responsibilities of the failed site,
allowing the system as a whole to remain available to perform useful
work.  This ability of the system to reconfigure itself is
accomplished using some form of replication.  To get an idea of what
is meant by the term available, some useful metrics were introduced,
along with an informal definition for availability:

\begin{itemize}
 \item {\bf Mean Time Between Failure (MTBF)}: the average time
between a failure of a component within a system or of the system
itself

 \item {\bf Mean Time To Repair (MTTR)}: the average time taken to
repair a failed component within a system or the system itself (note
that this includes the time taken to detect the failure plus the time
to repair plus the time to reintegrate it back into a state at which
it is performing useful work again)

 \item {\bf Availability}: the percentage of time that a component
within a system or the system itself is up and performing useful work
\end{itemize}

With these two metrics and our informal definition of availability we
can now write an approximate equation that yields a useful number for
availability:

\begin{center}
\[ Availability \approx \frac{MTBF}{MTBF+MTTR} \]
\end{center}

Note that this definition of availability assumes that the amount of
time taken for an operation on the component or system being measured
is very small when compared with the MTTR.  Examining this formula, it
is apparent that there are two approaches one can take to maximizing
availability:

\begin{itemize}
\item increase MTBF

\item decrease MTTR
\end{itemize}

\subsection{Redundancy}

The availability of a system can be increased intuitively by
duplicating parts of the system so that if one of the parts fails, a
duplicate is still available to continue providing service.  There are
two basic techniques that exploit this intuitive notion:

\begin{itemize}

 \item {\bf peer components with voting}: in this approach, each
component processes an operation and produces a result.  A voting
entity then examines the results produced by all components and
selects the result produced by a majority of the components.  Triple
Mode Redundancy (TMR) is an example of this kind of system.  This
system masks some failures of redundant components.

 \item {\bf Primary Copy}: in this approach, one component is viewed
as the {\em primary} component.  The primary is the only component
that actually processes operations and produces results.  The rest of
the duplicate components are useds as {\em backup} components.  The
backups receive update information from the primary that allow it to
keep its state up to date with that of the primary.  If a primary
fails, one of the backups will take over the job of being the primary;
if this take over can be done quickly, then this results in a short
MTTR.  If the client process that produces the operations does not
timeout before the takeover has been completed, then the failure will
have been successfully masked.

\end{itemize}

Primary copy is a popular approach due to its lower cost during normal
operation (only the primary does the operation, whereas in peer
components with voting all components do the operation, followed by a
voting phase).  There are two types of redundancy that Primary Copy
can provide, depending upon the implementation:

\begin{itemize}
 \item {\bf Static Redundancy}: statically assign a primary component
and a set of backups for that primary.  This is costly, in that if a
primary fails, the system continues with one less backup for that
primary; there is no way to replace that backup without repairing the
primary that failed.

 \item {\bf Dynamic Redundancy}: have a pool of components, one of
which being a primary, and some subset being backups for that primary.
In the event of a failure, dynamically redistribute the load on the
remaining components to free up a component that can then take the
place of the lost backup.  Less total components are needed in this
case to provide a given level of reliability.  However, the load
redistribution leads to a degradation in the overall performance of
the system.  This can be tuned by setting ``minimum acceptable
service'' thresholds that prevent load redistribution from being
attempted after a certain point.

\end{itemize}

To see how Primary Copy helps availability, we need to determine what
the MTBF is for the system.  It is going to depend upon the MTBF's of
both the primary and the backups and is going to depend upon how fast
a primary can be replaced by a backup (this is the MTTR).  If we
consider a system that has just a primary and a backup, and we assume
that the take over time is short enough that the client does not time
out and consider the primary failure as a failure (i.e., the primary
failure can be masked), then the MTBF of the (primary,backup) pair can
be written:

\begin{center}
\[ MTBF_{pair} \approx \frac{(MTBF_{component})^{2}}{MTTR_{component}} \]
\end{center}

An example system of this type is the duplex disk arrangement used in
Tandem computers described in the paper by Jim Gray.  A read operation
can read either disk, while a write operation must write both disks.
If a disk fails, then a write operation will fail (since both disks
must be written) unless one sets things up so that when there is only
one disk operating, writes need only be done to the one operating
disk.  This would mean that the failure must be detectable to enable
this switch of what happens on a write operation.  For the read
operation, we need also to assume that if a disk fails, then it just
stops, as opposed to continue running in an unpredictable state
(spewing out garbage data, for example).

In the paper by Jim Gray, it was stated that in 1987 Tandem reported a $MTBF_{disk}$ of 50,000 hours and a
$MTTR_{disk}$ of 5 hours.  If we apply our above equation to calculate the MTBF of a duplex disk pair,
we have:

\begin{center}
\[ MTBF_{pair} = \frac{(MTBF_{disk})^{2}}{MTTR_{disk}} = \frac{(\mbox{50,000 hours})^{2}}{\mbox{5 hours}} \approx \mbox{60,000 years!} \]
\end{center}

In practice, however, Tandem found that the $MTBF_{pair}$ was more like 1300 years.  Why is there such
a discrepancy?  Several factors were mentioned in the paper, including:

\begin{itemize}
\item the repairman may occasionally replace the wrong disk drive (i.e., not the disk that failed).

 \item disk faults are not uniformly distributed; a brand new disk may
suffer ``infant mortality'' type failures, and some older disks are
still in use (which have lower MTBFs than the 50,000 hour MTBF of new
disks available in 1987)

 \item the 50,000 hour $MTBF_{disk}$ is just an average number---not
all vendor's disks have this MTBF
\end{itemize}

Note that even with a $MTBF_{pair}$ equal to 1300 years, if one has a
system with a large number of these duplex disks, such as one with
1300 duplex disks, the $MTBF_{sys}$ becomes 1 year.  Thus even with
very reliable hardware, a large system can become unreliable fairly
quickly.

\section{At-Most-Once Actions}

At the very end of the lecture, the idea of At-Most-Once Actions was
introduced as a useful concept for use in a distributed system and in
particular one that uses Primary Copy Replication.  The basic idea is
similar to the At-Most-Once Messages studied during the lectures on
communications systems.  In this case, an action either gets done
exactly once, or a failure is reported to the client, in which case
the client does not know for sure whether any of the action's updates
took place or not.  A simple specification was presented for a module
with At-Most-Once semantics, with discussion to continue during the
next lecture.

\begin{verbatim}
MODULE AMO [                                    % At Most Once Transaction
       V,                                       % Value
       S WITH {s0: () -> S },                   % State
       A WITH { meaning: A -> S -> (V,S) ]      % Action
       
VAR s := S$s0()

APROC Do(a) -> V RAISE{failed} =
<<    
      VAR v,s' | (v,s') = a.meaning(s) =>
                 s := s'; BEGIN RET v [] RAISE failed END
      [] RAISE failed
>>

END AMO
\end{verbatim}

\end{document}