\documentstyle[12pt,pocs-header]{article}
\Scribe{Bill Kruger}
\Lecturer{Butler Lampson}
\LectureNumber{12}
\LectureDate{October 23, 1991}

\begin{document}
\MakeScribeTop

\section{Handouts}

\begin{center}
\begin{tabular}{|c|l|} \hline
\multicolumn{1}{|c|}{Handout} &
\multicolumn{1}{c|}{Title} \\ \hline
26 & Problem Set 5 \\
27 & Butler Lampson, "Sequential Transactions with Caching" \\
28 & Mendel Rosenblum and John Ousterhout, \\
   & "The Design and Implementation of a Log-Structured File System", \\
   & {\it Operating Systems Review}, volume 25, number 5, pp 1-15, \\
   & October 1991 \\
\hline
\end{tabular}
\end{center}


\section{Sequential Transactions Revisited}

Transactions were once again examined.  Recall that a transaction was a
mechanism for making a complex set of operations atomic.  In Handout 10,
a log-based implementation was given.  In this implementation, a number
of {\bf actions} could be applied to the state of the system, yielding a
new state and a value.  A sequence of these actions could be grouped together
into what was termed a {\bf transaction}, such that either all of the actions
were done as a single logical operation or none of them were done.  To implement
each action, a series of {\bf updates} were constructed which, when applied
to the state, transformed the state to a new state according to the
meaning of the action.  Each update could be applied atomically to the state.
The sequence of updates which made up an action were
written to a log, such that in the face of crashes the log could be replayed
to recover the state of the system before the crash.  However, updates that
were part of a non-committed transaction were distinguished from those of
a committed transaction in order to implement the "all or none" property of
the transaction.


There are two relevant types of atomicity:
\begin{itemize}
\item Atomicity in the presence of concurrent clients

\item Atomicity in the presence of failures.
\end{itemize}

This lecture discussed the specification of a sequential transaction
system with logging and caching, which provided atomicity in the
presence of failures only (the next lecture will discuss providing
atomicity in the presence of concurrent clients), and a more
acceptable level of performance compared with our previous discussion
of sequential transactions.


\section{Motivation}

In the previous discussions of sequential transactions (from Handout 10),
the presented specification and implementation already provided us with
atomicity in the presence of failures.  One might then question the need
for further study.  Upon close examination of the system presented in
Handout 10, however, one discovers a couple of impracticalities which
affect real-world performance:

\begin{itemize}
\item Reading Data - in the old implementation, one only had access
to the stable state.  Updates were only available
for reading after a commit.  Ideally, we would
like to have access to the most recent data (i.e.,
the last value written).

\item Commit - the old implementation always updated the Stable State
immediately upon invocation.  This is a costly
process.  It would be faster if we could decouple
commit from immediately applying all updates of
a single commit and instead just log the fact that the commit took
place.  By deferring the application of the updates, we can potentially
aggregate many updates of a single location into one application.
\end{itemize}

These inadequacies prevent us from implementing a transaction system
that offers acceptable performance.  It is clear that a new implementation
is necessary.

\section{Sequential Transactions with Caching}

The basic ideas for implementing a sequential transaction system with caching
are:

\begin{itemize}
\item We have a Stable State (SS), a Log (l) and a Volatile State (VS).
We don't want to represent the entire Volatile State- this is most
likely too big.  Instead, we can synthesize what we need.
In particular, we'll construct the Volatile State from the Stable
State plus some stuff that we'll keep in the cache.  Put another way,
an action will update the cache directly, such that the current state
of recently accessed parts of the base will always be reflected in the cache
(allowing quick access).

\item Since the cache will be of some finite size, we can't allow the cached
updates to grow without bound.  This means that we'll have to flush the
cached updates when it is determined that the cache is full.  Since the
updates are written to the cache directly before commit, this
means that the Stable Base may include actions not yet committed.

\item We want to defer updates for as long as possible, as we can aggregate
many updates of a single disk block and then just apply the most current one
to the Stable Base.  As mentioned above, however, by flushing the cache as
it fills, the Stable Base can contain updates not yet committed.  Recall that
the Stable State is equal to the Stable Base plus all committed updates.
To restore the Stable State from the Stable Base (which contains committed and
uncommitted updates), we'll need to calculate an "undo" action for each action,
so that any uncommited actions can be undone:

\begin{center}
SS = SB + All updates - uncommitted updates (undo)
\end{center}

\item We want to truncate the log of entries whose updates have been applied to
the stable base in order to reuse log space.  We'd like to truncate entries whether
or not the transaction responsible for them has committed (as we've decoupled commit
from the application of updates)- so long as they've been applied to the SB, we can
remove them from the log.

\item To enable recovery of the base in case it gets corrupted, we'll need to save
discarded entries (those truncated in the above process) to a Permanent Log (PL).
The PL is often kept on magnetic tape.
\end{itemize}

\subsection{Failure Types}

There are four levels of failure within this transaction subsystem, listed in order
of increasing severity:

\begin{itemize}
\item {\bf Transaction Abort}: the client aborts an in-progress transaction.  This is not
really a failure, but an expected behavior of the client or.

\item {\bf Crash}: this results in the loss of the volatile state and aborts all active
uncommitted transactions.  Committed transactions are recoverable from the Stable
Log and Undo Log.

\item {\bf Media Failure}: this results in the loss of the Stable Base, as well as the
Stable Log and Stable Undo Log.  The Stable Base is recoverable from the Permanent
Log.

\item {\bf Catastrophe}: the Permanent Log and the Stable Base are lost.
\end{itemize}

In practice, further steps are taken to handle failures, such as periodically taking
snapshots of the Stable Base and saving multiple copies to magnetic tape.

\section{Implementation: LogAndCache}

Handout 27 presents an implementation abstraction for sequential transactions with
logging and caching.  The rest of the lecture highlighted the important features of
the parts of the implementation.  The implementation has the following features:

\begin{itemize}
\item An action is turned into a sequence of updates that can be applied atomically.

\item Updates are applied to single disk blocks (which can be written atomically).

\item Undo Actions: for every action there exists a corresponding "undo" action which will
reverse the effect upon the base of that action.

\item Actions are written to the log atomically.

\item A cached update, or W, represents the new contents of a disk block.
\end{itemize}

The implementation is optimized for both crash recovery (due to logging) and for
speed of access (due to caching).  There are now two logs, each with two parts:

\begin{itemize}
\item Stable Log (SL) and Volatile Log (VL)

\item Stable Undo Log (SUL) and Volatile Undo Log (VUL)
\end{itemize}

An action is converted to a sequence of updates (U), which are logged to the VL.
The updates are also applied directly and immediately
to the cache (W) before commit, so that the current state is immediately accessible.
For each action, an Undo Action is computed and written to the Volatile Undo Log (VUL).
Note that the undo log contains the action, and not the sequence of updates into which the
action is converted- this is done for efficiency, as we normally don't expect to have to
undo actions, and so we can defer converting them to updates if and when we apply them.
Committing a transaction first moves its updates (U) from the VL to the SL and its undo
actions from the VUL to the SUL.  It then logs the commit in the SL and empties the
SUL (since the action is committed, we no longer need to be able to undo it).


As noted in Handout 27, the U's and W's have the following properties:

\begin{itemize}

\item Adding an action's associated updates (Us) to the log is an atomic operation.

\item Applying a cached update (W) to the SB is an atomic operation.

\item Converting a U to a W is cheap, and applying a W is cheap.

\item U's are idempotent (the hiccup property as discussed in Handout 10).

\item W's commute
\end{itemize}

We can write an abstraction function to {\it SequentialTransaction} from Handout 10:

\begin{center}
SS = SB+SL-SUL
\end{center}
\begin{center}
VS = SS+VL-VUL
\end{center}

\subsection{Implementation Routines}

The important aspects of many of the implementation routines were examined:
\begin{itemize}

\item {\bf DoOrUndoA}: We first construct the volatile state of the system (vb) from the 
SB and the cache by applying all cached updates (W's) to the stable base.  We then
convert the action to its sequence of updates based upon the volatile base.  If the
action is not an undo action, then we compute an undo action.  If we were performing
an undo action, then we need to log the fact that this undo action has actually been performed
(this assures idempotence of undo actions).
The updates are then logged to the VL, the undo action is logged to the VUL and the
updates are converted to cache updates (W's) and applied to the cache.  As specified,
the entire procedure is done atomically.

\item {\bf Commit}: Commit first forces all updates from the VL to the SL and all undo
actions from the VUL to the SUL.  It then logs a "commit" marker to the SL and
empties the SUL, these two operations done atomically.

\item {\bf Abort}: Abort undoes all uncommited updates by applying the undo actions in the
undo logs.  It then forces all updates in the VL and VUL to the
SL and SUL, respectively.

\item {\bf Crash}: The idea of Crash is that the volatile state has been lost.  This includes
the VL, the VUL and the Cache (C).  Recovery then consists of replaying the SL to reconstruct
the cache as it existed before the crash, and then applying any undo actions contained in the
SUL to undo all uncommited updates.

\item {\bf Undo:} Undo concatenates the SUL with the VUL and then applies the undo actions
contained in this concatenated log {\it in reverse order}.  The undo actions must be
applied in order, and since each was computed based upon the volatile base that existed
after the application of the previous undo action's associated action, this requires a
reverse chronological order.  Undo skips over previously applied undo actions until one
is encountered which has not yet been applied.  It is then applied by invoking DoOrUndoA(),
which in addition to converting the undo action to a sequence of updates, also marks the
undo action as applied in the VUL.  In this way undo actions are only applied once, assuring
idempotence of the undo logs.

\item {\bf Redo}: Redo rebuilds the cache as it was before the crash, thus containing volatile
uncommitted updates.  This is faster than rebuilding and applying all updates to the SB, and
uncommitted updates can be gotten rid of via Undo().

\item {\bf Flush}: Before applying a cached update to the SB, Flush makes sure that the corresponding
action is in the SL.

\item {\bf Truncate}: Truncate takes a contiguous portion of the SL that has already been applied
to the SB and removes it from the SL, appending it to the PL.
\end{itemize}

\end{document}