%% Scribe notes for Lecture 15.
%
% This file uses:
%    psadobe.tex  - for including postscript figures.
%    ch.ps	  - A postscript picture of a channel
%

\documentstyle[12pt,pocs-header]{article}
% cfj@jj.lcs.mit.edu, shep@lcs.mit.edu
\Scribes{Chris Joerg and Tim Shepard}
\Lecturer{Butler Lampson}
\LectureNumber{15}
\LectureDate{November 5, 1990}

\include{psadobe} % used to include figures in this document

%%%% Some more macros, mostly for including figures:
\def\illustrationfileprefix{/bk/cfj/6826/}

% \fig{name} -- reference a figure.
\def\fig#1{Figure~\ref{fig:#1}}

%\illustration{filename}
\def\illustration#1{\begin{center}\psadobe{\illustrationfileprefix#1.ps}%
                       \end{center}}

% \illfigure{filename}{Title}: Standard figure.
\def\illfigure#1#2{\begin{figure}[htbp]
			\illustration{#1}
			\caption{#2}
			\label{fig:#1}
			\end{figure}}

%% Put text on page with figures, even if text is at least .25 of page
\def\topfraction{.75}
\def\bottomfraction{.5}
\def\textfraction{.25}
\def\floatpagefraction{.75}
\def\dbltopfraction{.75}
\def\dblfloatpagefraction{.5}


%%%%%%%%%%%%%%%%%%%%%%%%%



\begin{document}
\MakeScribeTop

\section{Administrivia}

\begin{table}[h]
\centering
\begin{tabular}{|c|l|} \hline
\multicolumn{1}{|c|}{Handout} &
\multicolumn{1}{c|}{Title} \\ \hline
28 & Reading: Implementing Remote Procedure Calls \\
29 & Reading: Performance of Firefly RPC \\
30 & Reading: Lightweight Remote Procedure Call \\
31 & Solutions to Problem Set \# 3  \\
32 & Communication Primitives \\
-- & Graded Problem Set 3 \\
\hline
\end{tabular}
\end{table}


\section{ Communication in Distributed Systems}

We would like to build fault tolerant systems, even though the pieces that
make up the system are prone to failures.  There are three types of
mechanisms used in the system: computation, storage and communication.  Today
we will focus on communication.

Our basic strategy for building a fault tolerant system has two parts.
First we must be able to detect when something goes wrong. Second, we must
be able to redo what went wrong.  To do this we need some sort of
redundancy.  For example, in communication we need to be able to detect if a
message is correct.  This is done by adding redundant bits to the message.
Parity bits or a Cyclic Redundancy Code (CRC) checksum are two ways this
could be done; in general these added bits are called a checksum.  We must
choose a checksum that will make it highly unlikely that a message which has
been garbled will have the correct checksum.  The probability of a bad
message having a valid checksum must be so low that we can consider it to be
zero.\footnote{ If we want to provide Secure communication, then the
checksum should make it unlikely that some evil entity who purposely altered
the message could create one with a valid checksum.}

There are three error models that are commonly used.
\begin{itemize}
\item{\bf Single Bit Errors} In this model each bit has some probability of
being in error that is independent of what happens to other bits.  With this
model we want a checksum designed to detect upto $n$ independent single bit
errors.
\item{\bf Burst Errors}  This assumes that errors are due to some sort of noise
that lasts for more than one bit. To catch these errors the checksum should
always be invalid if all errors occurred within a span of $n$ bits.  The $n$
in this case is usually much larger than $n$ in the previous case.
A common example is an error correcting code used with disks.
\item{\bf Random Transformation}  In this model an error will randomly
transform the bits of the message.
To catch these types of errors, we want a checksum that gives a random
message a low probability of having a valid checksum.
\end{itemize}

We will assume that we are using a checksum that can reliably detect all
corrupted messages.  We do not assume that all messages sent are
received, or that they are received only once; our implementation must
deal with these problems.


\section{ Specification of a FIFO Communication Channel}

The specification for a FIFO (first-in-first-out) communication channel is
given on page 1 of handout 32. The operations are Push which adds a message
to the the end of a buffer, and Get which removes a message from the front
of the buffer.
There are four options on what effects crashes can have. The first, which is
not given in the handout, is that nothing happens to the channel.  This, of
course, would be very hard to implement.  Another possibility is that a
crash causes the entire buffer to be emptied.  This is also very hard to do.
A more realistic model is that a crash on the sender causes items on the
tail of the queue to be lost, and a crash on the receiver causes items on
the head of the queue to be lost. In this case crashes can be modeled by an
initial and/or tail portion of the buffer being lost.
The most realistic (and pessimistic) model is that a crash can cause the
buffer to lose any subsequence of messages.  This could happen due to
multiple crashes on a node.

There are two possible communication mechanisms, both of which are given on
page 2 of the handout.  The first models a wire and allows messages to be
lost (due to errors), but will never reorder or duplicate a message.
  The second model is a weaker model that models a network.  No erroneous
messages are received, but messages could be duplicated,
reordered or lost.\footnote{The code for Put should also allow the
possibility that a message is lost:\\
APROC Put (nm) = $<<$ n := n $++$ nm [] SKIP $>>$ }
Pragmatically, we still expect that each message sent is received only once,
and that unless 2 messages are sent very close in time, the messages will be
received in the same order they were sent.  Also, note that there is
information in the loss of a message; this may indicate congestion in the
network. 


\section{An Implementation: IDChannel}
This implementation is unrealistic in that it assumes that the processors do
not crash, and that we can tolerate unbounded message overhead and use
unbounded state in the processor.  However, this implementation does allow
for the bad behavior of the NET communication channel.

This implementation divides the buffer into four sections: sender's buffer
(S.b), staging area (S.sna), network, and Receiver's buffer (S.b).  These
components are as shown in \fig{ch}.
Communication begins when the sender PUTs messages into its buffer (S.b).
When the staging area (S.sna) is empty, the message at the front of this
buffer is moved into the staging area.  The message is given an ID which is
not currently in the senders ID Set (S.ids), and this ID is added to the set.
When a message is in the staging area it is continually sent out into the
network. The Network is simply the NET model described above.  So the
messages that enter the network may be lost, reordered or duplicated.
When a message exits the network the receiver examines the message's ID. If
the ID is already in its ID Set (R.ids) the message will be ignored.  If the
message is not there then the ID is added to the set and the message is
placed in the receiver's buffer.\footnote{Typo in the handout:  the last
PROC should be called RcvrS, not RcvrR.}
The receiver will also periodically generate acknowledgment messages for
all of the IDs in its ID Set.  (The network is modeled as two separate
networks, one in each direction.)
When the sender gets an acknowledgment for the message currently in its
staging area, that message will be removed from the staging area.
Although this is not efficient, it correctly implements the COMM
specification.
\illfigure{ch}{}

To see that the implementation is correct we need an abstraction function.
The abstraction function depends upon the state of the staging area. If the
staging area contains a message whose ID is not in the receiver's ID set then
the AF is the receiver's buffer plus message in staging area plus sender's
buffer (R.b ++ s.sna + S.b).  Otherwise it is simply R.b + S.b .

The only actions which change the value of the the AF are the Put and Get
functions, and it is easy to that these have the expected effect on the AF.
To prove the module correct we must show that the rest of the actions leave
the value of the AF unchanged.
As as side note, notice that computing this AF requires knowing the state of
two distributed nodes at the same time.  This would be very hard to
implement, but this is not a concern since the AF is used only for the
proof, and will not be implemented.

\subsection {Allowing Crashes}

We will now extend this model to allow for crashes.
We assume that a crash causes the values in s.b and r.b to be lost, but does
not effect the ID sets.  This causes the head or tail of the buffer to be
lost, and this is allowed by the specification.  But this could also cause
packets to get out of order and this is not allowed:\\
Assume that message 2 is in the network and has not yet been received when
the sender crashes. The sender then restarts and sends message 3.  It is
possible that message 3 would be received before 2, thus violating the FIFO
property.  The solution to this is to order the IDs and have the receiver
only accept a message if its ID is greater than any ID in its ID Set.
In the above example, if message 2 arrived after 3 it would simply be
ignored. This is allowable since message 2 would simply be part of the
message sequence that was lost in the crash.  This implementation is
contained in the OrderedIDChannel module of the handout (ignoring the
parts which are double underlined.)

It is easy to modify this scheme so that multiple messages can be
transmitted at the same time.  This is especially useful to do if the round
trip transit time is long. (A long round trip transit time could be due
either to a long one-way transit time, or to slow message processing by the
receiver.)
If we do this it is possible to send message 2 then 3, and for 3 to arrive
first.  We must prevent the later message from being lost.  We can do this
by having the sender order the IDs consecutively, and having the receiver
only accept a message if its ID consecutively follows the previous message's
ID.
Crashes are still a problem, because if the sender crashes there may be a
gap in the ID's of the transmitted messages.  This is fixed by having the
sender send a special crash message which the receiver always accepts
(if it has not yet received that message).  This crash message will
have an ID.  The next ID that the receiver expects will be the ID which
consecutively follows this one, and that is the ID that the sender will send
next.  Thus this resynchronizes the receiver to the sender.


\section {Three Way Handshaking}

The basic notion is to use a combination of stable and volatile
storage to avoid having to do a transaction with stable storage for
each message.    In in the three way handshake, the sender sends a
message to the receiver asking for an ID.  The receiver maintains two
sets of IDs: {\it NotReceived\/} and {\it NotHandedOut\/}.  The set
{\it Rcvd\/} from before is then given by subtracting these two sets
from the set of all possible IDs.   The NotHandedOut set is kept in
stable storage and the NotReceived set is kept in volatile storage.
>From time to time, the receiver removes some IDs from the stable NotHandedOut
set and keeps them in volatile storage so that they can be dispensed
quickly.  On crashes, these IDs will be lost.

When a sender asks for an ID, the receiver removes one from the cache
of the NotHandedOut set places it in the NotReceived set, and sends it
to the sender.  The receiver accepts only messages which arrive with
IDs which are in the NotReceived set.  When a message arrives, its ID
is removed from the NotReceived set.

The complete exchange consists of 5 packets.  It is called the ``Three
Way Handshake'' because three packets are sent in the first three
fifths of the protocol.


\begin{verbatim}
                        Three Way Handshake
                        -------------------

Sender                                    Receiver
------                                    -------- acking  NotReceived
make s             -----                           ------  -----------
 (an id)                ----- (s)                  {}      {}
                             ---------->
                                   -----  make r
                       (s,r) ------                {}      {r}
                   <---------
sna:= (b.head, s,r)
b := b.tail        -----
                        -----  (s,r,m)
                             ---------->
                                   -----           {(r,s)} {}
                    (s,r,ack) -----
                   <----------
                   -----
                        ----- (s,r,close)
                             ------------>
                                                   {}      {}

\end{verbatim}

\section{Practical Issues}

Real protocols reuse IDs. (Never reusing an ID might require an
impractically large ID space.)   In order to reuse IDs, some mechanism
is needed to insure that an old use of an ID will not interfere with a new 
use of an ID.   There are two basic methods.  One is to wait some
amount of time which you believe is long enough to ensure that the
message will not still be held by the network.   The other is to
encrypt all messages and to change keys when the ID space needs to be
recycled.

To prevent a sender from overrunning a receiver's resources, some sort
of flow control mechanism is needed.  Typically, a three way handshake
is used to set up connection ids, and then a sliding window is used to
control what the sender sends.  To implement the sliding window, the
receiver returns two numbers to the sender: the earliest id not yet
received (ack), and one past the last ID which may be sent (win).
When ack is equal to win, the window has ``gone to zero'' and the
sender is not allowed to send anything.  The id space for ack and win
typically may cycle on a single connection.
This is shown in the following figure.

\newpage
\begin{verbatim}

               Sliding Window


        20                                    30
        ack                                   win
         |---------------------+---------------|
                 sna              OK to send

\end{verbatim}



\end{document}
