%% Scribe notes for Lecture 15. % % This file uses: % psadobe.tex - for including postscript figures. % ch.ps - A postscript picture of a channel % \documentstyle[12pt,pocs-header]{article} % cfj@jj.lcs.mit.edu, shep@lcs.mit.edu \Scribes{Chris Joerg and Tim Shepard} \Lecturer{Butler Lampson} \LectureNumber{15} \LectureDate{November 5, 1990} \include{psadobe} % used to include figures in this document %%%% Some more macros, mostly for including figures: \def\illustrationfileprefix{/bk/cfj/6826/} % \fig{name} -- reference a figure. \def\fig#1{Figure~\ref{fig:#1}} %\illustration{filename} \def\illustration#1{\begin{center}\psadobe{\illustrationfileprefix#1.ps}% \end{center}} % \illfigure{filename}{Title}: Standard figure. \def\illfigure#1#2{\begin{figure}[htbp] \illustration{#1} \caption{#2} \label{fig:#1} \end{figure}} %% Put text on page with figures, even if text is at least .25 of page \def\topfraction{.75} \def\bottomfraction{.5} \def\textfraction{.25} \def\floatpagefraction{.75} \def\dbltopfraction{.75} \def\dblfloatpagefraction{.5} %%%%%%%%%%%%%%%%%%%%%%%%% \begin{document} \MakeScribeTop \section{Administrivia} \begin{table}[h] \centering \begin{tabular}{|c|l|} \hline \multicolumn{1}{|c|}{Handout} & \multicolumn{1}{c|}{Title} \\ \hline 28 & Reading: Implementing Remote Procedure Calls \\ 29 & Reading: Performance of Firefly RPC \\ 30 & Reading: Lightweight Remote Procedure Call \\ 31 & Solutions to Problem Set \# 3 \\ 32 & Communication Primitives \\ -- & Graded Problem Set 3 \\ \hline \end{tabular} \end{table} \section{ Communication in Distributed Systems} We would like to build fault tolerant systems, even though the pieces that make up the system are prone to failures. There are three types of mechanisms used in the system: computation, storage and communication. Today we will focus on communication. Our basic strategy for building a fault tolerant system has two parts. First we must be able to detect when something goes wrong. Second, we must be able to redo what went wrong. To do this we need some sort of redundancy. For example, in communication we need to be able to detect if a message is correct. This is done by adding redundant bits to the message. Parity bits or a Cyclic Redundancy Code (CRC) checksum are two ways this could be done; in general these added bits are called a checksum. We must choose a checksum that will make it highly unlikely that a message which has been garbled will have the correct checksum. The probability of a bad message having a valid checksum must be so low that we can consider it to be zero.\footnote{ If we want to provide Secure communication, then the checksum should make it unlikely that some evil entity who purposely altered the message could create one with a valid checksum.} There are three error models that are commonly used. \begin{itemize} \item{\bf Single Bit Errors} In this model each bit has some probability of being in error that is independent of what happens to other bits. With this model we want a checksum designed to detect upto $n$ independent single bit errors. \item{\bf Burst Errors} This assumes that errors are due to some sort of noise that lasts for more than one bit. To catch these errors the checksum should always be invalid if all errors occurred within a span of $n$ bits. The $n$ in this case is usually much larger than $n$ in the previous case. A common example is an error correcting code used with disks. \item{\bf Random Transformation} In this model an error will randomly transform the bits of the message. To catch these types of errors, we want a checksum that gives a random message a low probability of having a valid checksum. \end{itemize} We will assume that we are using a checksum that can reliably detect all corrupted messages. We do not assume that all messages sent are received, or that they are received only once; our implementation must deal with these problems. \section{ Specification of a FIFO Communication Channel} The specification for a FIFO (first-in-first-out) communication channel is given on page 1 of handout 32. The operations are Push which adds a message to the the end of a buffer, and Get which removes a message from the front of the buffer. There are four options on what effects crashes can have. The first, which is not given in the handout, is that nothing happens to the channel. This, of course, would be very hard to implement. Another possibility is that a crash causes the entire buffer to be emptied. This is also very hard to do. A more realistic model is that a crash on the sender causes items on the tail of the queue to be lost, and a crash on the receiver causes items on the head of the queue to be lost. In this case crashes can be modeled by an initial and/or tail portion of the buffer being lost. The most realistic (and pessimistic) model is that a crash can cause the buffer to lose any subsequence of messages. This could happen due to multiple crashes on a node. There are two possible communication mechanisms, both of which are given on page 2 of the handout. The first models a wire and allows messages to be lost (due to errors), but will never reorder or duplicate a message. The second model is a weaker model that models a network. No erroneous messages are received, but messages could be duplicated, reordered or lost.\footnote{The code for Put should also allow the possibility that a message is lost:\\ APROC Put (nm) = $<<$ n := n $++$ nm [] SKIP $>>$ } Pragmatically, we still expect that each message sent is received only once, and that unless 2 messages are sent very close in time, the messages will be received in the same order they were sent. Also, note that there is information in the loss of a message; this may indicate congestion in the network. \section{An Implementation: IDChannel} This implementation is unrealistic in that it assumes that the processors do not crash, and that we can tolerate unbounded message overhead and use unbounded state in the processor. However, this implementation does allow for the bad behavior of the NET communication channel. This implementation divides the buffer into four sections: sender's buffer (S.b), staging area (S.sna), network, and Receiver's buffer (S.b). These components are as shown in \fig{ch}. Communication begins when the sender PUTs messages into its buffer (S.b). When the staging area (S.sna) is empty, the message at the front of this buffer is moved into the staging area. The message is given an ID which is not currently in the senders ID Set (S.ids), and this ID is added to the set. When a message is in the staging area it is continually sent out into the network. The Network is simply the NET model described above. So the messages that enter the network may be lost, reordered or duplicated. When a message exits the network the receiver examines the message's ID. If the ID is already in its ID Set (R.ids) the message will be ignored. If the message is not there then the ID is added to the set and the message is placed in the receiver's buffer.\footnote{Typo in the handout: the last PROC should be called RcvrS, not RcvrR.} The receiver will also periodically generate acknowledgment messages for all of the IDs in its ID Set. (The network is modeled as two separate networks, one in each direction.) When the sender gets an acknowledgment for the message currently in its staging area, that message will be removed from the staging area. Although this is not efficient, it correctly implements the COMM specification. \illfigure{ch}{} To see that the implementation is correct we need an abstraction function. The abstraction function depends upon the state of the staging area. If the staging area contains a message whose ID is not in the receiver's ID set then the AF is the receiver's buffer plus message in staging area plus sender's buffer (R.b ++ s.sna + S.b). Otherwise it is simply R.b + S.b . The only actions which change the value of the the AF are the Put and Get functions, and it is easy to that these have the expected effect on the AF. To prove the module correct we must show that the rest of the actions leave the value of the AF unchanged. As as side note, notice that computing this AF requires knowing the state of two distributed nodes at the same time. This would be very hard to implement, but this is not a concern since the AF is used only for the proof, and will not be implemented. \subsection {Allowing Crashes} We will now extend this model to allow for crashes. We assume that a crash causes the values in s.b and r.b to be lost, but does not effect the ID sets. This causes the head or tail of the buffer to be lost, and this is allowed by the specification. But this could also cause packets to get out of order and this is not allowed:\\ Assume that message 2 is in the network and has not yet been received when the sender crashes. The sender then restarts and sends message 3. It is possible that message 3 would be received before 2, thus violating the FIFO property. The solution to this is to order the IDs and have the receiver only accept a message if its ID is greater than any ID in its ID Set. In the above example, if message 2 arrived after 3 it would simply be ignored. This is allowable since message 2 would simply be part of the message sequence that was lost in the crash. This implementation is contained in the OrderedIDChannel module of the handout (ignoring the parts which are double underlined.) It is easy to modify this scheme so that multiple messages can be transmitted at the same time. This is especially useful to do if the round trip transit time is long. (A long round trip transit time could be due either to a long one-way transit time, or to slow message processing by the receiver.) If we do this it is possible to send message 2 then 3, and for 3 to arrive first. We must prevent the later message from being lost. We can do this by having the sender order the IDs consecutively, and having the receiver only accept a message if its ID consecutively follows the previous message's ID. Crashes are still a problem, because if the sender crashes there may be a gap in the ID's of the transmitted messages. This is fixed by having the sender send a special crash message which the receiver always accepts (if it has not yet received that message). This crash message will have an ID. The next ID that the receiver expects will be the ID which consecutively follows this one, and that is the ID that the sender will send next. Thus this resynchronizes the receiver to the sender. \section {Three Way Handshaking} The basic notion is to use a combination of stable and volatile storage to avoid having to do a transaction with stable storage for each message. In in the three way handshake, the sender sends a message to the receiver asking for an ID. The receiver maintains two sets of IDs: {\it NotReceived\/} and {\it NotHandedOut\/}. The set {\it Rcvd\/} from before is then given by subtracting these two sets from the set of all possible IDs. The NotHandedOut set is kept in stable storage and the NotReceived set is kept in volatile storage. >From time to time, the receiver removes some IDs from the stable NotHandedOut set and keeps them in volatile storage so that they can be dispensed quickly. On crashes, these IDs will be lost. When a sender asks for an ID, the receiver removes one from the cache of the NotHandedOut set places it in the NotReceived set, and sends it to the sender. The receiver accepts only messages which arrive with IDs which are in the NotReceived set. When a message arrives, its ID is removed from the NotReceived set. The complete exchange consists of 5 packets. It is called the ``Three Way Handshake'' because three packets are sent in the first three fifths of the protocol. \begin{verbatim} Three Way Handshake ------------------- Sender Receiver ------ -------- acking NotReceived make s ----- ------ ----------- (an id) ----- (s) {} {} ----------> ----- make r (s,r) ------ {} {r} <--------- sna:= (b.head, s,r) b := b.tail ----- ----- (s,r,m) ----------> ----- {(r,s)} {} (s,r,ack) ----- <---------- ----- ----- (s,r,close) ------------> {} {} \end{verbatim} \section{Practical Issues} Real protocols reuse IDs. (Never reusing an ID might require an impractically large ID space.) In order to reuse IDs, some mechanism is needed to insure that an old use of an ID will not interfere with a new use of an ID. There are two basic methods. One is to wait some amount of time which you believe is long enough to ensure that the message will not still be held by the network. The other is to encrypt all messages and to change keys when the ID space needs to be recycled. To prevent a sender from overrunning a receiver's resources, some sort of flow control mechanism is needed. Typically, a three way handshake is used to set up connection ids, and then a sliding window is used to control what the sender sends. To implement the sliding window, the receiver returns two numbers to the sender: the earliest id not yet received (ack), and one past the last ID which may be sent (win). When ack is equal to win, the window has ``gone to zero'' and the sender is not allowed to send anything. The id space for ack and win typically may cycle on a single connection. This is shown in the following figure. \newpage \begin{verbatim} Sliding Window 20 30 ack win |---------------------+---------------| sna OK to send \end{verbatim} \end{document}