% here is lecture 16.  there are a few postscript files of the
% diagrams that i will mail following this.  you need to name these
% postscript files a particular name - which is written at the
% beginning of the file.			-suwei

%------------------------------------------------------
% to get hardcopy of this lecture, you need the following files (plus the
% normal latex and tex base directories):
%     lecture.tex:     latex source file
%     macros.tex:      figure macros
%     psfig.tex:       postscript figure (psfig) macro definition
%     sort.idraw:      first figure, created with idraw
%     security.idraw:  second figure, created with idraw
%     proof.idraw:     third figure, created with idraw
%     pocs-header.sty: Principles of Computer Systems lecture latex header
%
% run the following programs:
% latex lecture
%   -> Creates lecture.dvi, plus latex intermediate files
% latex lecture
%   -> Run a second time to get cross-references right
% dvi2ps lecture > lecture.ps
%   -> Merges the figures with the text, can also pipe to lpr
%
\documentstyle[12pt,pocs-header]{article}
\Scribes{Scott Sikora and Suwei Wu}
\Lecturer{Butler Lampson}
\LectureNumber{16}
\LectureDate{November 11, 1990}
\include{macros} % used to include figures in this document

\newcommand{\PSbox}[3]{\mbox{\rule{0in}{#3}\special{psfile=#1}\hspace{#2}}}

\begin{document}
\MakeScribeTop

\section{Review of the 3-Way Handshake Protocol}

In lecture 15, we covered the 3-way handshake protocol in detail.
Here we will review the more practical issues involved in implementing
the protocol and making it reasonable to use.

\subsection{Unique ID's} 

In the example given in lecture 15, the IDs used in the protocol were
assumed to be unique.  In implementing IDs we might try to create an
ID space so large that one would never have to be reused -- this is
obviously impractical.  Thus, we must develop a scheme to reuse
message ID's.  One way to make sure all old traces are gone is to set
a maximum message delivery time and, once past that time, purge all
old ID's.  However, several issues must be considered.  First, note
that all old messages must be gone from the system, which includes the
the network and the software at either end.  Second, in a situation
where a crash occurs we must consider where to start the ID's such
that no pre-crash messages resurface.  This becomes especially
important in the event of multiple crashes.  One possible message
scheme would be to use encryption.  By encrypting all new messages
with a new key, the old messages will become unreadable and thus be
effectively removed from the system.

\subsection{Flow Control} 

Another concern in implementing the 3-way handshake protocol is flow
control and congestion. Basically, we want to ensure that no overflow
of messages occurs in the receiver's buffers.  Note that flow control
is not needed from a correctness point of view, however in terms of
performance point it is absolutely essential.

One way to handle congestion is to simply "throw messages on the
floor."  Again, this is acceptable for correctness, however this does
nothing to attempt to manage unruly message flow.  Another scheme is
to use a "sliding window" scheme.  Basically the protocol requires
that the sender only send a credited number of messages to each
reciever.  When a message is sent, the number of credited messages is
reduced by one.  When a message is acknowledged (or "acked"), the
number of credited messages is increased by one.  The sender can only
send a limited number of un-acked messages. This concept is
illustrated in Figure 1.

\begin{figure}[ht]
\label{sliding}
\begin{center}
\PSbox{sw.PS}{4.25in}{2.25in}
\caption{Sliding Window}
\end{center}
\end{figure}

In implementing the sliding window control scheme, note that we have
to keep an index of the last acknowledged message.  In considering
this, we must decide whether to allow messages to be acknowledged
independently or for them to be acknowledged in a predetermined order.
In the previous case, a table would be required to determine whether
the message has been previously acknowledged.  Another possible
ordering protocol could be that once an ack is received for a message,
all previous messages are assumed to be received.  Obviously this
scheme runs into trouble when packets are lost, since subsequent
received messages can not be acked by the receiver.

\section{Networks}

Note that the limiting resource for flow control might not be the
receiver's message buffer, but rather the network itself.  There are
basically two flavors of network, a bus network and a switch network.
Each will be described separately in the sections below.

\subsection{Bus Networks}

The bus network consists of nodes connected off a common back plane.
Some common bus networks included the Small Computer System Interface
(SCSI) and basic internal CPU/memory busses in computers.  The bus
network has two basic properties: 1) only one message can be sent on
the bus at a time and 2) all nodes can see all messages.

The consequences of the properties are: 
\begin{itemize}
\item A built-in broadcast (although reliable broadcast is much harder).
\item A simple addressing scheme where each receiver just needs to
have an unique address -- the entire system need not know every
receiver's address.
\item All nodes are able to see the amount of load on the bus.
\end{itemize}

In implementing bus networks, we must decide how to resolve sending
messages so that collisions do not occur.  One possible solution is
bus arbitration where an arbiter selects one of the requesting nodes
to send its message.  Another solution is to pass requests down nodes
in a daisy chain arrangement.  This scheme is used by the ring
architecture.  The two major types of bus networks and their
particular collision schemes are discussed below.

\begin{figure}[ht]
\label{busnetwork}
\begin{center}
\PSbox{bus.PS}{4.25in}{2.75in}
\caption{Bus network}
\end{center}
\end{figure}

\subsubsection{Ethernet Architecture}

The ethernet [Figure 2] network is basically a number
of nodes connected to a bus.  The ethernet uses the Carrier Sense
Multiple Access with Collision Detection(CSMA/CD) or "Cocktail Party"
protocol to avoid collisions.  The properties of the system are as
follows.

\begin{itemize}
\item If a node is already talking, then another node cannot send a
message.
\item Many nodes are allowed to talk at once and the system can detect
when this occurs.  To correct it CSMA/CD requires that the node back
off for some interval of time and then startup again.
\item The system works on the assumption of a variable wait time interval.
\item The maximum size of the collision window = the diameter of the
network, or else messages would collide if two nodes at either end of
the bus started talking at the same time.
\item The minimum packet size is governed by the maximum transmission
time and the time to detect collisions.
\item When a collision occurs, the sender is unaware of it.
\end{itemize}

Note, this system can go haywire under a number of circumstances.
First, if the variable backup algorithm goes bad.  For example if a
node does not wait but rather immediately retries starvation can
occur.  This is does not occur for a single node failure of this type,
but rather when multiple nodes retry immediately.  This same condition
can occur when two nodes have the exact same backup algorithm.
Another failure can occur when a node does not compete as vigorously
as other nodes, (for example after just processing a message) and thus
loses out.  This can result in the node being aced out every time and
suffering starvation.

\subsubsection{Ring Architecture}

In implementing a bus ring [Figure 3], messages are passed in a
daisy-chain around the ring.  From an electrical engineering stand
point, this system is much easier to implement since messages are only
sent from point A to point B, rather than a number of The properties
of the system are as follows:

\begin{itemize} 
\item Each node receives and resends messages.  
\item The sender of a message removes the message from the ring once it has already cycled.  
\item A token scheme is used so that a node must wait until it gets the token
before sending its messages and then once done sending, passes the
token to the next node in the cycle.  
\end{itemize}

\begin{figure}[ht]
\label{busring}
\begin{center}
\PSbox{bus_ring.PS}{2.83in}{2.75in}
\caption{Bus Ring}
\end{center}
\end{figure}

In using a token, we need to ensure that one and only one token is
produced.  But how are we going to implement this in the case of a
node failure?  One solution is to have a relay in the cycle that
breaks and stops the system when a failure occurs in one of the nodes.
However, can we be assured of stopping the network?  Another solution
is to detect either multiple tokens or no tokens and either stamp out
a token or place one into the system.

One problem with the ring scheme is that if a node crashes the entire
network goes down.  A solution is to implement a relay that is open
when a node is working and closes when the node crashes.  In this way,
the relay simply bypasses the crashed node until it is repaired.  In
order to avoid relays distributed across a wide spatial area, the
relays are often grouped in a central location rather than next to the
nodes.  This leads to the hub architecture in Figure 4.  Relays are
ideal for this type of application since they have been well developed
from telephone systems.

\begin{figure}[ht]
\label{star}
\begin{center}
\PSbox{star.PS}{3.2in}{3.5in}
\caption{Hub Configuration}
\end{center}
\end{figure}

One of the newest advances is a fiber optic token ring scheme (FDDI).
While, the ethernet architecture discussed above has a bandwidth of
10M and the token ring has a bandwidth of 4-16M, the fiber optic token
ring has a bandwidth of 100M.  However, there is no sufficient fiber
optic equivalent to the relay.  Using normal relays suffers from the
limitations of the photon to electron to photon conversion time.
Thus, the FDDI implementors used a dual ring architecture to deal with
node crashes.  If a node fails, the inner ring is used to bypass the
crashed node.  This is illustrated in Figure 5.

\begin{figure}[ht]
\label{dli}
\begin{center}
\PSbox{double_links.PS}{4.25in}{2.72in}
\caption{Double Link Implementation}
\end{center}
\end{figure}

Note that in a bus network, congestion is not a real issue because
only one message can be on the bus at a time and all nodes can see all
messages.  The token ring is inherently fair to all nodes who want to
send, while the ethernet is only probabilistically fair to all nodes.

\subsection{Switch Networks}

The switch network is based on the grand wizard of all networks -- the
telephone company.  It is based on the assumptions that there exist
many point to point connections and that in a large network many
nodes may be failed at the same time.  Note that in switch networks we
no longer get broadcast for free.  This network is illustrated in
Figure 6.

\begin{figure}[hte]
\label{switch}
\begin{center}
\PSbox{switch.PS}{4in}{2in}
\vspace{.1in}
\begin{tabular}{||c|c|c||} \hline
Nodes& Route& Hops\\ \hline
B& 1& 1\\ \hline
C& 2& 1\\ \hline
D& 2& 2\\ \hline
E& 3& 2\\ \hline
F& 3& 1\\ \hline \hline
\end{tabular}
\vspace{.1in}
\caption{Switch Network}
\end{center}
\end{figure}

Message packets originate at end-nodes while the routers receive these
packets and pass them along on an appropriate path to their
destination. Therefore, either the end-nodes or the router must have
some way of finding the receiver in a reasonable amount of time.  One
solution involves keeping a table [Figure 6] of the route and the
number of hops to the final destination.

Each node would create such a table and query each of its neighbors to
find the optimal path.  However, this implementation proves to be
inefficient when a there exists a change in the system since it requires
a rather complicated algorithm to set up and update the table.

Another solution could be to utilize source rooting, in which each
node keeps a table with a route to each of the end nodes.  However,
this is large amount of information for each end-node to keep and
performance suffers when the network changes frequently.

Note that in each of these schemes there is no guarantee that once
packets are sent that they are actually received at their destination.
Many things internal to the network can go wrong.  One solution is for
a node that wants to send some amount of data to ask all other nodes
if it is okay to send.  However this is expensive to set up and
maintain.  Another solution is time division multiplexing as
illustrated in Figure 7.  However, this solution requires the network
to commit to a particular structure.

\begin{figure}[hte]
\label{tdm}
\begin{center}
\PSbox{tdm.PS}{5.5in}{2in}
\caption{Time Division Multiplexing}
\end{center}
\end{figure}


Another implementation involves flow control and back pressure.
However this may require too much time to manage properly.  Still yet,
we could just allow the nodes to throw the packets on the floor.
However, this produces a continual cycle of congestion and does not
manage the congestion in any way.  As a solution, though, we could
increase the delay time before resending a message when no
acknowledgment is received.

\section{Communication and Concurrency}

There seem to be two schools of thought about achieving concurrency
through distributed computers.  The first is that concurrency can be
achieved through message passing.  However, as we have seen it is
expensive to reliably send messages.  Thus, the religion of this
course is that concurrency is implemented through threads, not messages.
This is achieved through remote procedure calls (RPC).

There are three issues in implementing a remote procedure call, each
of which are captured in the code segment below.  They are binding
the call in the proper environment, marshalling and unmarshalling
the arguments and transport.

Binding the call simply means deciding where to send the message with
the remote procedure call.  Basically, we need to know the address of
the machine with local id of the call.  To transform a local procedure
call to a remote procedure call we need to include the local machine
address, the address space and the PC.  Why?  Well, tune in next
week... \\

\vspace{.2in}

\begin{tabular}{lll}
v := a& & \\
m := a.encode& & Marshalling\\
{\it change PC}&  & \\ 
PUT(m)&  & Binding\\
GET(m)&  & Transport\\
v := m.decode&  & Unmarshalling\\
f := v& & \\
{\it run.proc}  & & \\
v := r&  & \\
{\it change PC}  & & \\
x := v&  & \\
\end{tabular}

\vspace{.1in}


\end{document}