% to get hardcopy of this lecture, you need the following files (plus the
% normal latex and tex base directories):
%     lecture.tex:     latex source file
%     macros.tex:      figure macros
%     psfig.tex:       postscript figure (psfig) macro definition
%     pocs-header.sty: Principles of Computer Systems lecture latex header
%
% run the following programs:
% latex lecture
%   -> Creates lecture.dvi, plus latex intermediate files
% latex lecture
%   -> Run a second time to get cross-references right
% dvi2ps lecture |  egrep -v '^showpage$' > lecture.ps
%   -> Merges the figures with the text, can also pipe to lpr
%
\documentstyle[12pt,pocs-header]{article}
\Scribe{Albert Woo}
\Lecturer{Bill Weihl}
\LectureNumber{6}
\LectureDate{September 30, 1991}
\include{macros} % used to include figures in this document
\begin{document}
\MakeScribeTop

\section{Handouts and Announcements}

Lecture 3 notes were distributed at lecture.  Also, it was announced that
there will be NO exams given in this class.

\section{Overview of Performance Analysis and Techniques}

This lecture dealt with performance analysis and optimization techniques
for computer 
systems.  The primary performance metric is, of course, speed.
Making systems run fast is an important theme that will run throughout 
the class.

System performance is a key factor in designing computer systems because
even systems with the correct functionality are useless if performance is
poor.  The main difficulty of making systems run more quickly is that it
usually makes them more complex.
    
While analysis of desired functionality can often be limited to local,
isolated parts of a system, performance analysis cannot.  Contention
of certain shared resources such as the CPU, the memory, and the disk
often force performance anaylsis to be done on the system as a whole.
A change in one module may affect the the performance of another.

How can performance analysis aid a system designer?  Well, the use of 
performance 
analysis techniques allow system designers to:

\begin{itemize}
\item {\bf Compare Design Alternatives} --- By measuring the performance
of different prototype systems, desginers can learn which approaches yield
the most efficient results.  DEC designers used performance analysis 
techniques to determine which algorithms performed best for a
replicated distributed file system.

\item {\bf Size, Dimension, and Configure Systems} --- A system designer
can use performance analysis to decide how many disks are necessary in a 
particular system, how large a memory is needed, how much I/O bandwidth is 
required, etc.

\item {\bf Estimate Capacity Limits} --- For example, how many users can
a system support before it breaks down, what are the saturation levels 
before performance degrades, etc.

\item {\bf Predict Effects of Changes} --- Performance techniques such as 
modeling
and analysis can predict future performance if changes are made to the 
system.

\item {\bf Find Bottlenecks} --- Performance techniques can also give 
indications of what bottlenecks are present in a system.  Usually, poor
performance results are due to a small number of modules.

\end{itemize}

One important lesson to be learned from performance techniques is to
avoid premature optimization.  In other words, don't spend lots of time 
during design optimizing problems that may not exist.  However, it is
still important to anticipate obvious problems during design, such as the
impact a slow disk will have on the performance of a file system.  For an
interesting reference related to this topic please see DEC CRC's paper about 
AutoNet.

Also, it is important to remember that system designs contain many
tradeoffs of parameters, and that these tradeoffs affect performance.
For example, many systems exhibit tradeoffs between space and time.
Unfortunately, it is very difficult to optimize for all factors in a
simple manner.  The key is to optimize only those factors that really
need to be optimized.


\section{Major Performance Analysis Approaches}

These are some of the more prevalent methods of analyzing the performance
of a computer system:

\begin{itemize}

\item {\bf Mathematical Analysis} --- This can include equations, 
stochastic process analysis, or simple back-of-the-envelope calculations.

\item {\bf Simulation} --- A simulation is defined here as a simplified
model of the system, often based on mathematical analysis.  An example of
this would be a computer program that simulates a simple queueing system.

\item {\bf Prototype} --- A prototype is a model of a system that does not
necessarily contain all the modules of the system.  It does, however, contain
the more important ones that affect system performance.  A prototype is
less abstract than a simulation.

\item {\bf Measurement Methods} --- When all else fails, a
system designer can simply conduct measurements of the actual system.
The problem with this approach is that the measurement tools available
often aren't adequate.

\end{itemize}

\section{Performance Analysis Metrics}

A system designer must be able to measure tangible results that provide
vital information about system performance.  These results are
known as ``metrics'' and some of the more useful ones are:\\[1ex]

{\bf Latency} ---  Latency refers to the time it takes for a task to complete,
and is composed of
two major components: 1) The actual raw time it takes for the work to 
get done, and 2) The amount of time the process must wait for shared 
resources to become free.

{\bf Throughput} --- Throughput is defined as how many jobs/unit time a 
module can process
Throughput should not be confused with ``bandwidth'', which is a measure
of the maximum physical output limit of a device.  For example, a disk 
processing random I/O's will exhibit much less throughput than bandwidth.

{\bf Capacity} --- Capacity is a much closer analog to bandwidth.
Capacity reflects the maximum limits of a module.  For instance,
how many bits/sec can be placed on a wire, or how fast a system can be
driven before its performance degrades below some threshold.

Latency, throughput, and capacity are all visible to the user.  However,
these metrics are affected by ``hidden'' metrics, including:

{\bf Utilization} --- Utilization refers to the percentage of time that a 
module was used.  
If the CPU utilization in a system is only 10\%, a predicted throughput 
of 20 jobs/sec might only yield 2 jobs/sec.

{\bf Hit Rates} --- Mostly referring to caches, this metric is the
percentage of time that the desired data can be found in the cache.
On a more general level, this metric can refer to the percentage of
time that a ``fast path'' in a system is taken versus a ``slow path''.

{\bf Queue Lengths} --- The number of sets of input data waiting for a 
particular shared resource.

\section{Major Performance Optimization Techniques}

The following are some of the techniques used to optimize system performance:

\begin{itemize}

\item {\bf Make The Common Case Fast} --- Common cases, or ``fast paths''
(i.e. paths that are taken 90 - 95\%
of the time) should be designed to perform quickly.  The uncommon cases,
or ``slow paths'', can be general, simplistic, and slow, as long as they
handle situations the fast paths can't.  Hopefully, these slow paths won't
be taken very often.

\item {\bf Caching} --- A special instance of making the common case fast.
Caching takes advantage of natural spatial and temporal locality tendencies
of memory accesses to allow very quick referencing of data for the common
case (i.e. cache ``hit'').

\item {\bf Concurrency} --- Performing tasks in parallel is an obvious 
time-saver, although it is often difficult to implement.  The benefits are
substantial if the application is appropriate.

\item {\bf Burstiness} --- If a particular application exhibits ``bursty''
behavior (i.e. periods of high activity followed by long periods of 
low activity), its input data can be buffered, and then later performed in the
background during a period of low activity.  

\item {\bf Device Characteristics} --- Make a device do what it does best,
or what it can do with the best performance.  For example, a disk can 
do sequential I/O's quickly, and random I/O's slowly, so try to make it 
do more sequential accesses than random ones.

\end{itemize} 

%It is important to remember to apply these techniques only after you 
%have measured and analyzed the performance of the system thoroughly.  
%Don't try to guess beforehand!

\section{Measurement Methods}

As mentioned before, methods and tools for measuring system performance 
are not always very useful.  The following are two methods for measuring
which modules in a system are dominating temporally.


%Latency is obviously a very important metric.  Since speed is the major
%parameter of performance, methods of measuring latency of modules and thus 
%finding bottlenecks are crucial.  The following are some methods for 
%measuring in relative terms which modules in a system are taking the most time.

\begin{itemize}

\item {\bf Timing Code Blocks} --- This technique involves writing some
test code loops that call certain procedures multiple times.  The
overall time per invocation is calculated by dividing elapsed time by
the number of invocations.  The problem with this technique is that it
might not take into account memory system effects, primarily caches
effects.

\item {\bf Using Profiling Tools} --- Profiling tools allow a designer to find
out what percentage of time is spent in a certain procedure or module.
{\bf GPROF} is a profiling tool that works by sampling; it periodically
interrupts the program and records the value of the program counter.  The
problem with GPROF is that it may identify the wrong source of the
problem; although the disks may be consuming the most time in a
system, perhaps they are being activated too often.  Another profiling
tool is {\bf PIXIE}, which works by counting the cycles spent in each
procedure.  PIXIE counts cycles more accurately than GPROF, but it
does not take cache misses into account.

\end{itemize}

%In general, measuring computer systems is difficult.  Systems should be
%designed so that they can be measured easily.  Unfortunately this is a
%rather rare occurance.

When making performance measurements, system designers should also remember
to perform {\bf regression testing}.  When changes are made to a system to
improve performance in one area, another area's performance may suffer.
A designer should run many different types of tests on a new design to 
check that performance has not declined, or ``regressed'', in other areas.

\section{Analytical Techniques for Performance Analysis}

An analytical or mathematical model of a system is often desired to
provide a solid theoretical analogue of what is happening in a system.
For example, {\bf stochastic analysis} is often used to analyze
queueing network models.  Although quite prevalent, stochastic
analysis has some drawbacks.  For example, the underlying stochastic
theory must be WELL understood before it can be applied.  Many systems
are just too complicated to be analyzed in this way.  Also, many of
the equations used to analyze systems in this way are based on
assumptions that may or may not be true.

{\bf Operational Analysis} of queueing networks, on the other hand, is
not based upon schotastic processes, and thus displays several
advantages.  First, Operational Analysis is somewhat simpler to use
and understand than stochastic processes.  Parameters can be
measured more simply and directly.  The assumptions one must make are
simpler, and are more likely to be true.

A good reference for Operational Analysis comes from ``Operational
Analyses of Queueing Models'' by Denning and Buzen, 
in {\em Computing Surveys}, Semptember 1978.

\section{Operational Analysis Example}

At lecture, an example of Operational Analysis was introduced.  A
computer system can be abstracted as a ``server'', composed of
``subservers'', modules that have queues and perform functions on
data.  Server (i.e. system) input flows into one or more subserver
queues and is processed when at the head of the queue.  Outputs from
subservers can be either fed into other subserver queues or can leave
the server as its output.  Please see Figure 1 for a diagram of such a
server.


\begin{figure}
\centerline{\psfig{width=4in,figure=server.idraw}}
\caption{``Server'' representation of a queueing network model}
%\label{netfig}
\end{figure}


\subsection{Operational Analysis Example Parameters}

A designer using the server model would probably like to measure certain 
performance parameters of the server or a subserver during a given 
time {\it T} in which the system runs.  Some of these parmeters include:

{\it A} --- the number of arrivals the server accepted

{\it B} --- the total time that the server was busy

{\it C} --- the number of jobs that the server managed to complete\\[1ex]


\noindent In addition, some derived results can be calculated:

$\lambda$ --- the arrival rate $(\lambda = A/T)$

{\it X} --- the completion rate $(X = C/T)$

{\it U} --- the percentage utilization $(U = B/T)$

{\it S} --- the mean service time per completed job $(S = B/C)$

Two other metrics were mentioned: {\it R}, which is the average response time
from entry into the system until exit, and {\it N}, which is the average
number of jobs in the system.

\end{document}