\documentstyle[12pt,postscript,/nfs/thor/thor/6826/92/macros/times]{article} %\documentstyle[12pt]{article} \input{/nfs/thor/thor/6826/92/macros/lecture} %\begin{figure} %\PostscriptPicture{/nfs/thor/thor/6826/92/lectures/} %\caption{} %\end{figure} \Scribe{Mitchell Charity} \Lecturer{Bill Weihl} \LectureNumber{24} \LectureDate{December 9, 1992} \setlength{\parskip}{2mm} \setlength{\parindent}{0mm} \newcommand{\specword}[1]{{\tt #1}\ } \newcommand{\module}{\specword{VAR}} \newcommand{\var}{\specword{VAR}} \begin{document} \MakeScribeTop \begin{center} {\Large\bf Caching (cont)} \end{center} This lecture continues our discussion of concurrent caching (Handout 50).\\ Lecture topics: \begin{itemize} \item Cache coherence, using Update, Mutex, and competitive strategies \item Competitive algorithms \item Caching in distributed systems \item Limits of caching \item The coherence abstraction \end{itemize} \section{Handouts} Lecture Notes \#19 and \#20. \section{Cache coherence - using Update} To implement coherent memory out of multiple caches, one must maintain cache coherence. Last time, we discussed using invalidation to maintain cache coherence. To recap: one writes to main memory after having invalidated all other caches, and reads are then done from main memory. This can be expensive when the processor is close to the cache, but distant from the main memory. Update is another way to maintain cache coherence. Rather than invalidating other cache copies on a write, they are updated directly to the new value. Further, the update spec given in Handout 50 differs from invalidation spec in allowing data to be loaded from another cache besides the main memory. Note that the important invariant: ``Data in cache is current'' is still maintained. The key issues in deciding the performance tradeoff between invalidation and update are the application's pattern of usage, and the structure of the communication layer. Invalidation is suited for successive local writes because the first write invalidates other cache copies so that consequent writes are purely local operations. On the other hand, updating favors reads by increasing the hit ratio, because cached copies are not invalidated due to remote writes. Thus, if an application thread updates a variable many times between reads by other threads, invalidation is appropriate. But if other threads frequently use the updated variable, the cost of update broadcast can be less than the cost of cache miss messages. We now evaluate the cost of updating and invalidation for a bus-based network and a point-to-point network separately. The cost is measured in terms of number of messages exchanged. Assume that a processor does $k$ successive writes on a variable $x$, which is followed by the other $n$ processors reading $x$. We consider bus-based networks first. If invalidation is used, the first write broadcasts an invalidate message. Consequent writes do not need any bus transaction. However, later, each of the other $n$ processors miss on trying to read $x$, resulting in further $n$ transactions. In the update strategy, the $k$ updates each require $k$ update broadcasts. Thereafter, each processor can read its local value of $x$. Now, we consider point-to-point networks. With invalidation, the first write results in $n$ invalidation messages to other processors. Remaining writes do not require any messages. Later, each of the $n$ processors needs a net transaction to fetch $x$. With update, each of the $k$ update broadcasts comprises $n$ messages. No further messages are required for reading $x$. \vspace{1ex} \centerline{\begin{tabular}{|l||c|c|} \hline &Bus&Net\\ \hline\hline Update&k&kn\\ \hline Invalidate&n+1&2n\\ \hline \end{tabular}} \vspace{1ex} Thus, a bus-based network often favors update, while a point-to-point network might favor invalidation. \section{Cache coherence - using Mutual Exclusion} The final approach we consider maintains cache consistency using mutual exclusion rather than invalidation or update. Previously, we used R/W locking, permitting one writer or multiple readers. Mutual exclusion allows only one usable cached copy, thus permitting exactly one writer or reader. The advantage of mutual exclusion is simplicity, of both abstraction and implementation. Internal actions can move data between memory and cache fairly freely. The invariant about data in a cache is different. Cached data is only current if it is locked. \begin{verbatim} c(p)!a /\ locked(a) => Current(p,a) rather than just c(p)!a => Current(p,a) \end{verbatim} Locking is used explicitly, rather than implicitly by having only a single cached copy. The key invariant is that dirty data is locked. The implementation of coherent memory using mutual exclusion is close to that of incoherent memory. Only difference is that {\tt Read} and {\tt Write} now have a lock guard, and {\tt Acquire} and {\tt Release} routines are available. Indeed, this scheme is often used when the underlying hardware provides incoherent memory; the software uses mutexes or critical regions to make it coherent. {\tt Acquire} does a {\tt Barrier} to guarantee that the value will be fetched afresh, so that it reflects earlier writes. {\tt Release} does a flush to guarantee that the value fetched by the next {\tt Acquire} is up-to-date. The spec shows data being transferred between caches via main memory. This is just for simplicity. Cache to cache transfers are perfectly reasonable. \section{Competitive algorithms} Another way to maintain cache consistency is to use a combination of invalidation and updates. The choice, of which approach to use in a given circumstance, is made based on the expected application behavior. This is called a ``competitive strategy''. We saw an example earlier in the term: with mutexes. A thread wanting to gain a mutex needs to wait till the mutex is freed. One way of accomplishing this is to spin, repeatedly testing the condition. Another is to block, swapping out until the condition is met. Spinning has a small, but ongoing, cost. Blocking has a fixed large cost of thread swapping. The ideal combination would be to spin for short waits, and block for long ones. A practical combination is to spin for a period equal to that it takes for blocking, and then block. \begin{figure} \PostscriptPicture{/nfs/thor/thor/6826/92/lectures/24/mutex.ps} \caption{A Competitive Strategy to Acquire a Mutex} \end{figure} There has been a lot of research over the last 4-5 years on the theory of competitive algorithms. An algorithm's performance is compared with the best that might be done with complete knowledge of the future. For example, the strategy described above for waiting for a mutex costs no more than twice of an ideal scheme. A practical randomization optimization was recently published which brings this factor down to $1.58$. Cache consistency may increasingly be implemented in software, together with some hardware support. This permits tailoring the competitive algorithm to individual applications, bringing down the factor of 2. \section{Distributed systems and caching} Everything which has been said about cached systems also applies to distributed systems. So how do distributed systems differ from coherent ones? In having an issue of {\bf fault tolerance}. One must address the impact of faults. Define their semantics. Develop implementation techniques. Distributed systems can be said to have two flavors of failure: machine and network. In considering a distributed system whose machines have caches, one obvious concern is persistence of writes. One might write ``foo'' to cache, and then crash, losing ``foo''. But this behavior is already accepted in non-distributed systems to allow write-behind. In both cases, one accepts an incoherent memory spec, rather than requiring a stronger coherent one which ignores failure. A more serious complication of machine failure is in the availability of the locks. What happens if a machine acquires a lock located on a server, and then crashes? Is the locked resource inaccessible until the machine recovers? What if that never happens? This availability problem is addressed by replacing ``locks'' with ``leases''. A lease is just a lock that is valid for a bounded time interval. The client may periodically ask the server to extend the lease. If the client crashes, the server waits till the lease has expired, whereupon it is free to grant the lease to some other client. The decay time of a lease is a tradeoff between the communication cost of renewal requests, and non-availability after a crash. Note that leases require the client and server clock rates be similar. If not, the client may think that it holds the lease while the server thinks the lease has expired and grants it to some other client, resulting in two clients holding the lease simultaneously. Suppose there is a network partition because of which a client is unable to renew the lease on a cached item. If it has already made changes to its local copy, it must either abort those changes, or proceed under the awareness that some other client may be changing the item simultaneously. Thus, network partition can lead to divergent copies of data. How this is handled depends on whether machines are expected to ever operate disconnected from the net. If machines never disconnect, then one can simply add network redundancy to avoid partition. But if machines can operate disconnected, then either write permission should be limited to one partition, or one must reconcile the divergences. Currently the most common approach to reconciliation is manual intervention. New systems, such as Coda, provide some automated support for merging. The minimum support needed is simply divergence detection. Once detected, the most that can generally be done is to notify the user. However, when the divergent object has sufficiently constrained semantics (filesystem directories for example), some automatic update can be done. Note that object immutability doesn't resolve these problems, but just moves them to some other abstraction layer, such as a namespace. Different caching strategies have different performance implications. For instance, the client might maintain a cache of information that is available at both the server and in its local disk. On a cache miss, the client can go over the net to the server, or to its local disk. While disk latency might be larger than a network round trip time to the server, going to the server consumes network bandwidth and processing time at the server. If there are a large number of clients then it might be better for each client to offload work from the server by accessing its own disk. While most memory systems are coherent, most distributed filesystems do not provide a coherent abstraction. They permit race conditions, and all the other problems of incoherent memory. Some prototypes are moving towards an atomic semantics (coherent though non-persistent across crashes). \section{Limits of caching} The possible effectiveness of caching is limited in the presence of sharing, both practically and fundamentally. On a uniprocessor, one might increase cache size enough to basically eliminate cache misses. A large, fully associative cache could have the working sets of all processes. The hit rate asymptotically approaches 1 with increasing cache size. Multiprocessor distribution inherently bounds cache efficiency. Because of contention, shared data can never be perfectly cached (Figure~2). \begin{figure} \PostscriptPicture{/nfs/thor/thor/6826/92/lectures/24/cache.ps} \caption{Contention for Shared Data Disallows Miss-less Caching.} \end{figure} In multi-threaded systems with very fast context switches, one can minimize the cost of a miss by swapping to a different thread. An invalidation cache consistency approach can eliminate read misses, but only by adding write misses. \section{False Sharing} There are practical limits on cache efficiency as well. If a cache is not fully associative, then there may be mapped interference. This is an example of {\bf false sharing}, an extreme version of a phenomenon which occurs whenever blocking is used. Cache lines (a.k.a. blocks) are one thing which cause false sharing. Consider two data items adjacent in memory. If they both end up on the same cache line, then the cache line introduces sharing. Two processors using the two different items may thrash the cache line back and forth between themselves. This happens regardless of cache consistency mechanism (update or invalidation). This thrashing can have a big impact on performance, especially with fine grain concurrency. To avoid this problem, one tries to put different processor working sets on different lines. In this case, one could move the locks apart so that they land on different lines. An example of the more general blocking problem occurs with cache-naive compilers. The instruction cache often gets loaded with about twice the number of instructions actually used, because of needless loading of infrequently used paths. This is addressed by attempting to move infrequently used instructions to a different block. \section{The coherence abstraction} Generally, one does not implement coherence totally in hardware. One adopts software conventions (perhaps by using a smart compiler). These bridge the gap between incoherent hardware, and a coherent memory abstraction for applications. One convention is to use critical sections and mutexes. One acquires a mutex, creating a ``barrier'', performs an action, and releases it, forming a ``flash'' or ``fence''. The strategy of critical sections is to prevent more than one procedure from using a section simultaneously. Since generally only one or a few threads are sharing some data, the cost of this exclusion can be kept small. If one has an abstract machine with coherent memory, does one really need to worry about locality? Unfortunately, real machine hardware is incoherent, so one does. An alternative is to build a high power network, and processors with low cost context switching. The tradeoff is the power which can be obtained for a given cost of silicon. \end{document}