From: hossein.ahmadi@gmail.com on behalf of Hossein Ahmadi [hahmadi2@uiuc.edu] Sent: Tuesday, February 19, 2008 9:39 AM To: indy@cs.uiuc.edu Subject: 525 review 02/19 \section{Colyseus: A Distributed Architecture for Online Multiplayer Games} Current multi-player game applications use client server approach which has serious scalability issues. The main reason is that a p2p scheme can not maintain strong consistency efficiently. This paper proposes a distributed architecture called Colyseus which aims to address more relaxed consistency constraints where authors claim that a weak consistency is sufficient for real games. Colyseus use two key ideas to make a feasible distributed architecture: i) weak consistency, ii) limited interactions of each peer with the state of the game. Based on these assumptions, required objects at each peer are replicated from the primary copy; then, a replica manager handles synchronizing replicas with the primary copies. The state of the game is assumed to be stored in objects and each object interacts with only a few other objects (e.g. objects in range of sight). Colyseus uses a DHT based protocol for each primary object to subscribe for the other objects it is interested in. Using range-search DHT, paper explains how efficient each peer can obtain local replicas of the object that it may be interested in future. Since Colyseus is pre-fetching objects it would not have suffer much from network delays. On the other hand, replica manager propagates updates occurred to the primary copy to other replicas. Using the two key ideas mentioned above are the most important things making this paper interesting. The paper also considers variety of design approaches to the problem like different discovery schemes. The part on the locating distributed objects in this context is novel and interesting. Moreover, authors used real game data to evaluate their work. The main drawback of this paper is that it does not focus on replica management enough. One critical issue in making replicas consistent is the global serializability. In other words, in order to have the same view of the game at least in the long term, the series of events should happen with the same order at all peers. This issue have not been addressed in the paper. \section{OverCite: A Distributed, Cooperative CiteSeer} This paper presents a distributed architecture for well known CiteSeer application. Gathering and storing such large amount of documents for a single server or a cluster of servers can be very inefficient in terms of both storage and bandwidth usage. Using a p2p approach to have servers distributed all over the Internet can result in a more balanced resource usage. Authors propose OverCite in three tiers, one to address the high end user interface, second to perform indexing and searching and a low level DHT based service to store documents in participating nodes. Specifically, each node crawl the Internet for research papers and store them in the overlay network using one of current DHT approaches. The index file is partitioned among $k$ nodes such that there are $n/k$ copies of each partition in OverCite. The user interface tier is distributed using DNS techniques available for current server clusters to redirect web page requests to different nodes in the overlay. Each node receiving a query from the user sends keyword to $k-1$ other nodes with different index partitions. The node, then merges the replies to get the query result. The application which have been chosen for this paper can actually be extended to a great category of web application which requires high amount of both bandwidth and storage. Therefore, the overlay network solution can be very useful. The paper uses current DHT and partitioning techniques that have been developed for several years which make it more feasible to implement. However, the most critical bottleneck to the performance of OverCite is index partitions and delay to acquire queries back from different partitions. One of the issues in the paper is that it should propose a method to acquire better locality in order to have very low latencies in query processing. Also, one can question the tradeoffs of the partitioning itself. Having less partitions results in higher update propagation cost while more partitions makes the searching work slower. From: fariba.mahboobe.khan@gmail.com on behalf of Fariba Khan [fkhan2@uiuc.edu] Sent: Tuesday, February 19, 2008 9:23 AM To: Gupta, Indranil Subject: 525 review 02/19 Storage management and caching in PAST, a large-scale, persistent peer-to-peer storage utility , A. Rowstron et al, SOSP 2001 PAST is a large scale p2p storage system built on top of PASTRY. A client can insert k replicas of a file with certain fileID. Files can be looked up by fileID and also a file can be reclaimed back. PAST kind of has three properties that it provides: security, storage management, cache management. PAST has a security mechanism to identify nodes and pass around certificates like file certificate, store receipt and reclaim receipt. Storage management mechanism tries to maintain a balance of loads between nodes as system-wide limits are approaching and make sure that k copies are maintained. When all the replicas cannot save a file locally it uses file diversion where the salt to generate fileID is changed. In replica diversion when a node cannot store a file it diverts it to another node in its leaf set. Cache management uses unused space in nodes to cache popular files. If the node needs the space to store a file the cached file can be immediately discarded. Discussion: 1. It is too many options for me. I get lost into if its main purpose is storage or cache, reliability or availability. 2. No search option on filename 3. The security is too vague for me. I am not sure how signing receipts ensures a malicious node is actually saving a copy? And the smartcard for each node is so impractical. 4. Is there a system that uses PAST or some modification? I think I heard it somewhere, but I cannot remember. -- Fariba From: Riccardo Crepaldi [crepric@gmail.com] Sent: Tuesday, February 19, 2008 3:35 AM To: Gupta, Indranil Subject: 525 review 02/19 Colyseus: A distributed Architecture for Online Multiplayer Games This research, from CMU, aims at implement efficient distributed server architecture for increasing the scalability of online games. The paper focuses mainly on First Person Shooter games (FPS), but the authors claims, and that is reasonable, that Role Player Games, that are much less greedy in term of latency and performance, will get the same benefits than FPS from Colyseus. The standard client-server architecture is shown to be not efficient for a large number of users, as the new generation of games (and gamers) seems to require. The main bottlenecks are due to the number of objects that have to be tracked (computational power required) and the communications to the clients (bandwidth required). The strategies that Colyseus uses are parallelized computation among multiple servers, and replication to reduce the latency for fetches. Each object is stored in multiple replicas, one of which is the authoritative one, over which the computation is done. The other replicas, if any, are periodically refreshed, allowing a quick local access to reduce latency. The consistency guarantee that Colyseus provides is simply a weak consistency. That means that an optimistic write procedure is used and in some case the replicas can be inconsistent. Finally the use of a prediction system to pre-fetch the objects that are entering the view of interest of each user is used to improve the performance. This helps to solve the problem of the delay introduced by the query procedure on the DHTs used to store the objects. A large number of results is provided using Quake II and Quake III, assuming that these can be considered a representative sample of all online games. The paper addresses a very interesting problem and provides a good solution and set of results. Modifying a well-known game is a good idea because readers are more likely to be familiar with the problems and the quality of service required for a good game. The results set is well presented but there is some point that is not clear or well justified. The authors assume for example that the achieved level of inconsistency is not affecting the perceived quality of service, but computer players are very well known to be greedy about game performance. Additionally the problem of server failure is not addressed, while in distributed system connected through the Internet these kind of failures can happen, due to routes failures, or simply hardware-related issues. In addiction the UDP protocol is assumed for communications among peers, but there is no information on whether the experiments did consider a Packet loss value. The simulations are then ran at Emulab, while maybe Planetlab would have been probably a better testing platform because of the real internet connection among nodes that are so far from each other. There are some other consideration about the paper: first of all the authors are not clear in the definition of rect and square areas, and it should be a very important point, since the choice about these two definitions results in a big difference in the performance of the game. Additionally, it is never specified in the paper that objects like missiles, projectiles, and even people in the game sometimes are usually moving following strict rules that can be predicted on the client side. For example a missile is moving on a given trajectory and velocity and its new position can be computed based on the last one. Colyseus does not provide any mechanism to take advantage of that, but they surely would help and improve the performance, depending on the game. ---------------------------------------------------------------------------------------- OverCite: a Distributed, Cooperative CiteSeer CiteSeer is a very well known web service that is used from many users all over the world. A service of this kind requires a lot of storage capacity and bandwidth, and also a big computational power in the query process. This paper presents OverCite, a distributed version of CiteSeer that provides the same user inerface. The system is designed using a three-tier approach, where the first level, the highest one, provides the desired transparency to the user, that can benefit of the standard, well known interface of CiteSeer. The second and the third level are instead aware of the distributed nature of system and they share a lot of information to keep the system up to date. There is a noticeable increase in the storage dimension, even more important if we consider that OverCite saves only the original file, while CiteSeer saves a copy of each of a PDF, PS, Text of each article. The performance improving in terms of number of queries per second serves is high, but it is not clear if having mirrored services with the aim of split the load among multiple servers would achieve comparable results without the need for a complex, distributed system, since the storage capacity is still increased. There is probably a tradeoff in the number of servers that the system is made of, that can be considered to choose between a distributed or a mirrored solution. I find interesting the choice of applying a general concept in CS to a real and very famous Internet service, because it gives a case study familiar to the user and allows being sensitive to all the networking problems that can happen. However this is a very particular case study and it might not fit other Internet services with different requirements and storage dimensions. From: Alejandro Gutierrez [agutie01@gmail.com] Sent: Tuesday, February 19, 2008 2:15 AM To: Gupta, Indranil Subject: 525 review 02/19 =============================================================== “Storage management and caching in PAST, a large-scale, persistent peer-to-peer storage utility” by: Anthony Rowston and Peter Druschel Reviewed by: ALEJANDRO GUTIERREZ =============================================================== This paper presents a project from the Microsoft Research Labs and Rice University called PAST, a large-scale peer-to-peer persistent storage utility layered on top of Pastry ( Remember Pastry is an overlay and routing network for the implementation of a Distributed Hash Table). The way PAST stores files is by computing the hash of the filename, and then the content is routed to the node in the circular key-space closest to the hash. This node is in charge of sending copies of that file to the k nodes nearest to the actual key; most probably these hosts will be leaf nodes of this node and therefore directly reachable. PAST ensures both data redundancy and load distribution. As neighbor nodes the key-space is usually in different geographical regions the probability that all k nodes will be offline at the same time is very small. This is where Pastry comes into play because it tries to minimize the distance to reach the data. This is why the nearest node is likely to be the one that will reply to the query. Contributions: * Storage management: PAST wants the aggregate size of stored files to be close to the aggregate capacity in a PAST network, before insert requests are rejected. * Caching: cache management tries to minimize the access latency (routing distances), to maximize the throughput and to balance the query load in the system. * The authors provide experimental results of PAST using a prototype implemented in Java. This provides an indication of the efficiency of PAST using different parameters. Wide-scale replication has the potential to increase availability and durability; it also introduces 2 important challenges: * One must increase the number of replicas to achieve high durability for large systems. * The increase in the number of replicas increases the bandwidth and storage requirements of the system. Future Research: * Erasure-resilient codes use an order of magnitude less bandwidth and storage than replication for systems with similar mean time to failure. * Implications for distributed systems P2P storage is today a hot topic in computer science research. * Amazon.com, and other world-famous companies, are using today distributed hash tables to manage their data reliably at a massive scale. =============================================================== “Colyseus: A Distributed Architecture for Online Multiplayer Games” by: Ashwin Bharambe, Jeffrey Pang, Srinivasan Seshan Reviewed by: ALEJANDRO GUTIERREZ =============================================================== This paper presents a project from the Distributed systems, Network protocols & Applications DNA group at Carnegie Mellon University called Colyseus, a distributed system for increasing the scalability of first-person-shooter (FPS) online computer games, using master/slave data replication to spread the load of state update distribution among several game server nodes. The paper describes the basic design of FPS games, with a collection of whose behavior is evaluated and executed in discrete steps, typically at around 10-20 steps (frames) per second. I like the fact it provides empirical data showing how the execution of game logic and the use of bandwidth for sending state updates to players are the main server-side bottlenecks. Colyseus provides weak consistency guarantees only. The logic on one node will generally observe and act on a slightly outdated state with respect to other nodes, but the premise here is that FPS games tolerate weak synchrony, up to the point where human players begin to notice it. The solution they propose is for each node to only maintain at each time replicas of objects that the node’s primary objects are likely to read or write in the near future. An object specifies its area of interest as a range query (a conjunctive query over ranges of object attribute values such as the 3d coordinates of a particular object). A node tries to maintain replicas for all objects that currently fall in the disjunction of the range queries (also known as subscriptions) of its primary objects. Colyseus uses a Distributed Hash Table (DHT) technique to enable a node to discover the objects currently satisfying its subscriptions. Once an object is discovered, though, a replica is established for it and the node receives its updates directly from the object’s primary, not through the DHT, for lower latency. The authors evaluate performance along three metrics: bandwidth use, latency and inconsistencies due to missed replicas and publications. The main results, though, appear in figure 12, for games with computer-controlled players (bots). These indicate that a Colyseus-based cluster of servers can maintain bandwidth-per-node levels at a level an order of magnitude smaller than a centralized server using broadcasts for updates, or up to two orders of magnitudes smaller than a centralized server without the benefit of network broadcasting Colyseus takes no advantage of the structure and topology of the world; the volumes of interest essentially exist and move around in “free space”. Yet FPS maps are generally highly structured, and have graph-like connectivity information that should be readily exploitable. From: dkassa2@uiuc.edu Sent: Tuesday, February 19, 2008 2:04 AM To: indy@cs.uiuc.edu Subject: 525 review 02/19 Review 5: Paper Title: Colyseus: A Distributed Architecture for Online Multiplayer Games In this paper a design, implementation and evaluation of a distributed arcitecture for multiplayer games called Colyseus is presented. Colyseus is experimentally shown to effectively distribute game traffic across the participating nodes. This allows it to support low latency game-play for an order of magnitude more players than existing single server designs with similar per-node bandwidth costs. To address some difficult challenges of networked games Colyseus takes advantage of the fact that games tolerate weak consistency in the application state. Besides it exploits the other fact that game-play is usually governed by a strict set of rules that make the reads and writes to the shared state highly predictable. Colyseus provides a rich query interface over the system-wide collection of objects to identify and fetch required objects. Colyseus acts as a game object manager. Colyseus enables low-latency game-play by decoupling object discovery and replica synchronization, by using proactive replication for short-lived objects, and by pre-fetching of relevant objects using interest prediction. The paper is explained with sufficient details. However I cannot understand a specific creative contribution it makes. I feel like the paper gives a game specific approach which may be difficult to generalize to other games. I don't also know how the paper can address the game cheating issues it raises in the end. ==================================== Review 6: Paper Title: OverCite: A Distributed, Cooperative CiteSeer The paper presents a new digital research library system called OverCite which aggregates donated resources at multiple sites to provide CiteSeer-like document retrieval. In OverCite, presentation servers provide an identical user interface to CiteSeer's, application servers partition and replicate a search index to spread the work of answering each query among several nodes and a distributed hash table stores documents and meta-data and coordinates the activities of the servers. Based on experimental results OverCite increases query throughput considerably though it requires more total storage and network bandwidth than centralized CiteSeer. OverCite can use the resources of the sites to support document alerts and other new features. OverCite can scale its capacity to meet many new demands as its architecture allows it to include new resources as they become available. The paper explains the proposed system in sufficient details. I have no negative comments against it. From: ysarwar@gmail.com on behalf of Yusuf Sarwar [mduddin2@uiuc.edu] Sent: Tuesday, February 19, 2008 1:25 AM To: Gupta, Indranil Subject: 525 review 02/19 Review: OverCite: A distributed, Cooperative CiteSeer CiteSeer is a non-commercial popular website used as a respository of computer science research papers. The authors propose OverCite, a multi-site version of citeseer by acquiring end users support for disk space and network traffic. The solution was motivated by an observation that when a non-commercial site becomes popular, it may suffer from availability due to the very lack of resources required to run the service. But the involvement of the end users as participants of system with sharing some of their computing resources can offer more longevity and availability to the entire system. OverCite envisions citeseer be no more remaining as a 'private' service maintained by some dedicated party (here PSU), rather it will be expanded onto its users becoming a 'public' entity in terms of both the service itself and resources it needs. Procs: - A multi-site deployment of citeseer, in a form of a pure distributed content provider network based on a three-tier DHT-back design... - Three tier design... front end web servers, mid search engines, crawlers and lastly data storage, index server and DHT... at the back - Preserves the similar notion/definitions of meta-data, namely document ID (DID), citation ID (CID), groups and so on... that helped to make an extension to multi-site deployment... - Parallel index servers offer low query processing time and faster generation of search results. - Easy expansion with joining of new nodes..... - DHT provides quick lookup of specific files and file replication among several servers offer more availability. - Live implementation with 27 nodes... Cons: - How is the nodes responsibility defined? Which nodes are assigned which role, index server, crawler, DHT nodes, storage...? - Majority of nodes are at MIT and a user makes 128 queries also from MIT.. Does it anyway bias the query latency results? - Crawling hasn't been tested on real testbed for OverSite... looking for files in web pages and dragging them back to the server are really big tasks for citeseer... - Duplication detection on some centralized system is somewhat manageable, but in pure distributed operation it would be severely daunting task...., how do authors pose that problem? Comments: The work deserves additional praise. Building such a distributed content providing services for computer science publications could be of great use for CS researchers/students/practitioners across the globe. ===================================== Review: Storage management and caching in PAST, a large scale persistent peer-to-peer storage utility The paper proposes PAST, a storage management and caching on top of PASTRY, a peer-to-peer routing substrate designed the same authors. PAST distribute files across a large number of peer nodes with adequate replication and efficient lookup. Standard directory services like insertion, lookup, and deletion (reclaim) of files are provided by PAST along with necessary load balancing and caching. Pros: - Can support a gigantic file system with millions of files, tons of data volume,.... scalability is the salient feature of the system... - Highly fault tolerant, replicated storage can handle several failures, still the file can be retrieved. - Load balancing for uniform utilization among the nodes, - A perfect piece of work, considers almost everything required for persistent storage... Cons: - Allows only read only access to files... files might not be editable... once inserted and replicated they might not be changed.... - The authors discuss about file encoding, usage of some coding (may be fountain coding) on data blocks rather than just replication. It reduces the total storage requirements for replication. It will be nice to see this in PAST. - Just a confusion, does the entire file content traverses the hops when the file is routed along the PASTRY substrate...the cache insertion policy indicates this though. But can it be more efficient if the file is transfered only after the destination is reached along via PASTRY hops? =================================== From: Justin King [kingjkk@gmail.com] on behalf of Justin King [king1@uiuc.edu] Sent: Tuesday, February 19, 2008 12:31 AM To: Gupta, Indranil Subject: 525 review 02/19 ---------- Justin King, king1@uiuc.edu Reviews 2/19/08 ----- OverCite: A Distributed, Cooperative CiteSeer Stribling et al OverCite provides a distributed replacement for the CiteSeer academic paper storage and retrieval system. OverCite uses technologies we have discussed in class, including distributed hash tables and RON, in order to distribute and communicate between widely distributed nodes. OverCite is able to successfully increase the throughput of CiteSeer by a significant amount, and performance will scale upwards as more machines are added to the overlay. While the total storage and bandwidth required increases, the amount of storage and bandwidth required of each node also scales (downwards as the overlay grows). Pros - This paper presented a good and completely legal usage of distributed hash tables and p2p technologies. This is in contrast to the majority of uses (or at least, the majority of uses in the public’s perception), which illicitly share mpegs. I was thoroughly convinced that not only did the technology work, but that this presents a true legal usage of p2p technologies. - Significant thought seems to have been given to the design of the system. The three-tiered design (rather than a hacked-together monolithic system that I sometimes see in “real implementation” papers) should help with future versions of this project should the authors wish to continue its development - If I were a major technology company or research university, I’d be convinced to loan a machine or two to the overlay in exchange for improved throughput for my company/university’s members (which, I presume, being a member of the RON would provide). Cons - Does rely on a RON, and thus doesn’t play nicely with TCP. However, I doubt that for this single application, the amortized penalty on any large network is significant (likely, negligible, as few papers are being downloaded at a time, as it takes much longer for a human to process a research paper than an average web page). ----- Colyseus: A Distributed Architecture for Online Multiplayer Games Bharambe et al This paper presents Colyseus, which provides an engine upon which first person shooter (FPS) games can be built, using p2p principles such as distributed hash tables. Each node in the “p2p” network is responsible for a set of objects in an ongoing game. Other nodes interested in a given node’s objects can fetch (and usually, due to the real-time deadlines of FPS games, pre-fetch) local copies of the objects they are interested in, based on the location of a given node’s users in a game map. Rather than a standard DHT, where objects are distributed uniformly across the overlay, Colyseus uses a sequential proximity system, where objects are numbered, and closely numbered objects are hosted on the same, or nearby, nodes. A “weak” consistency is provided across the overlay, and consistency conflicts can be resolved “slowly”, as the game cannot proceed faster than humans can react to it (thus making stronger consistency unnecessary and a waste of processing resources and bandwidth). Pros - Real implementation, with real user traces used to do many of the studies. While they resorted to bots or AI systems playing in some instances, the use of real traces is more convincing to me. - Discussion of applying lessons learned from Colyseus to other types of games (MMORPGs and RTS games, especially) was good, though implementation would have been even better. Perhaps there has been some of this work done already. Cons - I’m not convinced that having a large majority of users clustered in an area of the map is properly dealt with, especially in heavy firefights (which is when - The sample machine used (and I find no specs for other machines in the paper) is quite old for a 2006 paper, but Quake II is even older. Would a more recent FPS reveal more problems with their strategy? - Their goal seemed to be 10 frames per second, which most gamers I know consider unacceptable. ---------- From: Justin Tulloss [jmtulloss@gmail.com] Sent: Tuesday, February 19, 2008 12:29 AM To: Gupta, Indranil Subject: 525 review 02/19 Hello, Here are my reviews for tomorrow. Thanks! Justin OverCite: A Distributed, Cooperative CiteSeer This paper examined the benefits of essentially converting an existing centralized system to one backed by a distributed storage platform. They did this using a three tier design with a web server frontend, an index searcher, and a massive storage of all the documents in the system. The biggest difference between this system and the old system was in the construction of the storage tier. Where the centralized approach presumably used a traditional replicated database of some sort, this system uses a DHT to replicate and distribute the data over as many nodes as choose to participate in the system. This increases traffic substantially per query, but the goal is to distribute this cost across more sites than currently share in the costs of CiteSeer. By having many sites volunteer to participate in OverCite, the maintainers of CiteSeer are able to greatly reduce their own costs. In addition, the DHT backed design promises to scale much more efficiently than the old implementation. According to the paper, this easily justifies the increased costs in bandwidth and time. I really liked this paper. While much research goes into how to make commercial systems handle more throughput with higher reliability, this paper addresses a very real issue amongst non-profit organizations. Namely, how to support high bandwidth applications with limited funds and maintenance. These organizations also often have the benefit of large user bases who are interested in the continuation of whatever service the non-profit may be providing, and perhaps less interested in getting the best user experience money can buy. The system in this paper seems to offer a very acceptable solution to these types of situations. By allowing users to donate a piece of their own resources cheaply and without long term commitment, you substantively increase the robustness and scalability of the system while simultaneously reducing the non-profit's cost. Storage management and caching in PAST, a large-scale, persistent peer-to-peer storage utility This paper examined implementing a simple, replicated storage system on top of a distributed routing overlay. The implementation demonstrated used Pastry as the routing overlay, and it seems to have been developed with Pastry in mind. The system is exceptionally simple. The user hashes the filename with a bunch of additional information and a random salt. This hash is used as the ID, which is then passed off k (a user specified number) times to the nodes whose ids most closely match that of the file. There are various mechanisms in place to address node failures and overloading specific sections of the storage network, but that's the general idea. It's highly reliable and has consistent performance characteristics up until 95% of capacity is full. I really liked the simplicity of this paper. It seems like this system would be perfect for what they described, which is an archiving system. The latencies seem high, and the semantics are such that it seems impractical as any sort of live-data storage system, but it does provide high reliability with motivation to contribute to the overall storage of the system. If there existed a large community of people who wanted to aide in document archival, this might be an efficient way of doing it. I personally would want to wait until the work on file encoding and caching had been fully explored as I think that has a lot of potential to increase the speed and storage efficiency of the system. From: qwang26@uiuc.edu Sent: Tuesday, February 19, 2008 12:01 AM To: Gupta, Indranil Subject: 525 review: p2p apps (2/19/2008) Storage management and caching in PAST, a large-scale, persistent peer-to-peer storage utility PAST system is composed of nodes connected to the Internet where node is capable of initiating and routing client requests to insert or retrieve files, and the PAST nodes form a self-organizing overlay network. In this paper, authors describe and evaluate the storage management and caching in PAST. Typically, in the PAST system, storage nodes and files are each assigned uniformly distributed identifiers, and replicas of a file are stored at nodes whose identifier matches most closely the file’s identifier. This statistical assignment of files to storage nodes approximately balances the number of files stored on each node. However, the results in this paper show that the storage load balance provided by statistical assignment is insufficient to achieve high global storage utilization, given typical file size distributions and non-uniform storage node capacities. Instead, in this paper, authors present new algorithms for replica and file diversion which can ensure the availability of files while balancing the storage utilization. In addition, they also describe how to maintain the storage invariant in the presence of new node addition, node failure and recovery. Moreover, they describe the cache management policies with the goal to minimize client access latencies, to maximize the query throughout and to balance the query load in the system. Access latency is represented by fetch distance, which is measured by Pastry routing hops. OverCite: a distributed, cooperative CiteSeer This paper presents an approach OverCite to solve the problems lying in CiteSeer, such as expensive network traffic, large disk storage and significant human maintenance. OverCite is a new digital research library system that aggregates donated resources at multiple sties to provide CiteSeer-like document search and retrieval by letting members of community to share the costs of running CiteSeer. This paper focuses on dealing with how to provide scalable and load-balanced storage and query processing with automatic data management. In the design of OverCite, authors adopt a three-tier design, which enables OverCite to serve more queries per second than a centralized server, despite the addition of DHT operations and remote index communication. The experiment results in this paper show that OverCite increases the query throughput by a factor of seven with a nine-fold increase in the number of servers. Although OverCite requires more total storage and network bandwidth than centralized CiteSeer, it balances these costs over all the sites. Furthermore, additional resources provided by OverCite make a wider range of features possible, and programmers can realize more applications upon the substrate of OverCite. Therefore, further exploring these resources with new features such as document alert would be good for future research. From: Zixia Huang [zhuang21@uiuc.edu] Sent: Monday, February 18, 2008 11:07 PM To: indy@cs.uiuc.edu Cc: Huang, Zixia Subject: 525 review 02/19 Paper Title: OverCite: A Distributed, Cooperative CiteSeer Author: Jeremy Stribling, Jinyang Li, et al Summary: The main contribution of this paper is to introduce a new research library system OverCite which aggregates donated resources at multiple sites to provide search and retrievel like CiteSeer. It uses a three- tier design which provides a scalable and load-balanced scheme and ensures that the performance of OverCite can increase as volunteers contribute node. The paper provides a description overview of design by emphasizing the the query life, global and local data structures, as well as web crawler. The evaluation methods are based on testing the nodes over North America. Pros: (1) Three-tier design to increase the query throughput Cons: (1) Increase the network bandwidth. Although the total bandwidth required can be scattered over all sites, but considering some sites may not have enough bandwith and my be unreliable, the design may not be as good as the centralized server which can at least provide reliable query and response. (2) Actually I'm quite curious about the evaluation results that scatter over all the world but not just the north America. Because there is potential latency between trans-continental nodes, I'm wondering whether and to what extent it will influence the whole performance of OverCite? Paper Title: Colyseus: A Distributed Architecture for Online Multiplayer Games Author: Ashwin Bharambe, Jeffrey Pang, Srinivasan Seshan Summary: The main contribution of this paper is to present a distributed architecture called Colyseus which is used in the interactive multiplayer games. The hidden challenge facing Colyseus is somewhat similar to OverCite. When players scatter around the world, there is possibly no way to provide reliable yet efficient distributed architecture for the multiplayer games. The potential latency between two nodes can be high and thus may influence the real-time manipulation of the game. And sometimes, a link between two nodes may break and it will false the system to find a new distributed architecture and thus causing more delays. From: marefin2@uiuc.edu Sent: Monday, February 18, 2008 10:43 PM To: Gupta, Indranil Subject: 525 review 02/19 Storage Management and Caching in PAST, a Large Scale, Persistent Peer-to-Peer Storage Utility Antony Rowstrone, Peter Druschel PAST is built on top of Pastry. Each node maintains a leaf set, routing table and neighborhood set. The insertion and lookup are almost same as Pastry but PAST does some further improvement like load balancing for insertion and cache management for lookup. PAST stores a file to its k numerically closest neighbors according to its hash value. So, lookup can be served by any of the k nodes. Also PAST performs replica diversion and file diversion for load balancing. While storing files, if any of the k nodes finds that it has reached its file storage limit then it selects a node from its leaf set that is not among the k closest nodes of the file and does not already hold a diverted copy and then requests it to hold the file. The policy of accepting replica by a node is always biased on the metric SD/FN, where SD is the size of the file and FN is the remaining free storage space of node N. A node rejects a file if SD/FN > t, where t is set to some threshold value. But if a node fails, the diverted files that the failed node has already stored to some nodes of the leaf set will be inaccessible. So, while diverting a copy, PAST selects (k+1)nth closest node of a file to point to that diverted copy and it is reasonable as it will be in the k closest set if a node fails. When a file insert operation fails because the any of the k closest nodes do not have enough storage and couldn’t find any node to send the diverted replica, then a failure is reported to the initial client node. It then generates a new nodeId (hash of filename, owner public key and randomly chosen salt) for the file using different salt value and retries the insertion. Now, when a new node joins, it is inefficient to request replica of all files for which it has just become one of the k numerically closest nodes. Instead it can store a pointer to the file tables referring to the node that has just ceased from the set of k node for that file. And the situation can be updated later lazily as a background operation. Also when one leaf node fails that contains the diverted copy, then the original node (from one of the k numerically closest node) tries to replicate it in some other leaf node. If it also fails due to storage limitation, it will asks the two distance node in its leaf set to check their leaf set to find any node (not among the k closest node) that can store the file. Otherwise PAST assumes that the system couldn’t have k copy at that time due to less available disk space. PAST even increases the efficiency of lookup by caching the data. PAST uses unused portion of advertised storage space for caching. A file that is routed through a node for lookup or insertion is actually inserted into local disk cache of that node. If the node needs to store a file as the normal insertion process, it removes the cache block with GD-S policy. PAST is an internet-based peer-to-peer storage utility, which aims to provide strong persistence, high availability, scalability and security. PAST nodes form a self-organizing overlay network. It uses replica diversion and file diversion for efficient storage utilization. Its Replica diversion prefers a long file to several shorter files (to reduce network overhead) and it doesn’t bother when the memory utilization at each individual node is low (which makes sense). The overhead of replica diversion is an additional entry in the file tables of two nodes, two additional RPCs during insert and one additional RPC during lookup. PAST has been implemented in JAVA and the result shows efficient load balancing and storage utilization. It adapts the efficient robustness and lookup structure from Pastry. PAST achieves global storage utilization in access of 98%. Also PAST is concerned about security. Malicious node attacks in routing are solved by randomization. Its caching approach increases the lookup performance. OverCite: A Distributed, Cooperative CiteSeer J. Stribling, J. Li, I. G. Councill, M. F. Kaashoek, R. Morris This paper explores the design and issues of OverCite, a multi-site version of CiteSeer repository of computer science research papers. The actual goal of OverCite is to distribute the system’s load over the few hundred servers (those want to join voluntarily) around the world. It addresses popular 3-tier design architecture. Tier-1 is the presentation layer. A subset of OverCite nodes run a web server interface, using round-robin DNS to spread the client load. They accept keyword queries from users and aggregate result from Tier 2 to present to users. Tier-2 is the application logic layer that performs keyword searches and crawling for new documents. Tier-3 is the data storage layer that uses DHTs to store Meta information of the documents. Nodes coordinate the crawling effort through a list of to-be-crawled page URLs stored in the DHT. Each crawler process periodically chooses a random entry from the list and fetches the corresponding page (if not already fetched). When the crawler finds a new document file, it extracts the document’s text words and citations, and stores the document file, the extracted words, and the document’s meta-data in the DHT. The node also adds the document’s words (a fixed number) to its inverted index. OverCite partitions the inverted index among many participating nodes. If there are n number nodes and k partitions, each partition contains n/k number of nodes. OverCite partitions the index by documents rather than keyword to avoid expensive join on multi keyword queries and also it limits the communication necessary on document insertions. OverCite sends a copy of each query to one server in each partition, so that only k servers are involved in each query. Each of the k servers uses about 1/k’th of the CPU time that would be required to search a single full-size inverted index. Each server returns only the DIDs of the m highest-ranked documents (by some specified criterion, such as citation count) in response to a query. The current implementation of OverCite includes four modules to each node, the OCWeb module that provides a web interface to accept keyword queries, the DHTStore module that acts as an interface between the rest of the OverCite system and the DHT, the Meta-data Storage module that contains information about DID, CID, GID, Shins, Crawl and URLs, and the Index/Search module that actually contains the query server daemon to query among the partitions. CiteSeer is a familiar online resource for computer science research community. It allows users to search and browse large number of research paper archive. But CiteSeer is expensive in term of bandwidth, storage and maintenance. Although members of the community are willing to donate hardware and bandwidth, the structure of CiteSeer doesn’t support it. OverCite provides a distributed and cooperative research library based on distributed hash table (DHT) that provides the facility of using distributed nodes around the world. Also the tier architecture provides parallelism in each tier of the design. That’s why the average query throughput and the throughput of serving files from the DHT increase with the increase of number of web servers. The paper also focuses some of the future works. OverCite could help by providing email alert to the researchers when a paper is entered in the database that might be of interest. Also CiteSeer still doesn’t show the conference name of the corresponding paper that is sometime necessary for the researchers. From: Mirko Montanari [mirko.montanari@gmail.com] on behalf of Mirko Montanari [mmontan2@uiuc.edu] Sent: Monday, February 18, 2008 10:04 PM To: Gupta, Indranil Subject: 525 review 02/19 "OverCite: A Distributed, Cooperative CiteSeer" This paper describes OverCite, a distributed implementation of CiteSeer. The system uses a three-tier architecture to distribute the load between multiple nodes. The first tier is the web interface, replicated on multiple nodes. Each of these nodes communicates with a second tier to perform keyword searches. The storage is provided by a DHT system that stores documents, meta-data and shared state for coordination. The system divides the nodes into k partitions. Each partition manages a part of the documents of the system. Every node in the partition maintains an inverted index of the documents that the partitions manages. For each keyword query the front end server sends the query to the different partitions and aggregate the answers. PRO: + The three-tier architecture is a very common web architecture that could ease the adoption of an system like OverCite in the industry. + Easy to add new nodes to the system and use their computation and storage power. + The DHT back-end can be applied in other systems. CONS: - Big overhead in bandwidth: the bandwidth used for crawling and fetching new documents is three times the one used by CiteSeer. Even if this bandwidth is not concentrated in only one server, it is still a big overhead that can probably be reduced. - The authors don't provide a way to select which k servers should be used by the front-end at every query. The selection of these servers can influence the response time of the system and the load balancing. - Big overhead in space: n nodes provides only n/4 as many documents as a single CiteSeer implementation. This storage is distributed, but the overhead can not be ignored. --------------- "Colyseus: A Distributed Architecture for Online Multiplayer Games" This paper describes the implementation of a distributed gaming environment suited for First Person Shooters (FSP), like Quake II. One of the main parts of FSPs is the state of the virtual world, represented by the union of the states of the single objects contained in the game. The state of the world gets updated at every frame computation (10-20 times per second). At every computation cycle, the system invokes a specific "think function" associated to each of the objects. This function reads the state of other objects and updates its own object's state. Objects can also be modified by the players through interaction with them. The main idea of Colyseus is to distribute this objects to multiple machines. The authors propose to associate every object in the system with a particular machine that updates the object's state by calling the think function. This machines also receives "write requests" from other players in the game. This association is made based on the "locality principle", where each user is usually interested only in a subset of the system's objects. Write requests are serialized and applied to the object. Other machines can keep a read-only copy of other objects in the system in order to improve performance. The lookup of new objects and the distribution of the object updates are delivered through a DHT layer. Range queries on the DHT are used to retrieve objects of interest and a publisher / subscriber mechanism is used to distribute updates. PRO: + The analysis is based on a real workload and the implementation allows a real test case of the system. + The authors performed a scalability analysis of the single server solution to compare the distributed solution with a "single server" one. + The authors propose evidences that the implementation of a distributed game architecture that integrates the users themselves to provide resources for the game is feasible. + The author uses dynamic load balancing to distribute the load of heavy loaded regions. CONS: - There is no description of what happens when a node joins or leaves the system while the game is running. What happen to the objects that it manages? How is the new object primary server elected? - The authors do not address the problem of failing nodes: what happen if one of the nodes fails? What is the delay introduced? - Users might be connected with a low-bandwidth connection to internet. What is the delay introduced in the whole system by such high-delay / low-bandwidth connections? - The DHT system seems to lose the connection between the geography and the distribution of data. Game geography seems to play a big rule in the interactions between players and object and having the ability to exploit this correlation in the DHT layer might be interesting. -- Mirko Montanari From: Anthony Cozzie [acozzie@gmail.com] Sent: Monday, February 18, 2008 8:57 PM To: Gupta, Indranil Subject: 525 review 02/19 PAST PAST is a read/insert filesystem built on Pastry. The primary improvements over Pastry appear to be replicating each file K times, load balancing by allowing the k nodes with closest nodeids to offload the file to one of their neighbors, and by caching heavily requested files. My biggest problem with this work is finding a good application. While the persistant nature of fileystems meshes well with the churn problems of DHTs, the question is what files to store on the thing. I don't want to store my files on other computes for privacy reasons if nothing else. I like the web cache, but I really cannot think of anything else. For example, is it really better to store Grid computing files on PAST as opposed to just hosting them on an FTP server? Also, I think I would prefer a block-oriented filesystem like CFS or GFS, which would give better parallelism and load balancing, although admittedly increasing the overhead. Technically I find this a pretty straightforward extension of Pastry, without significant additional complexity. OverCite OverCite is an attempt to build a principled distributed systems solution to the problems of Citeseer (since overshadowed by Google Scholar, but a big deal even 4 years ago). The theory being that Citeseer was representative of the new dynamic webpages of the day (although I disagree with this). OverCite works in three stages: webservers, index servers, and a DHT that stores the documents. + OverCite has pretty reasonable results (although this is also on Internet2) with distributing the CiteSeer workload through a very generic distributed system. - CiteSeer is not really an example of standard dyanmic webpage, because there are relatively few insertions, when compared to (say) a forum or Ebay (or any Web2.0 application that relies on users to build the contents). From: Hengzhi Zhong [hzhong@uiuc.edu] Sent: Monday, February 18, 2008 8:50 PM To: Gupta, Indranil Subject: 525 review 02/19 OverCite: A Distributed, Cooperative CiteSeer Summary: OverCite is the multi-site version of CiteSeer. It tries to answer the question, how to increase its capacity as more sites are added to a repository like CiteSeer? The idea is that with multiple sites added, we can parallelize dynamic operations over many sites so that we can answer more queries. Also, given multiple sites, we can tolerate network and site failures. The design of OverCite is a 3-tiered architecture with a DHT to store page content and metadata. The first tier is an interface between users and the data; the second tier does the keyword search and crawling. The third tire is the data storage, which consists of local index nodes and a DHT. On search, each index node searches its local indexes and looks up the metadata in the DHT. Then the results are merged and returned at the top tier. This design is meant for Web services that have a pretty static data storage and frequent dynamic operations. Pro: 1. OverCite can reduce network load since it can use multiple sites. 2. OverCite can handle more queries in parallel Cons: 1. Why not store page content WITH the local index nodes? It seems more efficient this way, in terms of returning results back to the users. 2. Increasing the number of index nodes slows down DHT lookups, which is expected. However, on the web setting, the repository size tends to be large and requires a large number of index nodes to maintain inverted index lookup speed. Can this design still work efficiently? How does DHT scale with respect to the number of index nodes? 3. All nodes are sharing in the same DHT. Doesn’t this cause some bottleneck? Storage Management and Caching in PAST, a large-scale, persistent peer-to-peer storage utility Summary: PAST is a large-scale, peer-to-peer storage utility. It is layered on top of PATRY and uses it for routing and file search. Using Pastry, PAST tends to find the “nearest” copy of a file. This paper evaluates its storage management and caching. In storage management, it needs to look at what happens when the system approaches maximum storage utilization. The goal is to ensure file availability and balance storage load. To do so, it uses replica diversion and file diversion. For caching, the goal is to use cache to maximize query throughput while minimizing access latencies. Further, the query load must be balanced. Whenever a file is routed through a node, the file is cached as long as there is space in the cache. The cache replacement policy is the GreedyDual-Size policy which mains weight for each cached file. Weights are updated on insertion or cache hit. A file is deleted when its weight is the smallest. Pros: 1. PAST uses the Internet to achieve high availability. 2. Since PAST uses Pastry to route the requests, it tends to find the “nearest” copy of a file. 3. PAST balances storage load while maintaining file availability. Cons: 1. Since the properties of PAST (as presented in the paper) depend on its routing algorithm and Pastry is the one it uses, local choices may not be the best choice. The chosen route may have failed nodes. A client may have to issues multiple requests until a good route is chosen. How bad can this bad? 2. Cache space is the unused portion of the disk space in the nodes. In other words, there will be space contention. When storage usage goes up, cache performance goes down. 3. The experiments do not show the situation when some pages are more popular than others. What happens on this case? From: rebolledodaniel@gmail.com on behalf of Daniel Rebolledo Samper [dreboll2@uiuc.edu] Sent: Monday, February 18, 2008 8:26 PM To: Gupta, Indranil Subject: 525 review 02/19 STORAGE MANAGEMENT AND CACHING IN PAST, A LARGE-SCALE, PERSISTENT, PEER-TO-PEER STORAGE UTILITY PAST is a Pastry-based DHT that doubles as a file store. It supports file insertion, retrieval and revocation (but does not guarantee permanent delition) in the pastry ting. PAST stores multiple copies of a single file to resist node failure: each of the k copies of a file is stored at the k nodes closest to the node whose nodeId is the closest to the fileId in Pastry. If the storage capacity of those each of those k nodes is insufficient to store the file (or the file is too large relative to the available size – this is configurable) the nodes make a few attempts to store the file in their leaf nodes or in other nodes. However, file insertion is not guaranteed and may fail, in which case the originating node is notified. PAST provides a measure of security by using public key encryption: each node has a public/private key pair and each time a file is inserted, stored or revocated, each of the nodes involved produces certificates or receipts proving the authenticity of the file or the fact that it was stored or deleted. The paper is very interesting in that it actually tries to provide a measure of security. Even though its properties are not studied in depth, it is important to think of these issues if we expect the systems to be deployed on a broader scale than in the lab. The authors also show impressive experimental results: they manage to obtain a better than 90% utilization rate, and the system doesn't really start degrading until about 85% of the total disk space has been used. Even then, it degrades gracefully. Another important idea is that of opportunistic handover: when a new node enters Pastry it may become responsible for a number of files, and consequently some other nodes are no longer responsible for them. Instead of immediately initiating costly file transfers, the new node creates pointers the files on those nodes, and transfers them in the background. The authors also suggest that instead of keeping k copies of the file (which seems rather wasteful) we should use redundant storage algorithms like Reed-Solomon: we could then provide high resilience to node failure while at the same time limiting file overhead. However, the paper has a couple of flaws: the effect of node failure is not really examined, though we would expect similar results as in Pastry. However, here again the impact of churn is not examined nor mentioned. Also, perhaps it could have been interesting to summarize the way the quota system works (as this, in itself, seems to suggest a central control entity). OVERCITE OverCite is a distributed system for research paper storage, indexing and retrieval. Its aim is to replace CiteSeer, which provides the same services but in a centralized way, to distribute the storage and bandwidth cost among several institutions. OverCite uses a three-layer architecture: the core of the system is a DHT that stores the documents and the tables describing them. An application logic layer talks to the DHT to execute user queries and merge their results. Users access the system through a third layer (the "presentation layer"), which is essentially a series of HTTP servers configured with round-robin DNS to increase load balancing. Experimental results show that the system can handle a number of requests linear in the number of front-end servers (at least as far as the available numbers are concerned) while delivering up to 500 times the throughput CiteSeer currently uses. In the experiments presented in this paper, the system uses four times more storage capacity than CiteSeer. The authors present a way to provide web services in a totally distributed way. However, their approach seems hard to generalize. Indeed they rely very heavily on the absence of updates: the requirements on the DHT are only that it be possible to add new files and append data to existing files. Yet in many web services, it seems that updating data is necessary. This paper does, however, remind us that theoretical research can have very interesting real-world applications. From: emenese2@uiuc.edu Sent: Monday, February 18, 2008 2:50 PM To: Gupta, Indranil Subject: 525 review 02/19 Paper: Storage management and caching in PAST, a large-scale, persistent peer-to-peer storage utility Reviewer: Esteban Meneses (emenese2@uiuc.edu) PAST is a peer-to-peer storage system which offers (according to the authors) strong persistence, high availability, scalability and security. A PAST system is composed of an overlay of nodes connected to the Internet, each of which is capable of starting or routing request to insert or search for files. Each file in the PAST system is assigned a given quasi-unique fileID that is created at the time of the file insertion. Then, files are immutable as they cannot be inserted several times with the same fileID. Therefore, PAST is intended as an archival storage and content distributed tool and not be considered as a general purpose file system. PAST was built on top of the Pastry substrate and thus relies on the ability of Pastry to route requests in log(N) time, to consider locality in lookups, to deal with faults in nodes, etc. However, at the same time it also inherits all the drawbacks of Pastry as it is the case of strong invariants required to keep the system working. Once a file insertion operation is performed over one node, a file certificate is created and signed with the private key of the owner of the file. Once the file is routed to its closest ID in the system, the file is replicated in the nearest k nodes of the system, using the leaf set table of the Pastry layer. Replica diversion is the process the allows a node that is not in the k closest nodes (but it is in the leaf set of one of such k nodes) to alternatively store the file and the objective is to equilibrate differences in the storage capacity and utilization of nodes within a leaf set. File diversion is executed when a node's entire leaf set is reaching its total capacity. The purpose is to load balance across large portions of the nodeID space. The way to divert the fileID into a difference section of the nodeID space is to change the salt in the generation of its fileID. The way PAST keep the number of replicas of a file around the value of k is by having a heartbeat mechanism. In this way, neighboring nodes in the nodeID space exchange alive messages. After a period of T, if a node hasn't received any message from its neighbor, it assumes that node has failed and a recover mechanism is put into place to reestablish the required entries in the tables as well as to replicate the file. PAST also uses a caching system in the free space of PAST nodes to run the GDS (greedy dual size) policy. The advantages of PAST reside in the fact that it exploits multitude and diversity of nodes in the overlay to gain availability and persistence. This is thank to the underlying DHT services of Pastry. The replication mechanism also provides a quick way to locate the file in case of failures (neighboring nodes are supposed to be close in the locality metric of Pastry). The caching system may help to reduce the latency in accessing files. On the other side, PAST is condemned to suffer from the same vulnerabilities of Pastry, the underlying DHT layer. It is not clear that PAST is resilient to high churn. The same overhead for keeping the invariants of the system has to be paid in PAST. The PAST system handles immutable objects. Should it be the case for most p2p systems? From a consistency point of view, the object model is trivial: there is no WRITE operations being performed over the objects. The system is write-once in this sense. I wonder how difficult it would be to develop a more relaxed version of PAST (not exactly a file system, though) where certain write operations were permitted. Also, I have seen that most of the p2p systems (the academic ones) are based on the DHT concept. Is it the only way to implement p2p capabilities? Another point stands for the storage in the nodes, which is considered to be fixed. What about failures in the storage system of one particular node? Could we extend PAST to deal with dynamic storage in nodes? Again, PAST most face the invariants of the underlying Pastry substrate, which are costly to maintain. Is there any way to minimize this cost without violating the invariants? Besides, the replication vs file encoding dilemma is also present in PAST. Replication is good for availability, but not as good for failure resilience as file encoding. I don't see any really big contribution in this paper (compared to the Pastry one). If you read the Pastry paper, you could guess what the decisions in PAST would be. I believe that most of the work on PAST was an straightforward continuation of the work on Pastry. Most of the ideas (replication, heartbeating, caching and the rest) come from many other areas and are not really original in the paper. In any case, PAST doesn't involve a further improvement of Pastry, the same set of deficiencies are present in PAST. Moreover, the failure issue is not treated in this paper. Where are the node failure experiments? Paper: OverCite: A Distributed, Cooperative CiteSeer Reviewer: Esteban Meneses (emenese2@uiuc.edu) Most of the graduate students in Computer Science have used CiteSeer web site at least once in their lives, either for looking up specific papers or for analyzing the references made over a particular citation. CiteSeer works by having a centralized approach in crawling the papers, indexing them and providing the query engine and file retrieval operations. OverCite is the distributed version of CiteSeer. What does “distributed” mean? Well, for the authors of the paper, it means distribute every task in the original CiteSeer design: crawling, indexing, storing, query solving and retrieving tasks are performed by a set of distributed entities. One of the features of OverCite consists in its three-layer design (very common): a load-balancing front-end, an application layer and finally a shared back-end database. What this distributed version adds to the CiteSeer is to increase the performance, minimize congestion in a particular site by partitioning the storage, improving the resilience to faults and parallelizing the crawling and query analysis. The three layers of OverCite attack this requirements by having multiple web front-ends that interact with the user, application servers that crawl the Web, generate indexes and perform keyword searches in the index and finally a back-end DHT that aggregates the storage space of donated machines to keep the documents, meta data and coordination state. However, the DHT layer provides several advantages as being self-managed with load balancing. It also performs data replication and routing of queries to provide high availability. Finally, it offers a rendezvous point for producers and consumers of meta-data. The advantage of any distributed approach for an original centralized tool is that we can add more failure robustness, performance increase, high availability, scalability, etc. OverCite seems to add many of them as all the tasks are distributed among the participants. OverCite is also based on a DHT model for its database layer, which ensures that many issues are solved at that level. However, there are several drawbacks in the OverCite approach. For several popular queries, the response time can be higher, as the overhead of a distributed approach prevents the caching system to act quickly. Also, several processes must be checked in order to prove they are equivalent to the centralized version (particularly the distributed indexing process). I found really exciting this paper, although it doesn't detail any of the topics. One of the points that I consider more interesting is the distributed caching system. It may be interesting to explore (I ignored if this is an old research problem) how the query caching system works. For example, one site can share all the recent and popular queries with other sites, so that answers reach the user more quickly. Furthermore, what about adding gossip-style protocols to do this? On the other side, it seams that OverCite is pretty static in the number of distributed nodes that conform the distributed substrate. How can we make it more flexible? If we had not only “append-only” blocks over the DHT, but fully mutable blocks. Using this idea we could have cooperative editing systems. This paper lacks many details in several of the sections. It is not clear how the global index is updated and how replication works in OverCite. On the other hand, the number of distributed nodes that carry on the information retrieval tasks is very static. It is not clear how fault tolerance is accomplished at this level. If one index node fails, then there is no way to access the files in that node.