deals with hardware
and software systems
containing more than one processing element or storage
processes, or multiple programs, running under a loosely or tightly controlled regime
In distributed computing a program is split up into parts that run simultaneously on multiple computers communicating over a network. Distributed computing is a form of parallel computing, but parallel computing is most commonly used to describe program parts running simultaneously on multiple processors in the same computer. Both types of processing require dividing a program into parts that can run simultaneously, but distributed programs often must deal with heterogeneous environments, network links of varying latencies, and unpredictable failures in the network or the computers.
Organizing the interaction between the computers that execute distributed computations is of prime importance. In order to be able to use the widest possible variety of computers, the protocol or communication channel should not contain or use any information that may not be understood by certain machines. Special care must also be taken that messages are indeed delivered correctly and that invalid messages, which would otherwise bring down the system and perhaps the rest of the network, are rejected.
Another important factor is the ability to send software to another computer in a portable way so that it may execute and interact with the existing network. This may not always be practical when using differing hardware and resources, in which case other methods, such as cross-compiling or manually porting this software, must be used.
Goals and advantages
There are many different types of distributed computing systems and many challenges to overcome in successfully designing one. The main goal of a distributed computing system is to connect users and resources in a transparent
, open, and scalable
way. Ideally this arrangement is drastically more fault tolerant
and more powerful than many combinations of stand-alone
Openness is the property of distributed systems such that each subsystem is continually open to interaction with other systems (see references). Web services protocols
are standards which enable distributed systems to be extended and scaled. In general, an open system that scales has an advantage over a perfectly closed and self-contained system. Openness cannot be archived unless the specification and documentation of the key software interface of the component of a system are made available to the software developer.
Consequently, open distributed systems are required to meet the following challenges: Monotonicity
- Once something is published in an open system, it cannot be taken back. Pluralism
- Different subsystems of an open distributed system include heterogeneous, overlapping and possibly conflicting information. There is no central arbiter of truth in open distributed systems. Unbounded Nondeterminism
- Asynchronously, different subsystems can come up and go down and communication links can come in and go out between subsystems of an open distributed system. Therefore the time that it will take to complete an operation cannot be bounded in advance.
Drawbacks and disadvantages
If not planned properly, a distributed system can decrease the overall reliability
of computations if the unavailability
of a node can cause disruption of the other nodes. Leslie Lamport
famously quipped that: "A distributed system is one in which the failure of a computer you didn't even know existed can render your own computer unusable.
Troubleshooting and diagnosing problems in a distributed system can also become more difficult, because the analysis may require connecting to remote nodes or inspecting communication between nodes.
Many types of computation are not well suited for distributed environments, typically owing to the amount of network communication or synchronization that would be required between nodes. If bandwidth, latency, or communication requirements are too significant, then the benefits of distributed computing may be negated and the performance may be worse than a non-distributed environment.
Various hardware and software architectures are used for distributed computing. At a lower level, it is necessary to interconnect multiple CPUs with some sort of network, regardless of whether that network is printed onto a circuit board or made up of loosely-coupled devices and cables. At a higher level, it is necessary to interconnect processes
running on those CPUs with some sort of communication system
Distributed programming typically falls into one of several basic architectures or categories: Client-server, 3-tier architecture, N-tier architecture, Distributed objects, loose coupling, or tight coupling.
- Client-server — Smart client code contacts the server for data, then formats and displays it to the user. Input at the client is committed back to the server when it represents a permanent change.
- 3-tier architecture — Three tier systems move the client intelligence to a middle tier so that stateless clients can be used. This simplifies application deployment. Most web applications are 3-Tier.
- N-tier architecture — N-Tier refers typically to web applications which further forward their requests to other enterprise services. This type of application is the one most responsible for the success of application servers.
- Tightly coupled (clustered) — refers typically to a cluster of machines that closely work together, running a shared process in parallel. The task is subdivided in parts that are made individually by each one and then put back together to make the final result.
- Peer-to-peer — an architecture where there is no special machine or machines that provide a service or manage the network resources. Instead all responsibilities are uniformly divided among all machines, known as peers. Peers can serve both as clients and servers.
- Space based — refers to an infrastructure that creates the illusion (virtualization) of one single address-space. Data are transparently replicated according to application needs. Decoupling in time, space and reference is achieved.
Another basic aspect of distributed computing architecture is the method of communicating and coordinating work among concurrent processes. Through various message passing protocols, processes may communicate directly with one another, typically in a master/slave relationship. Alternatively, a "database-centric" architecture can enable distributed computing to be done without any form of direct inter-process communication, by utilizing a shared database.
Distributed computing implements a kind of concurrency
. It interrelates tightly with concurrent programming
so much that they are sometimes not taught as distinct subjects .
A multiprocessor system is simply a computer that has more than one CPU on its motherboard. If the operating system is built to take advantage of this, it can run different processes (or different threads belonging to the same process) on different CPUs.
Intel CPUs from the late Pentium 4 era (Northwood and Prescott cores) employed a technology called Hyper-threading that allowed more than one thread (usually two) to run on the same CPU. The more recent Sun UltraSPARC T1, AMD Athlon 64 X2, AMD Athlon FX, AMD Opteron, Intel Pentium D, Intel Core, Intel Core 2, Intel Core 2 Quad, and Intel Xeon processors feature multiple processor cores to also increase the number of concurrent threads they can run.
A multicomputer may be considered to be either a loosely coupled NUMA
computer or a tightly coupled cluster
. Multicomputers are commonly used when strong compute power is required in an environment with restricted physical space or electrical power.
Common suppliers include Mercury Computer Systems, CSPI, and SKY Computers.
Common uses include 3D medical imaging devices and mobile radar.
The types of distributed systems are based on Flynn's taxonomy
of systems; single instruction, single data
(SISD), single instruction, multiple data
(SIMD), multiple instruction, single data
(MISD), and multiple instruction, multiple data
(MIMD). Other taxonomies and architectures available at Computer architecture
and in Computer architecture
A cluster consists of multiple stand-alone machines acting in parallel across a local high speed network. Distributed computing differs from cluster computing
in that computers in a distributed computing environment are typically not exclusively running "group" tasks, whereas clustered computers are usually much more tightly coupled. Distributed computing also often consists of machines which are widely separated geographically.
A grid uses the resources of many separate computers, loosely connected by a network (usually the Internet), to solve large-scale computation problems. Public grids may use idle time on many thousands of computers throughout the world. Such arrangements permit handling of data that would otherwise require the power of expensive supercomputers
or would have been impossible to analyze.
Nearly any programming language
that has access to the full hardware
of the system could handle distributed programming given enough time and code. Remote procedure calls
distribute operating system
commands over a network connection. Systems like CORBA
, Microsoft DCOM
, Java RMI
and others, try to map object oriented
design to the network. Loosely coupled systems communicate through intermediate documents that are typically human readable (e.g. XML
, and EDI
A variety of distributed computing projects have grown up in recent years. Many are run on a volunteer basis, and involve users donating their unused computational power to work on interesting computational problems. Examples of such projects include the Stanford University Chemistry
project, which is focused on simulations of protein folding
to find disease cures and to understand biophysical systems; World Community Grid
, an effort to create the world's largest public computing grid to tackle scientific research projects that benefit humanity, run and funded by IBM
, which is focused on analyzing radio-telescope data to find evidence of intelligent signals from space, hosted by the Space Sciences Laboratory
at the University of California, Berkeley
, which is used to help design and tune the Large Hadron Collider
, hosted by CERN
; and distributed.net
, which is focused on breaking various cryptographic ciphers.
Distributed computing projects also often involve competition with other distributed systems. This competition may be for prestige, or it may be a matter of enticing users to donate processing power to a specific project. For example, stat races are a measure of the work a distributed computing project has been able to compute over the past day or week. This has been found to be so important in practice that virtually all distributed computing projects offer online statistical analyses of their performances, updated at least daily if not in real-time.
- Attiya, Hagit and Welch, Jennifer (2004). Distributed Computing: Fundamentals, Simulations, and Advanced Topics. Wiley-Interscience. ISBN 0471453242.
- Lynch, Nancy A (1997). Distributed Algorithms. Morgan Kaufmann. ISBN 1558603484.
- Tel, Gerard (1994). Introduction to Distributed Algorithms. Cambridge University Press.
- Davies, Antony (2004). "Computational Intermediation and the Evolution of Computation as a Commodity". Applied Economics – Scholar search}}
- Kornfeld, William (1981). "The Scientific Community Metaphor". MIT AI (Memo 641):
- Hewitt, Carl (1983). "Analyzing the Roles of Descriptions and Actions in Open Systems". Proceedings of the National Conference on Artificial Intelligence .
- Hewitt, Carl (1985). "The Challenge of Open Systems". Byte Magazine
- Hewitt, Carl "Towards Open Information Systems Semantics". Proceedings of 10th International Workshop on Distributed Artificial Intelligence .
- Hewitt, Carl (1991). "Open Information Systems Semantics". Journal of Artificial Intelligence 47 79.
- Nadiminti, Dias de Assunção, Buyya (2006). "Distributed Systems and Recent Innovations: Challenges and Benefits". InfoNet Magazine, Volume 16, Issue 3, Melbourne, Australia
- Bell, Michael Service-Oriented Modeling: Service Analysis, Design, and Architecture. Wiley. (2008). .