|
![]() |
Computational Peer-To-Peer (P2P) is becoming a key execution environment. The potential of 100,000 of nodes interconnected to execute a single application is rather appealing, especially for Grid computing. Mimicking data P2P, one could start a computation that no failure would ever be able to stop (and maybe nobody).
The ProActive P2P aims to use spare CPU cycles from organizations or institution's desktop workstations.
This short document explains how to create a simple computational P2P network. This network is a dynamic JVMs network which works like, computational nodes.
The P2P infrastructure works as an overlay network. It works with a P2P Service which is a peer which in turn is computational node. The P2P Service is implemented with a ProActive JVM and few Actives Objects. The next figure shows an example of a network of hosts where some JVMs are running and several of them are running the P2P Service.
Example of a ProActive P2P infrastructure.
When the P2P infrastructure is running, it is very easy to obtain some nodes (JVMs). The next section describes how to use it.
Further research information are available here.
The outline of this short handbook:
The P2P infrastructure is self-organized and configurable. When the infrastructure is running you have nothing to do to keep it up. There are 3 main parameters to configure:
All parameters description and the way to change their default values are explained here. Next section shows how to configure the infrastructure when starting the P2P Service with the command line.
The bootstrapping or first contact problem is how a new peer can join the p2p infrastructure. We solved this problem by just specifying one or several addresses of supposed peers which are running in the p2p infrastructure. Next, we will explain how and where you can specify this list of peers.
Now, you have just to start peers. There are two ways to do it:
This method explains how to rapidly launch a simple P2P Service on one host.
ProActive provides a very simple script to start a P2P Service on your local host. The name of this script is startP2PService.
Before launching this script, you have to specify some arguments to this command:
startP2PService [-acq acquisitionMethod] [-port portNumber] [-s Peer ...] [-f PeersListFile]
rmi://applepie.proactive.org:8080
More options:
All arguments are optional.
Comment: With the UNIX version of the startP2PService script, the P2P service is persistent and runs like a UNIX nice process. If the JVMs that are running the P2P service stop (for a Java exception) the script re-starts a new one.
In this illustration, we will explain how to start a first peer and then after how new peers can create a P2P network with the first one.
Start the first peer with rmi protocol and listening on port 2410:
first.peer.host$startP2PService.sh rmi 2410
Now, start new peers and connect them to the first peer to create a tiny P2P network:
second.peer.host$startP2PService.sh rmi 2410 -s rmi://first.peer.host
third.peer.host$startP2PService.sh rmi 2602 -s rmi://first.peer.host
You could specify different port number for each peer.
Use a file to specify address of peers:
The file hosts.file:
rmi://first.peer.host:2410
rmi://third.peer.host:2602
file.peer.host$startP2PService.sh rmi 8989 -f hosts.file
Last, a new peer joins the P2P network by another peer of the first:
last.peer.host$startP2PService.sh rmi 6666 -s rmi://third.peer.host:2410
Usage example P2P network (after firsts connections).
The daemon aim to use computers in Peer-to-Peer computations, there will be a Java virtual machine sleeping on your computers and waking up at scheduled times to get some work done.
By default, the JVM is scheduled to wake up during the weekend and during the night. Next, we will explain how to change the schedule. The JVM is running with the lowest priority.
Go to the directory: ProActive/compile and run this command:
$ ./build daemon
Before compile you should change some parameter like the daemon user or the port in the file:
ProActive/p2p/src/common/proactivep2p.h
Ask to your system administrator to add the daemon in a crontab or init.d. The process to run is located here:
ProActive/p2p/build/proactivep2p
To compile daemon source (in c++), we don't provide any automatic script, you have to do it by yourself. All sources for Windows are in the directory: ProActive/p2p/src/windows. If you use Visual Studio or else in the same directory you found Microsoft projects files.
After that you are ready to install the daemon with Windows, you have just to run this script:
C:>ProActive\scripts\windows\p2p\Service\install.bat
To remove the daemon:
C:>ProActive\scripts\windows\p2p\Service\remove.bat
Comment: By default the port number of the daemon is 9015.
The daemon is configured with XML files in the ProActive/p2p/config/ directory. To find the correct configuration file, the daemon will first try with a host dependent file: config/proactivep2p.${HOST}.xml for example: config/proactivep2p.camel.inria.fr.xml if the daemon is running on the host named camel.inria.fr.
If this host specific file is not found, the daemon will load config/proactivep2p.xml. This mechanism can be useful to setup a default configuration and have a specific configuration for some hosts.
The reference is the XML Schema in proactivep2p.xsd. For those not fluent in XML Schema, here is a description of every markup tags--
The root element in <configFile> it contains one or many <p2pconfig> . This latter element can start with some <loadconfig path="path/to/xml"/> they will include the designated XML file. After these file inclusions, you can with <host name="name.domain"> specify which hosts are concerned by the configuration. Then there can be a <configForHost> element containing a configuration for the selected hosts and/or a <default> element if no suitable configuration was already found.
Keep in mind that the XML parser sees a lot of configuration and the first that matches is used and the parsing is finished. This means that the elements we have just seen are tightly linked together. For example if an XML file designated by a <loadconfig> contains a <default> element, then after this file no other element will be evaluated. This is because either a configuration was already found so the parsing stops, or no configuration matched and the <default> does, so the parsing ends.
The proper configuration is contained in a <configForHost> or <default> element. It consists of the scheduled times for work and the hosts where we register ourselves. Here is an example:
|
In this example we clearly see that the JVM will wake up Monday evening and shut down Tuesday morning. It will also work during the weekend. In the <register> part we put the URL in which we will register ourselves, in the example we used the short form which is equivalent to rmi://host:9301.
Follow commands work only with UNIX friendly systems.
$ProActive/p2p/build/p2pctl stop [hostname]
$ProActive/p2p/build/p2pctl killdaemon [hostname]
$ProActive/p2p/build/p2pctl restart [hostname]
$ProActive/p2p/build/p2pctl alive [hostname]
$ProActive/p2p/build/p2pctl flush [hostname]
hostname is the name of the remote host which the daemon command is sent. This parameter is optional, if host name is not specified the command is executed on the local host.
Under Windows you could use some littles scripts in ProActive//script/windows/p2p/JVM to do that.
All daemon logs are wrote in a file. All logs are available in:
ProActive/p2p/build/logs/hostname
You can customize some P2P settings such as:
With elements acq and port if a P2P Service is already running with this configuration the descriptor will use this one else a new one is started.
In order to get nodes, the peerSet tag will allow you to specify entry point of your P2P Infrastructure, where ProActive should ask for node.
You can get nodes from the P2P Infrastructure using the ProActive Deployment Descriptor as described above.
In fact you will ask for a certain number of nodes and ProActive will notify a "listener" (one of your class), every time a new node is available.
ProActiveDescriptor pad = ProActive.getProactiveDescriptor("myP2PXmlDescriptor.xml");
// getting virtual node "p2pvn" defined in the ProActive Deployement Descriptor
VirtualNode vn = pad.getVirtualNode("p2pvn");
// adding "this" or anyother class has a listener of the "NodeCreationEvent"
((VirtualNodeImpl) vn).addNodeCreationEventListener(this);
//activate that virtual node
vn.activate();
As you can see the class executing this code, must implements an interface in order to be notified when a new node is available from the P2P infrastructure.
Basically you will have to implement the interface NodeCreationEventListener that can be found in package org.objectweb.proactive.core.event For example this method will be called every time a new host is acquired :
public void nodeCreated(NodeCreationEvent event)
{
// get the node
Node newNode = event.getNode();
// now you can create an active object on your node.
}
You should carefully notice that you can be notified at any time, whatever the code you are executing, once you have activated the virtual node.
A short preview of a XML descriptor:
|
A complete example of file is available here.
The next figure shows a P2P Service started with a XML deployment descriptor (xml_path attribute). Six nodes are shared on different hosts:
A P2P Service which is sharing nodes deployed by a descriptor.
For more information about ProActive XML Deployment Descriptor see this page.
The next little sample of code explains how from an application you can start a P2P Service and deploy on it ProActive nodes:
import org.objectweb.proactive.ProActive;
import org.objectweb.proactive.core.ProActiveException;
import org.objectweb.proactive.core.mop.ClassNotReifiableException;
import org.objectweb.proactive.core.node.Node;
import org.objectweb.proactive.core.node.NodeException;
import org.objectweb.proactive.core.node.NodeFactory;
import org.objectweb.proactive.core.runtime.ProActiveRuntime;
import org.objectweb.proactive.core.runtime.RuntimeFactory;
import org.objectweb.proactive.p2p.core.service.P2PService;
import org.objectweb.proactive.p2p.core.service.StartP2PService;
import org.objectweb.proactive.p2p.core.service.node.P2PNodeLookup;
...
// This constructor uses a file with address of peers
// See the Javadoc to choose different parameters
StartP2PService startServiceP2P = new StartP2PService(p2pFile)
// Start the P2P Service on the local host
startServiceP2P.start();
// Get the reference on the P2P Service
P2PService serviceP2P = startServiceP2P.getP2PService();
// By the application's P2P Service ask to the P2P infrastructure
// for getting nodes.
P2PNodeLookup p2pNodeLookup = p2pService.getNodes(nNodes,
virtualNodeName, JobID);
// You can migrate the P2P node lookup from the p2p service
// to an another node:
p2pNodeLookup.moveTo("//localhost/localNode");
// Use method from p2pNodeLookup to get nodes
// such as
while (! p2pNodeLookup.allArrived()) {
Vector arrivedNodes = p2pNodeLookup.getAndRemoveNodes();
// Do something with nodes
...
}
// Your application
...
// End of your program
// Free shared nodes
p2pNodeLookup.killAllNodes();
The goals of this work are to use sparse CPU cycles from institutions' desktop workstations combined with grids and clusters. Consequently desktop workstations are not available all the time for sharing computation times with different users than the workstation owner. Grids and clusters have the same problem that normal users don't want share their usage time.
Managing different sort of resources (grids, clusters, desktop workstations) as a single network of resources with a hight instability of them needs a fully decentralized and dynamic approach.
Therefore, P2P is a good solution for sharing a dynamic JVM network, where JVMs are the shared resources. Thereby, the P2P Infrastructure is a P2P network which shares JVMs for computation. This infrastructure is completely self-organized and fully configurable.
Before going on to consider the P2P infrastructure it's important to define what is Peer-to-Peer.
There are a lot of P2P's definitions, many of them are similar to other distributed infrastructures, such as Grid, client / server, etc. There are 2 better definitions which are really describe P2P:
"[...] P2P is a class of applications that take advantage of resources - available at the edges of the Internet [...]"
"[...] Peers are accessible by other peers directly [...] Any arbitrary chosen peer can be removed from the network without fault [...]"
P2P's focus on sharing, decentralization, instability and fault tolerance.
A fresh (or new) peer which would like join the P2P network, will encounter a serious bootstrapping problem or first contact problem: "How can it connect to the P2P network?"
A solution for that is to use a specific protocol. ProActive provides an interface for a network-centric services protocol which is named JINI. JINI can be used for discovering services in a dynamic computing environment, such as a fresh peer which would like join a P2P network. This protocol is perfectly adapted to solve the bootstrapping problem, however there is a serious drawback for using a protocol as JINI as peer discovering protocol. JINI is limited to work only in the same sub-network that means JINI doesn't pass through firewall or NAT and can't be considered to be used for Internet.
Therefore, a different solution for the bootstrapping problem was chosen, the solution for ProActive first contact P2P is inspired from Data P2P Networks. This solution is based on real life , i.e. when a person wants to join a community, this person has to know first an another person which is already member of the community, after the first person took contact with the community member, the new person is introduce to all community members.
The ProActive P2P bootstrapping protocol works as follow:
Furthermore, int the case of the fresh peer can't contact any servers from the list, the fresh peer will try every TTU (see below, about Time To Update parameter) to re-contact all of them until one or several of them are finally available. At any moments of the peer knows nobody because all of its acquaintances are no more available, the peer will try to contact all servers like earlier.
An example of a fresh peer which is trying to join a P2P network is shown by the next Figure. The new peer has 2 servers to contact for joining the existed P2P infrastructure.
Example of first contact (Bootstrapping).
The main particularity of a P2P network is the peers high volatility. This results from various attributes which compose P2P:
As a result of these points is the instability of the P2P network. Hence the ProActive P2P infrastructure manages automatically and transparency ways this problem.
ProActive P2P infrastructure aims to maintain a created P2P network alive while there are available peers in the network, this is called self-organizing of the P2P network. On condition that P2P doesn't have exterior entities, as such centralized servers which maintain peer data bases, the P2P network has to be self-organized. That means all peers should be enabled to stay in the P2P network by their own means.
There is solution which is widely used in data P2P networks; this consists of for each peers to maintain a list of their neighbors, a peer's neighbors is typically a peers close to it (IP address or geographically).
In the same way, this idea was selected to keep the ProActive P2P infrastructure up. All peers have to maintain a list of acquaintances. At the beginning, when a fresh peer has just joined the P2P infrastructure, it knows only peers from its bootstrapping step. However, as long as is the list of servers, many of them could be not reachable, unavailable, etc. and the fresh peer lands up to know a small number of acquaintances. Knowing a small number of acquaintances is a real problem in a dynamic P2P network, when all servers will be unavailable the fresh peer will be unconnected from the P2P infrastructure.
Therefore, the ProActive P2P infrastructure uses a specific parameter called: Number Of Acquaintances (NOA). This is a minimum size of the list of acquaintances of all peers. More peers are highly dynamic more NOA should be high. Thereby, a peer must to discover new acquaintances through the P2P infrastructure.
In the next section, we will see in detail how the message protocol works, for the moment we will just explain briefly the discovering acquaintances process without go into detail in the message protocol.
The peer called "Alice" has 2 acquaintances resulted from its first contact with the P2P infrastructure and by default NOA is 10 peers. Alice must find at least 8 peers to could stay with a certain guarantee inside the infrastructure.
The acquaintance discovering works as follow:
In order to do not have isolated peers in the infrastructure, all peers registration are symmetric that means if Alice knows the peer "Bob", Bod knows also Alice. Hence, when a peer chooses to be or not an acquaintance, the peer has to check previously in its own acquaintance list if it doesn't already know the asking peer. Next, if it's an unknown peer, the peer decides with a random function to be an acquaintance or not. With the parameter of agree responses, it is possible to configure the percentage of positive response to an exploring message. The random function is a temporally solution to solve the flooding problem dues to the message protocol (see next section), we are thinking to use a new parameter Maximum Number of Acquaintances and improving the message protocol. For the moment, we don't consider peers IP addresses or geographical place of the peers as an acquaintances criteria.
As the P2P infrastructure is a dynamic environment, the list of acquaintances must be also dynamic. Many acquaintances could be unavailable, should be removed and in case of the size of the list is less than the NOA discovering new peers. Therefore, all peers keeps frequently their lists up-to-date that's what a new parameter must be introduce: Time To Update (TTU). This is the frequency which the peer must check its own acquaintances' list to removed unavailable peer and in case of need discovering new peers. To verify the acquaintances availability, the peer send an Heart Beat to all of its acquaintances. The heart beat is sent every TTU.
The next figure shows a peer which is sending an heart beat to all of its acquaintances.
Heart beat sent every TTU.
The main goal of this work is to provide an infrastructure for sharing computational nodes (JVMs). Therefore, a resource query mechanism is needed; there are 2 types of resources in this context thus 2 query types:
The mechanism is similar to the Gnutella's communication system: Breadth-First Search algorithm (BFS). The system is message-based with application-level routing.
All BFS messages must contain these information:
Our BFS inspired version work as follow:
The Gnutella's BFS got a lot of justified critics for scaling, bandwidth using, etc. It is true this protocol is not good enough but we're working to improve it. Thereafter, we'll inquire into solutions with a not fixed TTL to avoid network flooding.
The next Figure shows briefly the execution of the inspired BFS algorithm:
Asking nodes to acquaintances and getting a node.
The P2P infrastructure is implemented with ProActive thus the shared resource is not a JVMs but a ProActive node, nodes are like container which receive work to do that is active objects.
The P2P infrastructure is not directly implemented in the ProActive core at the ProActive runtime level because we choose to be above communication protocols, such as RMI, HTTP, Ibis, etc. Therefore, the P2P infrastructure can use RMI or HTTP as communication layer. Hence, the P2P infrastructure is implemented with classic ProActive active objects and especially with ProActive typed group for broadcasting communications between peers due to your inspired BFS.
Using active objects for the implementation is a good mapping with the idea of a peer which is an independent entities that works as a server with a FIFO request queue. The peer is also a client which sends requests to other peers.
The list of P2P active objects:
The Figure bellow shows connection between all active objects:
Nodes and Active Objects which are composing a P2P Service.
All communications between peers use Group communication but for sending response of a request message, it's a point-to-point communication. Though ProActive communications are asynchronous, it's not really messages which are sent between peers. Nevertheless, it's not a real problem; ProActive is implemented above Java RMI which is RPC and RPC is synchronous, however ProActive uses future mechanism and Rendez-vous method to turn RPC methods in asynchronous, that means ProActive is asynchronous RPC. Rendez-vous is interesting in your case because it guarantees the method is successfully received by the receiver. With the Heart beat message is sent a Java exception can be caught if an acquaintance is down.
The P2PAcquaintanceManager manages the list of acquaintances, this list is represented by a ProActive typed group of P2PService. This is the point of the next section.
ProActive typed group does't allow to access group elements and to make calls from different active objects to the same group, i.e. a group can not be shared. However, the point of the P2P infrastructure is to broadcast message to all member of the acquaintance list, ProActive typed group are perfect to do that. A typed group of P2PService is a well implementation of the acquaintance list design.
But a typed group is not applied from many active objects, especially making group method calls from different objects, adding / removing / etc. members in the group. Therefore, for the P2P infrastructure the P2PAcquaintanceManager (PAM) was made.
The PAM is a standard active object, at its initialization it construct an empty P2PService group. The PAM provides few access group methods, such as removing, adding and group size methods. All other active objects, such as P2PService or P2PNodeLookup, have to use PAM methods to access to the group. The PAM works as a server with a FIFO queue behind the group.
That solve the problem of group member accessing but not how other active object can call methods on the group. The ProActive group API provides a method to turn active a group that is made possible to get ProActive reference on the group. The PAM turns active the group after its creation. P2PService, P2PNodeLookup and all get the group reference by a PAM's getter.
The PAM during is activity send frequently heart beats to remove unavailable peers, the P2PService adds by the PAM new discovered acquaintance (P2PService) and the P2PNodeLookup calls group method to ask nodes by the group reference. The P2PService does also group method calls.
In short all is in the next Figure:
Dynamic Shared ProActive Typed Group.
We just explain how to share a typed group between active objects but that's not solve all problems. For the moment, the BFS implementation with broadcasting to all acquaintances each time is not perfect due to the broad-casted message is always sent back to the previous sender. We are working to add member exclusion in a group method call.
The sharing node mechanism provides an independent activity from the P2P service for all about shared nodes. Nodes are the sharing resource of this P2P network. This activity is handled by the P2PNodeManager active object.
At the initialization of the P2PNodeManager (PNM), it has to instantiate the shared resource. By default, it's 1 ProActive nodes by own machine CPU, for example on a single processor machine the PNM starts 1 node and on a bi-processors machine it starts 2 nodes. It's possible to choose to share only a single node independently of the number of processors. An other way is to share nodes from an XML deployment descriptor file by specifing the descriptor to the PNM which actives deployment and gets nodes ready to share.
When the P2P service receives a node request, the request is forwarded (after the BFS broadcast) to the PNM which is checking for a free node. In the case of at least 1 node is free, the PNM must book the node and send back a reference on the the node to the original request sender. However, the booking remains valid for a predetermined time, this time expires after a configurable timeout. The PNM knows if the node is used or not by testing the active object presence inside the node. Consequently, at the end of the booking time, the PNM kill the node, the node is no more usable. Though, some applications could need empty node for a long time before use them, thereby there is a pseudo expand booking time system: creating "Dummy" active objects in booked nodes for a later use. This system is allowed by the P2PNodeLookup.
The P2PNodeLookup could receive more nodes than it needs, for all additional nodes, the P2PNodeLookup send a message to all node PNM to cancel its book on the node.
The deployed applications have to leave nodes after using. Therefor, the PNM offers a leaving node mechanism that application send a leaving message for a specified node to the PNM which kills all node's active objects by terminate their bodies and kills the node. After that, the PNM creates a new node which is ready for sharing. However, if nodes are deployed by an XML descriptor the PNM does't kill the node, it just terminates all its active objects and re-share the same node.
The asking node mechanism is allowed by the P2PNodeLookup, this object is active by the P2PService when it receives a asking node request from an application. The P2PNodeLookup (PNL) works as a broker, it could be migrate to an another place (node, machine, etc.) to be more closer to the application.
The PNL aims to find trough the P2P infrastructure the application requested number of nodes. It uses the BFS to flood frequently the network until it gets all needed nodes or until the timeout reached. However, the application can ask the maximum number of nodes, in that case the PNL asks nodes until the end of the application with a greater frequency. The PNL provides a listener / producer event mechanism that is great for application which wants to know when a node is found.
Finally, the application kills nodes by the PNL which is in charge to contact all PNMs of each nodes and asks to them to leave nodes. The PNMs leave nodes with the mechanism of the booking timeout.
Lastly, the asking nodes mechanism with the PNL is fully integrated to the ProActive XML deployment descriptor.
A screen shot made with IC2D. You can see principaly 3 P2P services which are sharing 2 nodes (bi-processors machines). Inside nodes some Domain objects from the nBody application which is deployed on this small P2P infrastructure.
nBody application deployed on P2P Infrastructure.