Scalable and effective clustering, scheduling and monitoring of self-organizing grids

Self-Organizing Grid is a large-scale dynamic Grid, with low participating hurdle and human administrative costs. In the SOG model, participating resources self organize to provide effective support for applications that are submitted against them. Compared to traditional Grids, SOGs are more scalable and manageable, and easier to exploit idle resources available through the Internet. Our research tries to solve several challenges of SOGs, they are: (1) How to dynamically extract a set of resources to match the request according to different criteria. (2) How to efficiently monitor resources and schedule jobs in a distributed Grid environment. (3) Autonomous resource authorization with common rules and special rules. We propose a scalable solution that adopts a delay sensitive overlay structure, MDTree, to organize nodes based on their proximity to one another, with only a small number of delay experiments. This overlay provides a variable-size set of promising candidate nodes that can then be used as a cluster, or tested further to improve the selection. We design a structure for efficient resource discovery and monitoring. Resource information is stored distributedly on nodes and aggregated hierarchically. Queries are forwarded in a way that excluding nodes that do not satisfy criteria. Based on the MDTree overlay and resource aggregation, we propose a Group-Scheduling framework for SOG to allow scalable and effective scheduling of parallel jobs, each of which requires multiple resources, to available resources that can support each job most efficiently. We also propose a distributed authorization architecture for SOG environment. The major contributions of this research are: (1) Variable size automatic node clustering based on link delay. (2) Distributed hierarchical resource management with efficient updating and range query capability. (3) Efficient group-scheduling for parallel applications with inter-dependent tasks with communication costs taken into consideration. Less important contributions include: (1) Distributed resource authorization for SOGs. (2) A new simulation framework capable of accurate modeling and efficient simulation for P2P and SOG environment.