Book chapter
YT – эволюция системы распределенных вычислений
In book
The paper studies the performance of algorithms for scheduling of many-task applications in distributed computing systems. Two important classes of such applications are considered: bags-of-tasks and workflows. The comparison of algorithms is performed on the basis of discrete-event simulation for various application cases and system configurations. The developed simulation framework based on SimGrid toolkit provides the necessary tools for implementation of scheduling algorithms, generation of synthetic systems and applications, execution of simulation experiments and analysis of results. This allowed to perform a large number of experiments in a reasonable amount of time and to ensure reproducible results. The conducted experiments demonstrate the dependence of the performance of studied algorithms on various application and system characteristics. While confirming the performance advantage of advanced static algorithms, the presented results reveal some interesting insights. In particular, the accuracy of the used network model helped to demonstrate the limitations of simple analytical models for scheduling of data-intensive parallel applications with static algorithms.
The paper presents an approach to the design and implementation of web-based environments for practical exercises in parallel and distributed computing (PDC). The presented approach introduces minimal development and operational costs by relying on Everest, a general-purpose platform for building computational web services. The flexibility of proposed service-oriented architecture enables the development of different types of services targeting various use cases and PDC topics. The generic execution services support the execution of different types of parallel and distributed programs on corresponding computing systems, while the assignment evaluation services implement the execution and evaluation of solutions to programming assignments. As was demonstrated by teaching two introductory PDC courses, the presented approach helps to enhance students’ practical experience while avoiding low-level interfaces, reducing the grading time and providing a level of automation necessary for scaling the course to a large number of students. In contrast to other efforts, the exploited Platform as a Service model provides the ability to quickly reuse this approach by other PDC educators without installation of the Everest platform.
This chapter describes an economic model for independent job flow management in distributed computing environments with non-dedicated resources. The model is based on the concept of fair resource distribution between users and owners of computational nodes by means of economic mechanisms in a virtual organization. Scheduling is performed in cycles in accordance with dynamically updated schedules on local processor nodes. Schedule optimization is performed using dynamic programming methods using the set of criteria in accordance with the economic policy of the virtual organization.
Mobile (cellular) networks represent a fast evolving research field that take advantages of recent technological advances such as Big Data and distributed computing to provide extensive network monitoring for network operation planning and management purposes. Challenges related to making use of the large volume of streaming data generated by mobile networks include extracting relevant elements within massive amounts of signals possibly spread across different sources (data bases), reducing dimensionality, summarizing dynamic information in a comprehensible way and displaying it for interpretation purposes. The adequate network modeling provides both statistical data of network performance and important networking QoS related insights. Comprehensible mobile network information is needed that uncovers the role of each attribute and variable. To harness the complexity of mobile network data and to extract relevant information a dedicated distributed computing platforms and Big Data frameworks are needed, able to discover and deal with the inherent properties and complexities of these datasets. Smart network monitoring service plays a central role here.
Large-scale classification of text streams is an essential problem that is hard to solve. Batch processing systems are scalable and proved their effectiveness for machine learning but do not provide low latency. On the other hand, state-of-the-art distributed stream processing systems are able to achieve low latency but do not support the same level of fault tolerance and determinism. In this work, we discuss how the distributed streaming computational model and fault tolerance mechanisms can affect the correctness of text classification data flow. We also propose solutions that can mitigate the revealed pitfalls.
This book constitutes the proceedings of the 12th International Conference on Parallel Computing Technologies, PaCT 2013, held in St. Petersburg, Russia, during September 30-October 4, 2013. The 41 full papers presented together with 2 invited papers were carefully reviewed and selected from 83 submissions. The papers are organized in topical sections on all technological aspects of the applications of parallel computer systems High level parallel programming languages and systems, methods and tools for parallel solution of large-scale problems, languages, environments and software tools supporting parallel processing, operating systems, scheduling, mapping, load balancing, general architectural concepts, cellular automata, performance measurement and analysis tools, teaching parallel processing, software for grid and cloud computing, scalable computing, fragmentation and aggregation of algorithms and programs as well as programs assembling and reuse.
Distributed computing systems are widely used for execution of loosely coupled many-task applications. There are two important classes of such applications. Bag-of-tasks applications, eg, parameter sweeps or Monte Carlo simulations, represent a set of independent tasks. Workflows, which are used for automation of complex computational and data processing pipelines, consist of multiple tasks with control or data dependencies. The paper discusses the common problems related to the efficient execution of such applications on distributed computing resources and the relevant solutions implemented within the Everest platform.