Skip to content. | Skip to navigation

Personal tools
Log in

DIstributed Computing Environments (DICE) Team
We are a group of computer scientists and IT experts from the Department of Computer Science AGH and ACC Cyfronet AGH. We are a curiosity- and research-driven team, specializing in large-scale distributed computing, HPC, Web and Cloud technologies. We develop new methods, tools and environments and we apply these solutions in e-Science, healthcare and industrial domains.


Sections
You are here: Home DICE Blog RESTful (r)evolution for science gateways using HPC resources

RESTful (r)evolution for science gateways using HPC resources

Posted by Robert Pajak at Jul 02, 2015 12:21 PM |
This blog entry presents a service which allows to access HPC (high performance computing) resources via a REST interface taking care of all the underlying software stacks (job management using various middleware libraries and file transfers). As a result integration with the resources is plain and simple by using dominant web protocols with platform independence as a first-class citizen. The entry assumes a reader has basic knowledge of Grid job management specifics.

In the era of web dominance where information is exchanged using lightweight protocols and data formats building HPC-based portals is lacking in its agility. HPC integration in a web world always seems like series of compromises. It is difficult to stay independent of the underlying technology (middleware) and use all the goodies of various web frameworks available today. Usually, it is the simple case where running a single job on a cluster of choice is perfectly sufficient to fulfill the HPC demands of an application. However, accessing a local cluster's queuing system directly is seldom possible. Also, in many cases still the preferred way of accessing such systems is through the SSH session. On the other hand using Grid-like software which spans an overlay over many HPC resources tends to enforce a certain technology stack to be used. With many scientific domains it is also a tendency to reinvent the wheel over and over again by implementing wrappers on libraries giving access to HPC resources.

Being part of a larger team of people who were dealing with the above problems for many years allowed me to participate in preparation of a service which covers all the cumbersome aspects of HPC integration and exposes a fresh and clean REST interface allowing science gateways developers to make the work necessary to use HPC a breeze.

rimrock-logoThe work resulted in creating a service called rimrock which simplifies the management of processes and tasks executed on high performance computing infrastructures. The service stands out by unifying access to computing and storage resources with the popular REST interface communication paradigm and by utilizing user rights delegation making it possible to chain multiple, intermediary services within a single user session. The service was deployed and is actively used within the PLGrid computing infrastructure. It now allows for local job submission as well as Grid management exposing the whole infrastructure comprising five top computing centers in Poland over a single REST interface. Rimrock apart from the standard job management facilities enables interactive work with the infrastructure through various interpreters to making up for more interactive use cases.

In the picture below a simplified architecture of rimrock is presented. It shows that both job management and file handling is covered by the REST interface and none of the underlying middleware components are exposed to the client. As depicted rimrock is intended to be used by researchers and developers using HPC infrastructures, especially by teams designing and implementing domain-specific science gateways but also by individual users who take care of maintaining their own computation scripts.

rimrock-concept-en

Going further with the development of rimrock will allow for immediate responses from job submission requests which now take up to a couple of seconds because of the underlying middleware communication. Also, building pure JavaScript applications contacting the service directly from a user's browser will be possible thanks to integrating the CORS mechanism. This feature will expand even more the landscape of available web frameworks no more limited to those requiring server side. The future HPC-based web development looks brighter with rimrock on the horizon :)

Comments (0)

NOTE! This web site uses cookies and similar technologies (if you do not change the browser settings, you agree to this).

cyfronet agh