Skip to content. | Skip to navigation

Personal tools
Log in

DIstributed Computing Environments (DICE) Team
We are a group of computer scientists and IT experts from the Department of Computer Science AGH and ACC Cyfronet AGH. We are a curiosity- and research-driven team, specializing in large-scale distributed computing, HPC, Web and Cloud technologies. We develop new methods, tools and environments and we apply these solutions in e-Science, healthcare and industrial domains.


Sections
You are here: Home Downloads GridSpace releases 2.4.0 GridSpace 2.4.0 Tutorial

GridSpace 2.4.0 Tutorial

Welcome to GridSpace2! This document is meant as a tutorial for first-time users, providing basic information which will enable you to begin using the GridSpace platform in your research environment.

GridSpace2 enables users to develop and execute virtual experiments on the underlying computational and storage resources. If you are a scientist involved in computationally-demanding research activities then GridSpace2 may just be the right answer for you! Read on to find out how to begin using GridSpace2 in support of your research.

This document is divided into the following sections:

  1. A (very) brief introduction to GridSpace2
  2. What you will need
  3. Accessing the Experiment Workbench
  4. Your first experiment
  5. Basic Workbench features
  6. Advanced Workbench features

A (very) brief introduction to GridSpace2

The GridSpace2 platform was designed with simplicity of use in mind. We have extensively analyzed the ways in which scientists conduct their day-to-day research activities and tried to come up with a framework which closely mimics this experience while still providing access to powerful distributed computing architectures. Thus, GridSpace2 is founded on the following principles:

  • The platform enables you to write experiments which assume the form of scripts.
  • Scripts can be written using popular scripting languages, including Ruby, Python and Perl, as well as in the Unix shell (bash). No syntax changes are involved - your existing code can run as a GridSpace experiment without any modifications!
  • The platform provides a range of specialized scientific packages which can be invoked within your experiment scripts - simply include the required library and send your data for processing. You can find the full list of the available packages on the the PL-Grid website.
  • GridSpace2 supports exploratory programming: you can divide your experiment into so-called snippets, which are small, self-contained pieces of code. Each snippet can be written in a different programming language and you can freely compose snippets to form complex experiments.
  • Naturally, GridSpace2 enables you to persist and share your experiments, as well as to securely store sensitive data using a handy wallet mechanism.

What you will need

Getting to know a new software platform is always a hurdle; however we've tried to reduce the learning curve for new users and enable them to hop right in and try their own hand at developing experiments using the GridSpace2 Workbench. You will probably be happy to hear that you do not need to install any software on your computer – all you need is the following:

  • A Web browser. Most folks have one already. At the very least, there should be an installation of Internet Explorer available on your machine (if you're running Windows); otherwise we recommend Firefox which you can download from the Mozilla website.
  • An Internet connection. We assume you have one, since you're reading this tutorial. :)
  • A user account on some server - we call experiment host and this is the host where computation of experiment takes place. This can be either head node of the cluster where one can log in via SSH or more or less ordinary single computer

Accessing the Experiment Workbench

Simply point your browser to GridSpace2 Experiment Workbench instance, scroll down to the section labeled This installation of the GridSpace2 supports the following experiment hosts and select the machine on which your user account has been activated. The system will respond by displaying a login screen where you need to input your username and password. Once authenticated, you will be presented with the "Message of a Day" pop-up windows, which shows the custom information for users of the host left here by host administrators. After clicking anywhere outside the pop-up you see basic view of Experiment Workbench, which looks a bit like this:

ew-basic-view.png

Before we get into a discussion on what's important and what is less so, why not try your hand at writing and executing your very first experiment?

Your First Experiment

See the big blank area under the Snippet #1 label of your Workbench? This is where you should input your experiment code. Let's take the Workbench for a spin by writing a simple snippet in bash (the default choice, as indicated by the Bash 3.1 (or similar) label at the top of the snippet window):

echo 'Hello, world!'

Now position your mouse next to the Actions menu and select the image (this orders the Workbench to execute your entire experiment on the host machine and retrieve its results). You should see the following output in the console window:

[Snippet#1 Bash 3.1 29/03/2011 13:50:52]
$ echo 'Hello, world!'
Hello, world!

As you can see, the Workbench has executed your snippet and reported its results. Note that as long as the experiment remains in execution mode, a red square button will be displayed in the console window. This feature does not matter much for single-snippet experiments, but it will come in handy when developing additional snippets on the basis of the results of your initial snippet.

Speaking of multiple-snippet experiments, let's try adding another snippet to our experiment. Click the button next to the Actions menu. This will add another top-level snippet where we again can select an interpreter and enter the code we wish to execute (optionally, you can insert a new snippet before the current one by clicking Actions and selecting ). If you click the interpreter tab (which currently says Bash 3.1), the Workbench will display a list of available interpreters. Let's try selecting Ruby 1.8.6 (or any other version of Ruby) for a change. Once that's done, type in the following Ruby code:

puts 'Hello again!'

You can now execute the entire experiment from scratch by clicking . Alternatively, you might want to rerun only the first snippet in your code - to do this, go back to the first snippet window and click the button placed there. The general rule is thus: clicking in a selected snippet window will execute all snippets up to and including the selected one. In our example, executing your second snippet will output the following to your console:

[Snippet#2 Ruby 1.8.6 03/08/2010 13:56:51]
>> puts 'Hello again!'
Hello again!
=> nil

Since we're dealing with interactive Ruby, each line returns a value which is also echoed to the console (hence the nil value generated by the puts command).

Once your experiment is ready, you may want to save it. Clicking the button enables you to store the experiment in your home directory on your server account. Optionally, you can also specify a name, a description and some comments regarding the contents of your experiment (see here for a more involved description of experiment metadata). To save the experiment you just need to provide a file name and click OK. GridSpace2 serializes experiments in the form of an XML file. If you're curious about the serialization format, consult the final section of this tutorial.

Basic Workbench Features

The Workbench automates common actions associated with developing experiments. Let's take a look at what you may do with the Workbench:

Adding and removing snippets

Each snippet window contains a button labeled Actions where you can activate actions relating to the current snippet. This includes adding another snippet either as a direct successor/predecessor of the current snippet, or as a branch thereof. Note that the whole experiment may have a branching structure, where each snippet plays the role of a node and some snippets are treated as subcomponents of other snippets.

  • To add a new top-level snippet, open the Actions menu and select the button.
  • To add a new top-level snippet as the predecessor of the current snippet, open the Actions menu and select the button.
  • To add a subsnippet, open the Actions menu and select the button.

You can also delete existing snippets by clicking the button in the Actions menu. The button will display a list of useful keyboard shortcuts.

Rerunning your experiment

To run all snippets up to the selected one, click the button in the appropriate snippet window. Snippets are always executed in a top-to-bottom fashion, beginning with the first snippet and ending with the final snippet in the current branch.

Saving your experiment

Once your experiment is ready, you can save it by clicking the Save Experiment icon in the experiment toolbar (). If this is a new experiment, the Workspace enables you to provide some metadata describing the purpose and usage of the experiment. Providing such metadata is optional, though it may enable your collaborators to rerun and extend the experiment with little difficulty (see here for a description of the metadata mechanism).

Saving the experiment serializes it to a file in the current directory on the UI machine, the contents of which are displayed in the left-hand panel in the Workbench. Experiment serialization is XML-based and covers all aspects of the experiment with the exception of any data declared as secret with the use of the wallet mechanism (see next section for a description of how GridSpace2 handles secrets).

Once serialized, the experiment may be reopened in the Workbench window or shared with other users of the platform. In order to open an existing experiment in the Workbench simply click the name of the experiment file in the File Management panel.

Naturally, you can also close the current experiment by clicking the x icon in its tab above the snippet window. If you wish to start a new experiment, click the + button adjacent to open experiment tabs and select New experiment (the New console option can be used to start an interactive console session instead).

Advanced Workbench Features

In addition to basic experiment operations, the Workbench also provides some advanced features which may prove helpful. You may want to make use of the following tools:

File management

The Workbench enables you to manage the contents of your UI home directory. This directory can be used to store experiments as well as any arbitrary files you may deem useful while working with GridSpace2. A sample view of the GridSpace2 directory browser is presented below.

The following operations are available in the browser:

  • Return to your home directory:
  • Navigate to parent directory:
  • Refresh directory contents:
  • Create a new file:
  • Create a new directory:
  • Toggle display of hidden files:
  • Upload files from your PC:

Moreover, a pull-down operations menu is available for each listed file. You can access it by clicking the small arrow next to the selected filename. This menu contains the following options:

  • Experiment: (Only available for experiment files) - Brings up another menu which enables you to open, run or view the experiment's contents in a separate window.
  • Open with: Brings up another menu which enables you to open the selected file using a specific application. The list of available applications corresponds to the file extension: GridSpace2 provides a selection of customized openers for popular file formats (such as Jmol and JQplot).
  • Make accessible to: Shares the experiment with users belonging to a selected group (Note: functionality not yet implemented – bear with us :).)
  • Use path in snippet: Pastes the file path in the currently active snippet window.
  • Use contents in snippet: Pastes the file contents in the currently active snippet window.
  • Show link location: Obtains a qualified path to the selected file.
  • Copy: Copies the contents of the selected file to a new file with a name which you can specify.
  • Rename: Changes the name of the selected file.
  • Delete: Deletes the selected file.

Secrets and credentials

Some experiments may require you to provide data which you would rather keep secret and protect against unauthorized access - for instance database passwords, personal information or security tokens required to access certain external resources. We will refer to such data as secrets.

The Experiment Workbench enables you to use secrets in your experiments and to store them in a secure way in a personal wallet. You can take advantage of either a local wallet, the contents of which are stored in a separate file in your home directory, or a centralized GridSpace2 wallet mechanism backed by a Remote Central Wallet service which can be accessed from any location (given proper security credentials). Moreover, we support storage of string-type data (basically text fragments) or entire files which are made available to your experiments on demand.

In order to include a secret in your experiment, you need to:

  • Upload your secret to one of the two available wallets and assign a name to it;
  • Tell the Workbench that your experiment requires a secret with a given name.

In fact, the latter step is optional unless you intend to share the experiment with other developers who may not have access to your personal secrets (note that secrets registered with the GridSpace2 wallet mechanism are never shared with external users, even if you save your experiment and then distribute the resulting file).

Let's start by uploading a secret to your user wallet. Clicking the button next to your login name (in the top right-hand corner of the screen) will bring up a window where you can manage your secrets. Each secret is described by the following properties:

  • Name: in the context of your experiments, the secret will be available as an environmental variable with the specified name.
  • Payload: this is the actual string representing your secret data.
  • Storage method (each secret can be stored either in the local wallet or the Remote Central Wallet).
  • Optional description.

To add a new secret, click the Add a new secret button in the manager window. You can store either a string (line of text) or an entire file (which you select from the local filesystem by pressing Browse...). Once a secret is added, it can be referenced in your experiments by its name (see below).

  • The wallet icon represents the type of storage in effect for a given secret
    • means local wallet
    • means Remote Central Wallet
  • The button displays the current value of the secret.
  • The button allows you to modify the secret value.

If you wish to remove a secret from your wallet, click the button next to its name.

Having uploaded the necessary secrets to your wallet, you can use them in snippets by adding the formula <SECRET:STRING:secret_name> or <SECRET:FILE:secret_name>, where secret_name is the name of the secret in the wallet. Requesting the secret as a STRING will simply paste the payload of the secret in your code, whereas requesting it as a FILE will instantiate a file with the specified name in your local secrets directory (~/.gs2/.local_secret_directory or ~/.gs2/.remote_secrets_directory, depending on whether the secret is string- or file-based). In this case, the above invocation will be replaced by a qualified path to the secret file when your snippet is executed. For instance, if the value of secret_name was SECRET, the following code in bash:

echo 'This is my <SECRET:STRING:secret_name>'

will yield the following result:

[Snippet#1 Bash 3.1 24/08/2010 14:06:32]
$ echo 'This is my SECRET'
This is my SECRET

However, if you would rather spawn a file with this secret's payload as content, you would need to type:

echo 'My secret is in <SECRET:FILE:secret_name>'

will yield the following result:

[Snippet#1 Bash 3.1 24/08/2010 14:06:32]
$ echo 'My secret is in /home/myuser/.gs2/.local_secret_directory/secret_name'
My secret is in /home/myuser/.gs2/.local_secret_directory/secret_name

Instead of typing <SECRET:STRING:secret_name> (or <SECRET:FILE:secret_name>) directly in the snippet code, you can select a secret to be used in the snippet from the list available after clicking the icon in the snippet header. The list contains all the secrets stored in your wallet (either local or remote) and the secret already declared for the current experiment (appearing in this or other snippets contained in the current experiment). A new secret can be added through the form at the bottom of the panel. It will be present in the experiment scope, and the user will be prompted for its value when the experiment starts. If you now serialize and share the experiment with other developers, your secrets will not be automatically shared. Instead, the recipients of your experiment will require a secret with an identical name to be present in their wallets before they are able to run your experiment.

Sharing experiments

GridSpace2 enables you to pack up your experiment and share it with other members of your team. To share an experiment, all you need to do is save it in your account – this will generate an experiment file which you can then download and send to any of your collaborators. Upon receiving an experiment file from you, the recipient may upload it to his/her Workbench (by using the file management tools described here, then open the experiment and execute it.

Note that any secrets which are part of your experiment are not saved along with the experiment file. Rather, GridSpace2 expects the recipient to have access to secrets with identical names inside his/her Workbench instance.

WebGUI mechanism

With the WebGUI mechanism you can interweave your experiments with rich web user interfaces. This feature can be used to fetch extra data from the user while presenting intermediate results of an experiment run which is blocked until the user submits requested information. Another use case that can be implemented with this mechanism is fine-grained flow control of the experiment. In this case, at each step of an experiment the user may be presented with a question (with additional data rendered, if necessary). By answering these questions, the user can guide experiment execution.

In order to use the WebGUI mechanism, a set of required libraries needs to be downloaded to your account. This step can be accomplished with standard Workbench tools. Use a bash snippet to execute the following code (compliant with ui.grid.cyfronet.pl):

if [ ! -e _webguimodule.so ]
then
 wget http://dev-gs.cyfronet.pl/~daniel/webgui/_webguimodule.so
fi

if [ ! -e webgui.py ]
then
 wget http://dev-gs.cyfronet.pl/~daniel/webgui/webgui.py
fi

Let us now look at a simple example in which we will request a title and a rich-text block from the user, The following Python experiment accomplishes this goal.

import webgui
import json

result = webgui.process_query("http://localhost/rest/webgui/render",
"{webguiDocType: 'request',
label: 'Sample Input',
data: [{name: 'title', label: 'Title', pref: 'text'},
{name: 'text', label: 'Sample text', pref: 'richTextArea'}]}")

jsonResponse = json.loads(result)
title = jsonResponse['data']['title']
text = jsonResponse['data']['text']
print "Title: %s, text: %s" % (title, text)

The first two lines are Python imports which enable the WebGUI module and JSON processing library. Next, we use the WebGUI mechanism by calling the process_query method with the external web application address (in this case, this is the basic address of your Workbench instance with the /webgui path added), and the JSON definition of the web application. When called, the experiment script blocks and the user is presented with a web interface. Data provided by the user is assigned to the result variable. The last four lines extract the response (also in JSON format) and print a suitable output message.

Web interfaces used by WebGUI are simple external web applications that integrate with experiments by using JSON objects. It is therefore possible to implement your own graphical user interface and use it within your experiment. For reference, a sample web application is shipped with the Experiment Workbench. It enables composing a simple application by providing a JSON description according to the format described in the next paragraph. For your own web applications, you may need to create you own communication protocol, keeping in mind that user interaction proceeds in a typical request/response fashion (similar to the example described above). The web application has to accept POST requests with the following request parameters:

  • id - unique identifier of the current request/response exchange (generated) by the WebGUI library
  • json_request - JSON request document
  • respond_to - address which should be used by the web application to send back user data
  • gs2SessionId - cookie value (with the name=value form) to be passed back to the EW server in a form of a request cookie

Response data should be returned to the WebGUI application by sending a POST request to the given respond_to address with the following parameters:

  • id - identifier of the current dialog (sent to the web application in the previous POST request)
  • json_response - JSON response with user/application data

Additionally a cookie with the session id should be sent. The cookie's value can be obtained by reading the gs2SessionId parameter from POST request.

The out-of-the-box WebGUI user interface lets you construct a simple form-like web interfaces by defining a set of form widgets which are stacked vertically on the final web page. You can choose from the following widgets (defined in the pref element of the JSON definition):

  • label - read-only text block; displayed text is defined by the value element
  • text - one-line text input
  • textArea - multiple-line text input
  • richTextArea - multiple-line rich text input
  • radio - set of radio buttons
  • check - set of check boxes
  • select - drop-down selection list

If applicable, a default value for a given input can be defined by using the value (for label, text, textArea, richTextArea and radio inputs) or values (for check and select inputs) elements. Each input is decorated by a label which can be defined by using the label element. For inputs with multiple selection values options element should be used (see below).

Some examples

Let us assume that the developer wants to use the out-of-the-box WebGUI web application to request user gender (via radio inputs, with the female option checked by default), age (via single-line text input) and CV (via rich text input). The following JSON definition would be valid:

{webguiDocType: 'request',\
label: 'User data',\
data: [{name: 'gender', label: 'Gender', pref: 'radio', options:[\
{label: 'Male', value: 'male'}, {label: 'Female', value: 'female'}], value: 'female'},\
{name: 'age', label: 'Age', pref: 'text'},\
{name: 'cv', label: 'CV', pref: 'richTextArea'}]}

Depending on the provided data the following response will be assigned as the return value of the process_query method:

{"webguiDocType":"response", "data":{"gender":"female", "age":"25", "cv":"short cv"}}

In custom web application, the response/request message bodies need to be composed by the experiment creator. It is, however, recommended to use the JSON notation for all messages sent to and received from the uer.

Experiment metadata

The button placed at the top of the experiment window brings up a menu which enables you to manipulate experiment metadata and define its properties. Experiment properties consist of the following items:

  • Experiment creator and creation date
  • Experiment name
  • Experiment description
  • User manual
  • Comments

Most of these properties can be manipulated by the Workbench – a sample properties window is presented below.

GridSpace2 provides a handy tool with which to prepare experiment manuals (if needed)- clicking Add manual displays another window where you can type your experiment manual with the use of basic formatting tools. Once you're satisfied with the properties you have defined for your experiment, click OK. These properties will be serialized along with the experiment and stored in the experiment file.

Management of interpreter gems

A "bare-bones" interpreter may often be insufficient to run a given script – developers frequently make use of external modules and libraries, provided by their own collaborators or third-party developers. In GridSpace, such modules are called gems (akin to the Ruby concept of a scripting gem). Gems can be imported and installed, whereupon they become available to experiment developers and can be invoked in GridSpace experiments – provided you're using the correct interpreter.

To install a new gem, you need to know its name. Go to the Gem management panel (click Gems management below the file management panel), select the tab which corresponds to the scripting language you wish to use, and type in the name of your desired gem (or part thereof). GridSpace will automatically scan public repositories and present a list of available gems. For each gem, the following options are provided:

  • Display a description of the selected gem
  • Install the selected gem (+)
  • Uninstall the selected gem, if already installed (-)

Back to the top

NOTE! This web site uses cookies and similar technologies (if you do not change the browser settings, you agree to this).

cyfronet agh