Skip to content. | Skip to navigation

Personal tools
Log in

DIstributed Computing Environments (DICE) Team
We are a group of computer scientists and IT experts from the Department of Computer Science AGH and ACC Cyfronet AGH. We are a curiosity- and research-driven team, specializing in large-scale distributed computing, HPC, Web and Cloud technologies. We develop new methods, tools and environments and we apply these solutions in e-Science, healthcare and industrial domains.


Sections
You are here: Home Downloads GridSpace releases 2.5.0 GridSpace 2.5.0 Tutorial

GridSpace 2.5.0 Tutorial

Welcome to GridSpace2! This document is meant as a tutorial for first-time users, providing basic information which will enable you to begin using the GridSpace platform in your research environment.

GridSpace2 enables users to develop and execute virtual experiments on the underlying computational and storage resources. If you are a scientist involved in computationally-demanding research activities then GridSpace2 may just be the right answer for you! Read on to find out how to begin using GridSpace2 in support of your research.

This document is divided into the following sections:

  1. A (very) brief introduction to GridSpace2
  2. What you will need
  3. Accessing the Experiment Workbench
  4. Your first experiment
  5. Basic Workbench features
  6. Advanced Workbench features

A (very) brief introduction to GridSpace2

The GridSpace2 platform was designed with simplicity of use in mind. We have extensively analyzed the ways in which scientists conduct their day-to-day research activities and tried to come up with a framework which closely mimics this experience while still providing access to powerful distributed computing architectures. Thus, GridSpace2 is founded on the following principles:

  • The platform enables you to write experiments which assume the form of scripts.
  • Scripts can be written using popular scripting languages, including Ruby, Python and Perl, as well as in the Unix shell (bash). No syntax changes are involved - your existing code can run as a GridSpace experiment without any modifications!
  • The platform provides a range of specialized scientific packages which can be invoked within your experiment scripts - simply include the required library and send your data for processing. You can find the full list of the available packages on the the PL-Grid website.
  • GridSpace2 supports exploratory programming: you can divide your experiment into so-called snippets, which are small, self-contained pieces of code. Each snippet can be written in a different programming language and you can freely compose snippets to form complex experiments.
  • Naturally, GridSpace2 enables you to persist and share your experiments, as well as to securely store sensitive data using a handy wallet mechanism.

What you will need

Getting to know a new software platform is always a hurdle; however we've tried to reduce the learning curve for new users and enable them to hop right in and try their own hand at developing experiments using the GridSpace2 Workbench. You will probably be happy to hear that you do not need to install any software on your computer – all you need is the following:

  • A Web browser. Most folks have one already. At the very least, there should be an installation of Internet Explorer available on your machine (if you're running Windows); otherwise we recommend Firefox which you can download from the Mozilla website.
  • An Internet connection. We assume you have one, since you're reading this tutorial. :)
  • A user account on some server - we call this the executor and it is where computation of experiments takes place. The executor can be either the head node of a computing cluster with SSH access, a standalone external server or even an ordinary personal computer.

Accessing the Experiment Workbench

Simply point your browser to GridSpace2 Experiment Workbench instance, scroll down to the section labeled Login and select the machine on which your user account has been created. The system will respond by displaying a login screen where you need to input your username and password. Once authenticated, you will be presented with a "Message of a Day" pop-up window, which presents custom information for users, provided by executor admin. After clicking anywhere outside the pop-up, you will see the basic view of the Experiment Workbench, which looks a bit like this:

2.5.0workbench1.png

Before we get into a discussion on what's important and what is less so, why not try your hand at writing and executing your very first experiment?

Your First Experiment

See the big blank area under the Snippet ID=a1 label of your Workbench? This is where you should input your experiment code. Let's take the Workbench for a spin by writing a simple snippet in bash. The list of available interpreters will depend on the executor. In our case, we need to click the name of the interpreter (which is currently ADF) and select Bash X from the pull-down menu, as shown below:

2.5.0interpreter_menu1.png

As you can see, GridSpace offers a plethora of interpreters, both general-purpose and application-specific. For now we will resort to writing a shell snippet; thus we select Bash X from the list and can now type in our code in the snippet window:

echo 'Hello, world!'

Now position your mouse next to the Actions menu and select the icon_play_all.png image (this orders the Workbench to execute the entire experiment on the executor and retrieve its results). You should see the following output in the console window:

$ echo 'Hello, world!'
Hello, world!

As you can see, the Workbench has executed your snippet and reported its results. Note that as long as the experiment remains in execution mode, a red square button will be displayed in the console window. This feature does not matter much for single-snippet experiments, but it will come in handy when developing additional snippets on the basis of the results of your initial snippet.

Speaking of multiple-snippet experiments, let's try adding another snippet to our experiment. Click the button in the snippet control panel. This will add another top-level snippet where we again can select an interpreter and enter the code we wish to execute (optionally, you can insert a new snippet before the current one by clicking Actions and selecting ). If you click the interpreter tab (which should currently says Bash X), the Workbench will display a list of available interpreters. Let's try selecting Ruby 1.8.7 (or any other version of Ruby) for a change. Once that's done, type in the following Ruby code:

puts 'Hello again!'

You can now execute the entire experiment from scratch by clicking icon_play_all.png. Alternatively, you might want to rerun only the first snippet in your code - to do this, go back to the first snippet window and click the icon_play.png button placed there. The general rule is thus: clicking icon_play.png in a selected snippet window will execute all snippets up to and including the selected one. In our example, executing your second snippet will output the following to your console:

puts 'Hello again!'
Hello again!

Once your experiment is ready, you may want to save it. Clicking the button enables you to store the experiment in your home directory on your host account. Optionally, you can also specify a name, a description and some comments regarding the contents of your experiment by clicking the icon_info3.png button (see here for a more involved description of experiment metadata). To save the experiment you just need to provide a file name and click OK. GridSpace2 serializes experiments in the form of an XML file. If you're curious about the serialization format, consult the final section of this tutorial.

Basic Workbench Features

The Workbench automates common actions associated with developing experiments. Let's take a look at what you may do with the Workbench:

Adding and removing snippets

Each snippet window contains a button labeled Actions where you can activate actions relating to the current snippet. This includes adding another snippet either as a direct successor/predecessor of the current snippet, or as a branch thereof. Note that the whole experiment may have a branching structure, where each snippet plays the role of a node and some snippets are treated as subcomponents of other snippets.

  • To add a new top-level snippet, open the Actions menu and select the button.
  • To add a new top-level snippet as the predecessor of the current snippet, open the Actions menu and select the button.
  • To add a subsnippet, open the Actions menu and select the button.
  • To merge two snippets into one, open the Actions menu and select the icon_merge.png button.
  • To set a snippet secret, open the Actions menu and select the icon_lock.png button. You can read more about secrets below.

You can also delete existing snippets by clicking the button in the Actions menu. The button will display a list of useful keyboard shortcuts.

Rerunning your experiment

To run all snippets up to the selected one, click the icon_play.png button in the appropriate snippet window. Snippets are always executed in a top-to-bottom fashion, beginning with the first snippet and ending with the final snippet in the current branch.

Saving your experiment

Once your experiment is ready, you can save it by clicking the Save Experiment icon in the experiment toolbar (). If this is a new experiment, it may also be useful to provide some metadata describing its purpose and usage by clicking icon_info3.png. Providing such metadata is optional, though it may enable your collaborators to rerun and extend the experiment with little difficulty (see here for a description of the metadata mechanism).

Saving the experiment serializes it to a file in the current directory on the UI machine, the contents of which are displayed in the left-hand panel in the Workbench. Experiment serialization is XML-based and covers all aspects of the experiment with the exception of any data declared as secret with the use of the wallet mechanism (see next section for a description of how GridSpace2 handles secrets).

Once serialized, the experiment may be reopened in the Workbench window or shared with other users of the platform. In order to open an existing experiment in the Workbench simply click the name of the experiment file in the File Management panel.

Naturally, you can also close the current experiment by clicking the x icon in its tab above the snippet window. If you wish to start a new experiment, click the + button adjacent to open experiment tabs.

Advanced Workbench Features

In addition to basic experiment operations, the Workbench also provides some advanced features which may prove helpful. You may want to make use of the following tools:

File management

The Workbench enables you to manage the contents of your UI home directory. This directory can be used to store experiments as well as any arbitrary files you may deem useful while working with GridSpace2. A sample view of the GridSpace2 directory browser is presented below.

2.5.0file_mgmt1.png

The following operations are available in the browser:

  • Return to your home directory: icon_home_disabled.png
  • Navigate to parent directory: icon_up_disabled.png
  • Refresh directory contents:

The Menu link provides additional operations, including the ability to create new files, subdirectories and experiments. Moreover, a pull-down operations menu is available for each listed item. You can access it by clicking the small arrow next to the selected filename. This menu contains the following options:

  • Experiment: (Only available for experiment files) - Brings up another menu which enables you to open, run or view the experiment's contents in a separate window.
  • Open with: Brings up another menu which enables you to open the selected file using a specific application. The list of available applications corresponds to the file extension: GridSpace2 provides a selection of customized openers for popular file formats (such as Jmol and JQplot).
  • Make accessible to: Shares the experiment with users belonging to a selected group (Note: feature not yet implemented – bear with us :).)
  • Use path in snippet: Pastes the file path in the currently active snippet window.
  • Use contents in snippet: Pastes the file contents in the currently active snippet window.
  • Show link location: Obtains a qualified path to the selected file.
  • Copy: Copies the contents of the selected file to a new file with a name which you can specify.
  • Rename: Changes the name of the selected file.
  • Delete: Deletes the selected file.

Publishing experiments with Collage

GridSpace2 comes integrated with a sophisticated mechanism for publishing the contents of your experiments. Create real live papers by embedding your results and code directly within the scientific publication and enabling readers to interact with it! It's easy - here's how:

  • For each snippet a list of input and output assets can be declared. Each asset is a file that is either required for your snippet to do its work, or produced by the snippet as a result of computations. The snippet itself is also an asset (more on this later).
  • Each asset can be uniquely identified by an URI which is embeddable in any type of Web-based publication: you can link assets in blogs, wikis, electronic documents; you name it.
  • Once exposed, the users can reenact your experiment with input data of their own to see how the experiment processes various types of input files.
  • Access to interactive content can be password-protected and limited to those users who possess accounts on a given Experiment Hosts.

To show you how it works, let's start with a very simple bash experiment which produces an output file. The experiment consists of a single snippet:

echo 'Hello, world!' > output_file.txt

As is easy to predict, executing this experiment will produce a file called output_file.txt in your current working directory. This file is then the result of your experiment. Let's say we want to publish it as an asset. In order to do so, we click the menu bar at the bottom of the snippet window, where it currently says No outputs defined for this snippet. This will bring up a menu from which we select Add new simple output. The Workbench will grace us with the following window where we can define the properties of our new output asset:

2.5.0output_asset1.png

The name and description are arbitrary elements of the asset's metadata record and can be set to whatever values you feel best describe the asset in question. Clicking the icon_pencil2.png icon next to the URL field will bring up the list of executors you're currently logged into and enable you to select a single file from any available executor as the asset payload. This will also automatically populate the file name field. The sendable option enables your copy of the output file to be viewed by external clients (otherwise any user who reenacts your experiment will obtain a separate copy of its output assets.)

Once you're satisfied with the contents of our experiment and have defined the relevant input and output asset, the experiment can be released. Releasing an experiment enables other user to interact with its contents. To do so, open the drop-down menu next to the icon_save.png button in the top right-hand corner of the Workbench and select the release scope (you can release the experiment to members of specific groups on the Executor host, or to everyone). Selecting one of the options will save the experiment in its current state and enable its publication through embedding asset URLs in external HTML documents. The Workbench also creates a "sample" view of your publication, with all assets embedded (remember to enable popups for the Worbench site in your browser!). Your sample view should look a bit like this:

2.5.0-sample-publication1.png

The goal of this page is to showcase how your assets will look when embedded in a real document. Following some basic information about the experiment, the Workbench displays a numbered list of assets, headed by the so-called Master Asset, which is used to authenticate users and review the status of all payload (input/output/snippet) assets. Since you are already logged into the Workbench, the Master Asset will not require you to log in again - instead, it will simply display your username and the Executor you're using. The horizontal bars below your username correspond to experiment assets - a green bar indicates that a given asset has successfully been loaded into your document window. You can also click the selected bar to reload a given asset.

The View embed code button displays a HTML code fragment which you need to paste into your external publication in order to embed a given asset. Assets can be embedded into any HTML document, just by pasting in their corresponding embed code fragments - thus, Collage can be used to easily enrich HTML documents, wikis, blogs, fora etc. with interactive content!

icon_excl_red.png Important note: All external documents to which Collage content is to be added must embed the Master Asset in order to enable external users to log into the Executor and display the payload of other assets. Unauthenticated users will not be able to view interactive content.

You will notice that each asset type (input, output, snippet) comes with its own set of controls displayed at the bottom of the asset frame. These buttons enable users to interact with Collage content by uploading files, rerunning code and collecting output:

  • icon_upload_asset.png This icon enables you to upload a file from your local computer to serve as the input for computations. This option is only available for input assets.
  • icon_download.png This icon enables you to download the file which currently serves as the payload of the given asset. This option is available for input and output assets.
  • icon_new_tab.png This icon will open a new browser tab and visualize the content of the given asset using inbuilt browser plugins. Most general-purpose file formats (including graphics and animations) can be visualized in this way. This option is available for input and output assets.
  • icon_refresh.png This icon will toggle between the original version of the given asset and your own version (if you have introduced any changes, e.g. by uploading new input files). This option is available for input and output assets.
  • icon_play.png This icon will execute the code currently displayed in the asset window, using whatever input data has been provided as input assets. This option is available for snippet assets.

Additionally, snippet assets display an Output tab above the main text area - clicking this tab will visualize any standard output that may have been generated by the snippet during execution.

Taken together, the three types of assets (input, output and snippet) facilitate broad interaction with the published experiment, enabling users to review and reenact your code, as well as perform computations on their own datasets. Results can be visualized directly in the browser window or retrieved as files.

icon_excl_black.png Note: When interacting with Collage, each user will receive their own temporary copies of all input, output and snippet assets. Modifying any of these does not impact the original contents of the published experiment. Once the user ends his/her session (e.g. by closing the browser window) these temporary copies are purged. Interaction with assets is therefore transparent from the point of view of other users and does not carry the risk of corrupting the original experiment.

Experiment metadata

The icon_info3.png button placed at the top of the experiment window brings up a menu which enables you to manipulate experiment metadata and define its properties. Experiment properties consist of the following items:

  • Experiment creator and creation date
  • Experiment name
  • Experiment description
  • Terms of use
  • User manual
  • Comments

Most of these properties can be manipulated by the Workbench – a sample properties window is presented below.

2.5.0metadata1.png

GridSpace2 provides a handy tool with which to prepare experiment manuals (if needed)- clicking Add manual displays another window where you can type your experiment manual with the use of basic formatting tools. Once you're satisfied with the properties you have defined for your experiment, click OK. These properties will be serialized along with the experiment and stored in the experiment file.

Management of interpreter gems

A "bare-bones" interpreter may often be insufficient to run a given script – developers frequently make use of external modules and libraries, provided by their own collaborators or third-party developers. In GridSpace, such modules are called gems (akin to the Ruby concept of a scripting gem). Gems can be imported and installed, whereupon they become available to experiment developers and can be invoked in GridSpace experiments – provided you're using the correct interpreter.

To install a new gem, you need to know its name. Go to the Gem management panel (click Gems management below the file management panel), select the tab which corresponds to the scripting language you wish to use, and type in the name of your desired gem (or part thereof). GridSpace will automatically scan public repositories and present a list of available gems. For each gem, the following options are provided:

  • Display a description of the selected gem
  • Install the selected gem (+)
  • Uninstall the selected gem, if already installed (-)

Back to the top

NOTE! This web site uses cookies and similar technologies (if you do not change the browser settings, you agree to this).

cyfronet agh