Skip to content. | Skip to navigation

Personal tools
Log in

DIstributed Computing Environments (DICE) Team
We are a group of computer scientists and IT experts from the Department of Computer Science AGH and ACC Cyfronet AGH. We are a curiosity- and research-driven team, specializing in large-scale distributed computing, HPC, Web and Cloud technologies. We develop new methods, tools and environments and we apply these solutions in e-Science, healthcare and industrial domains.


Sections
You are here: Home Projects Projects full descriptions Mapper-files MaMe User Manual

MaMe User Manual

Introduction

MAPPER Memory (MaMe) main responsibility is to provide rich, semantics-aware persistence store for other components to record information. The MaMe registry is meant to deliver its functionality based on a well-defined domain model which includes all important elements of MAPPER metadata defined in MML: scale modules, mappers and filters, together with their ports, implementations and other constituting elements and attributes. Thanks to MaMe, other tools may store, publish and share common registry of such elements throughout the entire project.

Another component of MaMe registry concerns the repository of xMML application descriptions. The main objective of this effort is to deliver the users a persistent and shareable storage for constructed multiscale application descriptions in the form of xMML documents.

Finally, MaMe serves a web user interface for project administrators and applications developers. They are able to use it for setting up their multi-scale application modules and parts. The following document describes how to use MaMe to set up one's multi-scale application in the MAPPER environment.

This documentation assumes the user has the knowledge of (x)MML multi-scale modeling language: its purpose and its main constituting elements, since MaMe internals depend heavily on this description notation.

 

Main Menu

The official deployment (or installation) of MaMe for the MAPPER Consortium is available here:

http://gs2.mapper-project.eu/mame/

This URL is available openly for everyone in the read-only mode. If you'd like to change things there or register your own application, you'll need the permission of the MAPPER Project's leads (please contact Katarzyna Rycerz in order to obtain credentials). Otherwise, you are welcome to setup your own installation of MaMe with the help of MaMe Administrator Manual, also published on these web pages.

In any case, the main menu of MaMe you're about to encounter, looks like this:

mainmenu.png

Fig. 1: MAPPER Memory main menu with the links to the two main elements of the registry.

As explained before, there are two main building blocks of MaMe, one being the Models Registry (which is the most important from our point of view) and the other is the Experiments Repository for xMML application documents. Since the other one is mainly for administrators, we'd navigate to the Models Registry, in order to learn what's inside.

 

Registry of Modules Metadata

The primary purpose of the module metadata registry is to persistently store and publish descriptions of scale models, mappers and filters developed for multi-scale applications. To deliver such functionality, MaMe employs three-layer architecture, with persistent database, semantic domain model and external user and API interfaces. The current version provides a set of possibilities of interaction which we elaborate on in the following sections.

Browsing registered elements

Inside the registry you are able to see all the registered scale models, mappers and filters as the basic building blocks for any multi-scale or multi-physics application. The defined ports and implementations of the elements are also presented. Before we get to them, however, let's see a sample scale module (or model).

Scale models

Let us start with a sample scale model presented in MaMe:

singlemodel.png

Fig. 2: Single scale module, as registered in MaMe, with its temporal and spatial scales defined.

The model metadata starts with its name and then goes to the list of its scales. The obligatory temporal scale (or timescale) tells us what is a minimal step on its grid (in order words, what is the least time period recognized by the model), called delta, and how much time the total model spans (more or less, how long is the time period the model is able to simulate). The same goes for the spatial scale (might be more than one) – there are the delta and total attributes describing it.

There are also some other, optional elements of model metadata. First is a description, solely for human user purposes (its presence is depicted with a small scroll icon). Each model might (or might not) also be:

  • initial – the model should run as an initial submodel, so the runtime system should activate it before other elements (which, for instance, can be used to bootstrap a closed loop),
  • stateful – normally, a model does not have any memory or state beyond its final observation, sometimes, however, a model can use the results of a previous computations, after being reinitialized (the runtime system may use this information to schedule the next instantiation of the model on a different machine, and even at overlapping time points, if it is not stateful),
  • interactive – the model allows interactivity with the user, for instance to have a visualization or a supervised selection (in case of mappers, see below).

With the pencil icons you are able to edit the model, its time and space scales. You are also able to add new space scales with the plus icon and to remove them with the x icon.

modelports.png

Fig. 3: List of ports of a sample scale model in the registry.

By clicking the "Load ports" button you unroll the list of ports of this model. Let's see it now.

Each port is characterized with its name, the operator, the type and the datatype. Now, in order to learn what it precisely means, we advise you to consult the MML documentation. For now, however, let's just iterate through them briefly:

  • operator – it decides what information the model is exposing to the external entities and, in turn, what information it may require to steer its internal computations; there are several types of multi-scale MML operators: finit, Oi, Of, B, S,
  • type – either normal or state (for now, only the normal setting is supported by MAPPER runtime),
  • datatype – what kind of data elements the port is able to produce or consume (a notable example is "file", which denotes that elements of the multi-scale application communicate through files),
  • direction (depicted with the small arrow to the left of the port name) – does the port produce (out) or consume (in) data.

When you click the plus icon near the "Ports" label, you'll be presented with the dialog for defining new ports.

newport.png

Fig. 4: A user dialog to define a new port.

Apart from the MML settings we just mentioned, one more metadata element about ports is whether its name is fixed or not. If a port name is fixed, it is being denoted with a small exclamation mark icon by the port name. If the fixed flag is enabled for a given port, its name will not be changed during MML transformations. This is necessary for cases where the implementation of a corresponding submodel/mapper expects input or output files to have specific names.
Up to now, we were dealing with the abstract description of a scale model. While it provides important information about the model to the system, it will not allow us (or anyone else) to execute it. For that we need an implementation.

modelimpl.png

Fig. 5: Implementation of a scale model with its runtime properties and execution parameters.

What we have here is a java-based implementation. Apart from an implementation name, two other important settings are class and bundle – they tell the runtime MAPPER components what element of which distribution to call in order to have the simulation model running. In this case we have a Java class and a Java archive (jar) given, though there may be different values in here, for instance it could be a directory (in the file system) or a system module. In case of MUSCLE 1 implementations bundle property should contain a list of all jar files and directories holding Java class files. For MUSCLE 2 implementations it should contain the name of a system module which needs to be loaded before the execution takes place.

The list of parameters depends on the implementation – these are the values the implementation code expects to be provided on the model execution. The name of a parameter is followed by its type and the default value. The type has two elements to it; it described the nature of the parameter and the datatype it is represented with (after a colon sign). While the second part is rather self explanatory ('number' for all numeric values and 'string' for string values), the first element can add a special meaning to the parameter in the following cases:

  • globalmuscle – sets a global MUSCLE parameter in the CxA configuration section,
  • libpath – appends the parameter's value to the native library configuration paths in CxA.

e.g.
one can define max_timesteps parameter as a global one with:

max_timesteps of type [globalmuscle:number] = '100'

one can define native library path:

nativelibs of type [libpath:string] = '#{ENV['HOME']}/application/build/'

In case of some implementations (e.g. script languages - Ruby, Perl or domain specific languages for scientific software like Gaussian, Matlab, LAMMPS or CPMD, etc.) the author wants to supply an additional configuration file, which is used afterwards in the process of model implementation execution. MaMe does not impose any restriction on the form or content of this file. The file is used by the runtime to prepare the code which is executed by the configured interpreter.

Now, if you'd like to add your own model, please navigate to the bottom of the view and click the “New submodel” button. You will be presented with a form dialog where you can set all the metadata about the model we have described above. Don't worry that there are no spatial scales, ports and implementations in the dialog – you will be able to add them later on using the plus icons in the description of the newly added model.

Mappers

The second type of building block of any multi-scale application, according to the MML standard, is a mapper. The main difference of mapper, compared to a scale model, is its lack of scale. In order words, a mapper is scaleless. It is, however, a first class citizen of any multi-scale application and as such it can have its ports and implementations like any scale model does.

mapperfull.png

Fig. 6: An example of a mapper with its ports and implementations list presented.

The nature or a mapper is defined mainly by its type. It might be either of:

  • fan-in: the mapper accepts input from many modules and passes it as output to single module,
  • fan-out: the mapper accepts a single module input and is able to pass output to many scale modules,
  • fan-in-out: a combination of the cases above, a many-to-many scenario.

When it comes to ports, implementations and parameters of implementations, mappers are fairly similar to scale models. You can use the same dialogs and forms to fill in information about mapper's ports or implementations. In the example above we show an example of a mapper with three fixed-name ports and two different implementations. As you may note, this particular mapper has still some of its metadata pieces missing, so it's probably in the middle of the preparation process.

In order to add a new mapper to the Registry, again navigate to the bottom of the view and use the "New mapper" button for the appropriate form.

Filters

Filters are additional elements of multi-physics applications, which may or may not be used, according to the specific case.

filterfull.png

Fig. 7: A filter of converter type with a single implementation (not shown in its entirety).

Again, filters like mappers lack scales. What is more in this particular case, filters also are not equipped in ports. That is so because they work on a single link of data – each filter accept a single input and produces a single output, after applying its internal processing to the passing data.

There are three types of filters available in MML:

  • reduction – the filter reduces the number of messages that are sent in, based on a given factor or algorithm,
  • interpolation – the filter increases the number of messages that are sent in, based on a given factor or algorithm,
  • converter – the filter performs any other data or time conversion.

The type of the filter is currently not taken into consideration when running it on the computing infrastructure, but it might be important in the future releases of MAPPER software (for instance, the task graph generation algorithm or scheduling algorithm can treat these types differently).

A new filter may be added to the registry with the "New filter" button on the bottom of the view. There, apart from the mentioned elements, you are able to define the following additional metadata:

  • datatype in – what data elements the filter accepts on entry,
  • datatype out – what data elements the filter produces as its output,
  • factor – the factor with which the data is manipulated, which may apply to a multiplication of numeric data, or a time scale multiplication (this depends on the filter algorithm),
  • dimension – the only dimension the filter does conversions over (this might be used to perform additional checking: if only a single dimension is modified, the others should stay the same, and this should be reflected in the scales).

Again, the last two xMML settings are not used at the moment and are left here for future implementations.

Model Registry Programmers API

Apart from its persistence capabilities and its user-oriented web interface for metadata browsing and maintenance, MaMe also provides API interface for other tools to connect with the registry and to publish or retrieve stored information. Currently, MaMe API provides the following remote operations (all based on the REST principles).

Models Registry part:

  • /models-list: lists all the registered models, mappers and filters with their full descriptions,
  • /add_base/Submodel: registers a new scale model,
  • /add_base/Mapper: registers a new mapper,
  • /add_base/Filter: registers a new filter,
  • /add_implementation/(Mapper|Submodel|Filter)/'id'/: adds an implementation of a given type of element (either scale model, mapper or filter).

The API is documented online so each mapper developer has access to it at anytime (simply click the "API Help" buttons by the items of the main menu to see the list of operations).

api-help.png

Fig. 8: Each available API call is described within an online API documentation provided with MaMe.

The API is based on a simple HTTP/REST protocol, where you use GET or POST calls of specified endpoint in order to have the expected result. The listed parameters should be passed as an HTTP call parameters; according to the used method, they should be placed either in the URL (GET) or in the form data (POST). In order to call write operations (the ones that add, modify or remove something) you will need to provide the user and password in the process of HTTP Basic Authentication.

 

Experiment Repository of xMML Descriptions

Another component of MaMe registry concerns the repository of xMML application descriptions. The main objective of this effort is to deliver the other MAPPER tool users a persistent and shareable storage for constructed multiscale application descriptions in the form of xMML schema. These descriptions may be written by the users by hand but the preferred and better method of planning applications is to use the MAD tool to design a new multiscale application in a user-friendly way. MAD is able both to load saved XMML description and the save the new ones using this repository. As such, the Experiment Repository web user interface inside MaMe is not very important, not interesting for the users. We will, however, present it briefly, for the sake of completeness.

mame-exprepo.png

Fig. 9: A portion of the Experiment Repository inside MaMe.

Each xMML experiment created and saved with MAD is being shown in the list. You are able to browse the list, see the contents of experiments, their descriptions and the time of creation.

In order to provide the method of removing unnecessary, older versions of experiments, the experiment archive is being provided - removing any xMML experiment description from MaMe will move it to the archive (instead of deleting it completely). Such archived experiments do not clutter MaMe and MAD displays but are easy to retrieve back with the use of "Reload from Archive" capability of MaMe.

Experiment Repository Programmers API

In order to accommodate MAD and similar tool using MaMe internally, a HTTP/REST interface was developed for the Experiment Repository. It is very similar in its mode of operation to the API of the Model Registry described above. Currently, it contains three operations:

  • experiments_list – returns the full list of available (i.e., not archived) experiments and their metadata, in JSON format,
  • experiment_content – returns the full xMML document for a specific experiment saved in MaMe,
  • save_experiment – performs an "upsert" operation on a given experiment: that is it saves it when no such experiment existed of it updates its content, if it had been already recorded by MaMe in the past.

 

Summary

The current version of MaMe (both the Models Registry and the Experiment XMML Repository) is permanently provided under the link

http://gs2.mapper-project.eu/mame/

It has completely open read access to every interested party – so any multi-scale application developer or any scientist running such applications is able to read about available models and experiments, developed within the MAPPER Consortium. The write access is, however, blocked for such external people and only the members of the Project are presented with appropriate credentials to be able to modify the contents of the registry.

The same rule applies to the REST API for other tools: the read operations are open and the insert/delete/modify operations are secured.

The current revision of the MaMe source code is available at two distinct parts:

  • https://gforge.cyfronet.pl/svn/sint/trunk/mame (the MaMe tool),
  • https://gforge.cyfronet.pl/svn/sint/trunk/sintmodel_mapper (the semantic Mapper data model).

In order to checkout the source code of either of these projects, please use the following command:

svn checkout --username anonsvn <SOURCE_URL>

supplying one of the URLs given above and the 'anonsvn' password. This is for read only access.

This distinction is made in order to keep the tool design clear and properly decomposed. In order to install your own deployment of MaMe, please consult the MaMe Administrator Manual provided alongside this document.

 

Copyright owner of MAPPER Memory is DIstributed Computing Environments (DICE) Team who makes it available to public under open source MIT license .

The MaMe domain model and the web interface are being developed by the Academic Computer Centre Cyfronet AGH and are partially funded by the MAPPER project, RI-261507.

NOTE! This web site uses cookies and similar technologies (if you do not change the browser settings, you agree to this).

cyfronet agh