FlyElephant as a tool for calculations in C++, R, Python, or Octave
Greetings to all!
Today I will tell you about the possibilities of the FlyElephant platform for scientists and engineers who work on various calculations using C++, R, Python, or Octave. These tasks may be scientific calculations, data analysis, modeling, or other. On January 26th 2016 I will give a webinar on "Introduction to FlyElephant", and I will tell you about the platform in detail.
FlyElephant is a platform that provides scientists with a ready computing infrastructure for calculations. It automates routine tasks and allows you to focus on the core issues of your research.
As of now we are using Azure cloud as a computational resource. But our team is continuously working on the construction of its own computing cluster by connecting to the Amazon cloud and a significant number of supercomputers and GRID systems, which will significantly expand our computing power.
A frequently asked question is: ***"Why are you better than the clouds?"***. Of course each user can take the cloud, deploy his/her own infrastructure, configure it, maintain the system and change it if necessary. If you use FlyElephant, we take care of all of these time consuming aspects! You get a complete or custom-made infrastructure of set parameters with the necessary software. In addition, in the future, you will not only be able to perform calculations in the cloud, but to use supercomputers and other specialized computing resources that are not always available to public access. You will be able to integrate with the platform with your own computing capacity, cluster or cloud, and if necessary expand your capacity or lease other available resources that we can provide to you.
FlyElephant Platform is not just a tool that helps you to perform calculations, but it is also a community.
One of the priorities of our team this year is the creation of a strong community around FlyElephant.
We want to unite experts from various fields in one place, which require the use of large computing power for solving problems, help them to share knowledge and solve problems, as well as to motivate them to collaborate on projects. While working full time on this functionality, I invite all to our slack-chat.
What languages and tools are available?
Users can run computing tasks written in C++ (with support for OpenMP), R, Python (v2 & v3), Octave and Scilab on servers with a core number from 1 to 32 and memory up to 448 GB. In addition, participants in the beta program are now helping us to debug the operation for the following tools: Java (v7 & v8), Julia, OpenFOAM, GROMACS, Blender and Public API. Next in line are (visual) result evaluation tools and the creation of a database to collect and store large amounts of information. We are actively working on expanding our catalog, so if you have the need for any tools, please, let us know and we will try to integrate it into the platform as quickly as possible.
How to start a task?
Using the example of a program written in C ++, R, Python, or Octave, we now explain step-by-step the whole process of working with FlyElephant. To get started, we need three things:
First - you need to register
If you have not done it yet, register here. After registering you’ll have $ 10 in your account; using the promotional code 127575063277 and entering it in the “Dashboard”, you can recharge your account for another $100.
Second - a repository with the code of your program is required.
We have prepared four test repositories, which can be used:
The repository must contain files with the code of your program, as well as two mandatory files: “build.sh” and “run.sh”. At first you need to write the commands which must be executed on the server before starting the calculation (this could be some kind of further configuration of the environment, read-me file, or something else). In the second file you write the start command of the computer program.
Third - to load the data.
If for your calculation there is a need to work with some data that is not in the repository or in an external source, you will need to download it to the “Storage” section.
When everything is ready, you can start the calculation. To do this, go to the system and perform the following steps:
- Go to the "Computing" and choose the language or tool you need.
- Click "Create a task".
- A fill-out form appears in which you must fill the fields as follows:
- Task name - give the task a name.
- Choose your data file - if necessary, specify the file you want to upload to the server to perform a task. You can accomplish it without downloading additional files; select the file that you downloaded in the "Storage"; select the file, which is the result of another task. The file will be uploaded to the server in the root directory of your repository.
- Result file name - specify the name of the file, where the program will save the results. This file will be loaded in the "Storage" section and will be available there as well as the details of the task.
- Repository – specify the address of the repository in which your program is located. With this version only work with public repositories is possible, but in the next update there will be a possibility to work with closed ones.
- Server Settings - choose a server configuration in which the program will be launched for execution. There is a possibility to run a task on a server with the number of cores from 1 to 32 and memory up to 448 GB. In next releases there will also be updates related to the possibility of a more flexible configuration of computing infrastructure and the ability to run the task not only in the cloud, but also on the HPC cluster or in GRID.
- Click "Start task". The task starts. It automatically becomes available in the section "Tasks". Here you can see the details of the implementation, results and logs.
Thus, by following a few steps, you get the result.
We are working on improving the user interface and in the nearest future we will expand the list of supported tools and computing resources. If you are interested in adding tools, which are necessary for you, we invite you to take part in a survey.
Our team is open for cooperation. We look forward to working with the owners of computing resources and producers of specialized software. We actively support scientists and are ready to provide the required amount of resources for your open scientific research for free. If you are interested in using FlyElephant, we are happy to hold a DEMO presentation of the platform for you.
Any comments, suggestions and feedback are very important for us. We will answer in the comment section of this post, or via e-mail: firstname.lastname@example.org.