Support

Support Options

Submit a Support Ticket

 
You are here: Home » Tools & Resources » Tools » Data Transformer » About

0 Citation(s)

0 questions (Ask a question)

0 review(s) (Review this)

3 users, detailed usage

0 wish(es) (Add a new wish)

Data Transformer

Transform experiment data files in different ways. For example, this tool can transform: units, format, and sampling rate.

Launch Tool

This tool is distrubuted as a downloadable file. To download, please visit the 'Docs and Attachments' section, or click the button below.

Download

Version 1.0 - published on 16 Oct 2012

Open source: license | download

View All Supporting Documents

See also

No results found.

Category

Tools

Published on

Abstract

Introduction

 

The purpose of the Data Transformer tool is to assist engineers and researchers transform experiment data files in various ways.  Some transformations can easily be applied using a tool such as Matlab or Microsoft Excel.  Other transformations are more complex and cannot be performed by standard tools due to time constraints or missing functionality.  Data Transformer is much faster than using Matlab to convert data.  This tool will allow engineers and researchers to perform more advanced transformations on data files using custom processing rules.  Also, this tool is optimized to run efficiently over large data sets and batch transformation operations.  It can be useful for post-experiment data processing.   Finally, the system was designed to be extended. New custom processing nodes can be easily coded and hooked into the Data Transformer. It’s designed as a pluggable software architecture.
 
The first release has a very limited set of transformation nodes implemented.  Future versions of this tool will include the implementation of new transformation operations based on the needs of the NEES community.

Domain Model

The domain model for this tool is shown in the diagram below.  The software is a graph-based data processing system.  The data signals have already been recorded in files (e.g. CSV, or XML).  These files have instances of samples.  These samples are snapshots in time for a given signal's value.  This software allows an engineer or researcher to construct a graph of "SampleNode" instances that read data, transform it, and write it back out to a file.

Processing Rules

This software takes a graph of processing rules as a launch parameter.

Here is an example XML file of rules:

The rules above define a graph of processing nodes (also known as Sample Nodes).  These nodes are connected in a parent-child relationship.  The parent nodes feed sample data to child nodes.  Each node performs some operation on a sample.  For example, the "sample-rate-transform" node above filters data from a given sample rate to a new sample rate in Hertz.  That means that for every ten samples that are put into this node, only one sample is every sent to the child node.

 

Prerequisites

  • Apache Ant (1.8.3 was tested)
  • Java Development Kit 1.7+

Building

These are the instructions for building the software on Linux:

  1. Check out the code from NEEShub using Subversion.
  2. Navigate to the directory where the code is checked out at the command line.
  3. Build the code using Ant as follows:
    $ ant dist
  4. A ZIP file will be created in the "target/zip" directory that contains everything you need to run the program.
  5. Copy that ZIP file to a location where you want to extract it.  For example:
    $ cp target/zip/data-transformer.zip /tmp/
  6. Unzip the ZIP file.  For example:
    $ unzip data-transformer.zip
  7. Navigate to the new directory containing the JAR file.  For example:
    $ cd data-transformer

Now you are ready to run the program.

Command Line Usage

 

To run this program, you need to specify an XML file containing the transformation rules.  For example:

$ java -jar data-transformer.jar -g example-transform-graph.xml

After the command above is run, the rules in the XML file will be invoked and your data will be transformed.
 
Here is a description of the command line arguments:
usage: data-transformer
 -g   path to XML graph file containing transform rules

 

 

Powered by

Java

Sponsored by

NEES and the University of Minnesota

Cite this work

Researchers should cite this work as follows:

  • Michael Boldischar (2012), "Data Transformer," http://nees.org/resources/datatransformer.

    BibTex | EndNote

Tags

  1. CSV
  2. experiment
  3. xml