Mittwoch, 13. Juli 2016

Developing a custom code generator by example of SCXML

This blog post explains how to develop a custom code generator for the open source framework YAKINDU Statechart Tools. As an example, we will generate Statechart XML (SCXML) from a simple stop watch state machine. The stop watch use case is taken from the Apache Commons SCXML site. The stop watch model consists of 4 different states (ready, running, stopped and paused) and the state transitions are triggered by events that represent the stop watch buttons (watch.start, watch.stop, watch.split, watch.unsplit, watch.reset). 

Stop Watch example
Note that this example is not a full fletched Statechart XML code generator; only the basic concepts of states, transitions and events are supported to showcase how to get started. We expect the following XML fragment to be generated from the model above:
<?xml version="1.0" encoding="UTF-8"?>
<scxml xmlns="http://www.w3.org/2005/07/scxml" version="1.0" initial="ready">
   <state id="ready">
      <transition event="watch.start" target="running" />
   </state>
   <state id="running">
      <transition event="watch.split" target="paused" />
      <transition event="watch.stop" target="stopped" />
   </state>
   <state id="paused">
      <transition event="watch.stop" target="stopped" />
      <transition event="watch.unsplit" target="running" />
   </state>
   <state id="stopped">
      <transition event="watch.reset" target="ready" />
   </state>
</scxml>

SExec and SGraph meta models

Before starting with a code generator, we have to decide which of the two available meta models is most suitable for the use case. In model driven software development, a model always conforms to another model commonly referred to as a meta model. This meta model defines the building blocks a model can consist of. When developing a generator for YAKINDU SCT you can choose between two different meta models, SGraph and SExec. In broad terms, SGraph represents the structure of the modeled statemachine (States, Transitions, Events) while SExec contains sequences that encapsulates our execution semantics (for example Checks, Calls and Stateswitch).
The SGraph meta model is similar to the state machine meta model known from the UML. We have a top level element Statechart that contains a set of States and Transitions. Transitions may have events and a source and a target State. The following shows a very simplified version of SGraph.
sgraph.png
In contrast, the SExec meta model defines an ExecutionFlow that contains a set of ExecutionStates. Every ExecutionState has an entry and an exit sequence of several Steps. Each Step can be of a different type, for example a StateSwitch or a Check.
sexec.png
If you want to generate code for a runtime environment that interprets a statechart structure (for example SCXML or Spring Statemachine) you should pick SGraph meta model. If you want to generate a code-only approach for another programming language, and you want to ensure that it behaves exactly in the way all the other code generators behave, you should pick the SExec meta model. 

The example Statechart XML generator

Creating custom code generators is really easy. It is a first-level concept in YAKINDU Statechart Tools. You can develop them directly within your project workspace which has the nice benefit that adoptions to the model or the generator templates are executed instantly, hence you don't have to bother with long code-compile-test cycles. More information about how to setup a new code generator project is out of the scope of this blog post and can be found here
When choosing the language for developing a new generator, you should consider using Xtend instead of Java. The syntax is familiar for every Java developer, but it adds some nice features like template expressions and dispatch methods that are really useful for developing code generators.
The example Statechart XML generator implemented with Xtend could look like this:
The SCXMLGenerator class extends AbstractWorkspaceGenerator to inherit some utility functions and implements the ISGraphGenerator interface (line 17)This interface defines which meta model will be used for the code generator. If you want to implement a code generator based on the ExecutionFlow meta model you would implement IExecutionFlowGenerator interface instead. 
The generate function in line 23 uses Xtends template string to generate the SCXML header. Most of the text ist static except for the value of the initial attribute  - this is calculated in the initialState function (line 30).  Below the SCXML header, the generate function for the head region (parallel regions are not implemented) is called (line 26). This function (line 34) simply iterates over all states in this region and calls the generate function for states (line 40). Other vertex types, for example Histories, Synchronizations or Final States are filtered out. Last, the generate function for states creates a new XML element and iterates over all outgoing transitions to generate the nested transition element.
You can download the example generator project from our new examples repository. However, this is just a first step towards a full fletched SCXML code generator. Feel free to take this example as a starting point but don't forget to contribute your extensions to our repository! If you have any questions or need further support on implementing custom code generators feel free to contact us.

Dienstag, 15. Dezember 2015

YAKINDU Statechart Tools 2.5.0 released!


Today, we released the new version, 2.5.0, of YAKINDU Statechart Tools to pass the time until Christmas Eve!
This is the first release of the statechart tool that has been based on the results of our ongoing survey. Thank you to everyone who took part. We really appreciate your help.
We provide an update site that is compatible with Eclipse Luna and Mars as well as a product bundle that is shipped with an Eclipse Mars.SR1. Get it for free.
In total, 54 bugs have been fixed and a lot of feature requests have been implemented. You can find a complete list of all fixed issues here. We also did some major refactoring "under the hood" as preliminary work for the upcoming YAKINDU Statechart Tools Professional Edition.
Let's have a look what's new and noteworthy.

Extended Documentation and Tutorials

The new product bundle is shipped with a welcome page for easy access to example projects and tutorials.
This is especially useful for statechart tool beginners that do not have experience with Eclipse yet. We also added some new examples that demonstrate the different features of YAKINDU Statechart Tools.

Constant Declarations

We extended our expression language to allow the definition of constant values. Instead of using the 'var' keyword you can now use the 'const' keyword to define constant values. All code generators are adapted to generate constant definitions.

Task Markers

With the release of YAKINDU Statechart Tools 2.5.0 we added task markers to the expression language. By default you will get markers for TODO and FIXME in single and multiline comments. These tags will also be highlighted and tasks will appear in the tasks view. 

You can open the view via Window -> Show View -> Tasks.

Enhanced Source Code Generation

The quality of the generated source code has been further improved towards a MISRA compatible c code generation. The following issues have been fixed:
  • generated functions have a single return point (MISRA-C:2004 Rule 14.07).
  • query functions have constant pointers (MISRA-C:2004 Rule 16-07).
  • control statements are enclosed with braces (MISRA-C:2004 Rule 14.09).
  • single line comments replaced with multi line comments (MISRA-C:2004 Rule 02.02 for C99).

Runnable Wrapper for the Java state machine

The GeneratorModel for Java Statemachines contains a new optional feature for a runnable wrapper class now. It can be configured like this:

The additional generated Java class provides a thread-safe wrapper for the generated state machine implementation. In addition to the specific state machine interface it implements the Runnable interface and can be executed in a thread. It implements an event queue and event-driven execution semantics. As the wrapper adds thread safety any number of client threads may call the state machine.

Improved simulation and sub diagram highlighting

The highlighting support that is used during simulation to highlight active states and transitions was reworked to improve performance when simulating large statecharts.

Subdiagrams are highlighted in the image tooltip on the subdiagram icon as well as in the editor when diving into the sub diagram.

Miscellaneous

We also fixed a lot of issues reported via our User Group, for example, a whole chart documentation mode switch was implemented and a reported problem with sc_string type in C-Generator. So if you have any issues regarding the new version, we would be happy to get feedback!
Download YAKINDU Statechart Tools

Dienstag, 13. Oktober 2015

Yakindu Statechart Tools Survey - Intermediate Results

Ten days ago we started a short survey about YAKINDU Statechart Tools to get a better understanding what we have to improve and which features we should implemement next. The survey is still online - so if you are a Statechart Tools user and could spend 5 minutes to complete that'd be great. 

Begin Survey

Thanks to all the participants so far! Let us have a detailed look at the survey results. For all those nitpickers out there who notice that the overall sum is above 100 % for some questions - this is because multi selection was allowed ;).

Working area and kind of software

More than half of the survey participants work in the Industry area and want to use YAKINDU Statechart Tools for their commerical products. Nearly the same amount of users work in the Research (30%) and Education (20%) domain. Nice to see that there are also some people using YAKINDU for their Private projects just for fun. I would love to hear from you and the stuff you are building with Statechart Tools, just drop me an email or leave a comment!

The majority of all respondents want to develop software for Embedded Systems (70 %) and Human-Machine-Interfaces (40 %) as well as Automation Software (35%). About 20 % want to use Statechart Tools for Enterprise Software, Web Applications and Mobile Applications. Until this time, Statechart Tools are not used in the area of game development.

Experience with Statemachines and other Statechart Tools

24 % of the overall respondents state that they never used statemachines before. 60 % use statemachines from time to time and no fewer than 16% call themselfs statemachine experts. Nearly 50 % worked with other Statemachine Tools before using YAKINDU Statechart Tools. Most popular Statemachine Tools mentioned are Matlab Stateflow, Enterprise Architect, IAR Visual State, SCADE and Rational Rhapsody.

Why are you using YAKINDU Statechart Tools?

The majority of all users want to generate C code out of their statemachines (65%), while 50 % want to generate Java and C++ Code. At least 24% of our users want to generate code for other programming languages, most mentioned Javascript, Phyton and VHDL.



About 24 % of our users use YAKINDU Statechart Tools to learn about statecharts, 13 % of them also want to teach statecharts. But the majority (58%) want to develop software and want to use it for system modeling (40%). Really surprising to me was the fact that nearly 30 %' of all participants 'care about modeling toolchains and want to integrate' SCT in their IDE. This is fairly easy - thanks to open source and the underlying Eclipse Platform.

How do you rate the tool and what can we improve?

Last but not least we asked our users how they rate different aspects of YAKINDU Statechart Tools on a scale from 1 (poor) to 4 (perfect). As you can see in the diagram on the right, the average rating for Graphical Editing, Simulation,Validation, Code Generation and Usability is 3 (good). Of course, 'good' is not good enough, we will further improve these tool features. 

Only Documentation & Tutorials was rated with an average of 2 (does the job). We will improve the documentation in the future and we are also planning an ebook with additional examples.
The 'feature-wish' section of our survey was a great source of inspiration for us. Of course, we can not implement all those cool features at once, but we are open for external contributions. If  you want to integrate a model checker like UPPAAL (someone is already prototyping this), or add an importer for stateflow or other features just fork us on GitHub!



Mittwoch, 19. August 2015

Building Eclipse Plugins with Maven Tycho and Travis-CI



A couple of weeks ago, we moved our open source project Yakindu Statechart Tools from google code SVN to GitHub. Until today we packaged and deployed our software using a self hosted Jenkins Server, but since Travis CI integrates seamlessly with GitHub and is free for open source projects we decided to give it a try. This blog post tries to explain how to set up Travis CI for a Maven Tycho based eclipse build.

There are plenty of blog posts out there, that stated how easy it was to setup a Travis CI build for a 10 lines of Java code 'Hello World' example. This is great, but our builds have a lot more work to do:
  • 170,000 lines of Java code
  • Code generation of Xtend classes and Xtext grammars
  • more than 1000 JUnit Tests (including UI Tests)
  • ~100 C and C++ Google Tests to run 
Let's see how well Travis can handle this.
 
Initial Setup

To get started with Travis you first have to log in to travis-ci.org with your GitHub account and grant access to your GitHub repositories This requires admin access to the repository. It is well documented in the User Guide. Next, you have to create a file .travis.yml in the root folder of your repository to tell Travis what to do. By the way, there is a really useful online syntax checker for yml files available here.

This is the simplest possible .travis.yml file for our build based on Maven Tycho:
 language: java   
 script:    
  - cd releng/org.yakindu.sct.releng   
  - mvn clean verify   

Immediately after pushing the .travis.yml file to our Git repository Travis started a new build.
and we ended up in a 5 MB log file full of exceptions like this one:

Caused by: org.eclipse.swt.SWTError: No more handles [gtk_init_check() failed]
    at org.eclipse.swt.SWT.error(SWT.java:4517)
    at org.eclipse.swt.widgets.Display.createDisplay(Display.java:908)


org.yakindu.sct.generator.genmodel.ui.SGenExecutableExtensionFactory
    at org.eclipse.swt.SWT.error(SWT.java:4517)
    at org.eclipse.swt.widgets.Display.createDisplay(Display.java:908) 

Running UI Tests

After analyzing the log files it points out that our UI dependent tests are not able to create an SWT Display object. Luckily, Travis allows the use of the virtual framebuffer xvfb for UI tests. It can be started as follows:

  language: java    
  env:  
   global:  
    - DISPLAY=:99.0  
  before_install:  
   - sh -e /etc/init.d/xvfb start - sleep 10  
  script:    
  - cd releng/org.yakindu.sct.releng    
  - mvn clean verify    

After pushing these changes to Git the build finished successful!

User Interface

Travis provides a simple yet powerful web based user interface that displays a list of all your repository builds and the console output. This is really helpful to see what is going wrong with your build. It also allows to start a build without pushing something to your repository.
What I really miss in the UI is a better integration for Unit Tests, like the Jenkins JUnit plugin. But such a feature is unfortunately not planned and one has to browse the (huge) log files for failed Unit tests.

 GitHub Integration

Per default, the master branch as well as all pull requests are build. The integration with GitHub works out of the box, the running CI jobs are shown in the GitHub UI as checks and updated automatically when the job finishes. This is really awesome for a configuration file with only 9 lines!


 Caching and container based infrastructure

Every time a new build job starts, a new Linux image is setup. To prevent that Maven downloads the internet every time a new build is started, Travis provides a container based infrastructure that allows the use of caches. One drawback of this is that the use of the sudo command is prohibited.

 sudo: false  
 language: java    
 cache:  
  directories:  
   - $HOME/.m2  
 env:  
  global:  
   - DISPLAY=:99.0  
 before_install:  
  - sh -e /etc/init.d/xvfb start - sleep 10  
 script:    
  - cd releng/org.yakindu.sct.releng    
  - mvn clean verify    

As shown above, starting the .yml configuration file with sudo: false allows the use of caches. When the build starts and a new Linux image is created, all cached directories (in our example the maven repository .m2) are restored at the beginning. This really boosts up the build time.

Performance

Our existing Jenkins CI Server is hosted on a Linux KVM virtual machine with 12 core Opteron 2,6 Ghz, 32 GB RAM and  Raid 10 HDD. The average build time is about 13 - 20 minutes.
Travis, running somewhere on Amazon cloud services, took about 10 - 11 minutes for a build. I don't know how they do it, but this is incredibly fast!


Adding google test framework for C and C++

Since Yakindu Statechart Tools is shipped with a C and C++ code generator, we have a couple of C and C++ tests that have to be run during build. This was the not-so-funny part of the configuration, because google recently decided to ship libgtest only as source code and without the static libraries. On the Linux machine we are running Jenkins it was straight forward to install google test, compile it and copy the static library to /usr/lib with the following commands:

 sudo apt-get install libgtest-dev  
 cd /usr/src/gtest  
 sudo cmake CMakeLists.txt  
 sudo make  
 sudo cp *.a /usr/lib  

Running these commands at the beginning of the Travis build would be possible, but we want to use caching so we are not allowed to use sudo, as explained above. Fortunately, there is an apt get whitelist that allows the installation of external libraries via addons.apt.packages:

 sudo: false  
 language: java  
 addons:  
  apt:  
  packages:  
  - libgtest-dev  
... 

Now the GTest source code is available on the image, but it has to be compiled. The - somehow quite hacky - solution is to copy the source into the build dir, compile it and create a environment variable GTEST_DIR that is used in our C Tests for the library lookup. Copying to usr/lib did not work because without sudo the permission is denied.
 env:  
  global:  
   - GTEST_DIR=${TRAVIS_BUILD_DIR}/gtest
  before_script:  
  - mkdir gtest   
  - cd gtest  
  - cp -r /usr/src/gtest/. .  
  - ls  
  - cmake CMakeLists.txt  
  - make   
... 

 At least this works, if you know a better solution please let me know :)
  
Publish Releases
Last but not least, we want to publish successful release builds to GitHub releases automatically.
To publish artifacts to GitHub releases a secret api_key is required for Travis to get access. These api_key should be encrypted, especially when the .travis.yml file is part of an open source project. ;-)
To set up the deploy section of the configuration, you need to install the Travis Ruby Gem.
First, download Ruby version 2.0.0 (I tried to install the Travis Gem with a newer version but it did not work) and install the ruby gem as described here.
If everything works properly, run the following command in the root folder of your GIT repository:

travis setup releases

This will setup a basic deployment section to your .travis.yml file. Do not forget to add the skip_cleanup: true, otherwise the build artifacts you want to release got deleted before the deployment step. In our case, a release is indicated with a tag that follows the naming convention release.x.x.x and we only want those tags to be automatically published, so we added a condition to the deployment section.

... 
 deploy:  
  skip_cleanup: true  
  provider: releases  
  api_key:   
   secure: BSEYtMYXInrXum0eO........  
  file: releng/org.yakindu.sct.repository/target/updatesite.zip  
  on:  
   tags: Yakindu/statecharts  
   condition: "$TRAVIS_TAG =~ ^release.*$"  

Now, every time Travis indicates a release tag, it will automatically publish the release to GitHub in case of a successful build.

Conclusion: Travis provides an incredibly powerful and customizable infrastructure to open source projects for free. Thank you very much for that and keep up the good work!


Mittwoch, 4. März 2015

SCT - Seamless Integration with C and Java type systems


Yakindu Statechart Tools is shipped with a powerful expression language to model the reactive behavior of a state machine. It is a statically typed programming language that is familiar to everyone who knows C or Java. It provides all basic expressions  like logical expressions,arithmetic expressions, bitwise arithmetic and bit shifting.

The language is built on a default, platform-independent type system that supports the primitive types Integer, Real, Boolean, String and Void out of the box. During code generation, these built in types are then mapped to reasonable platform specific types. For example, a String is mapped to a java.lang.String in Java and a char * in C/C++.

However, this platform-independence comes at a price: You can't leverage all the captabilities of the underlying technologie in your state machine and you have to write glue code to integrate the generated code with your platform. These issues are well illustrated when it comes to the definition of operations. In SCT, it is possible to define operations, which can be called from an expression:


operation max(int1: integer, int2: integer): integer 
entry / myVar = max(1,2) 

The Java code generator then generate interfaces for all defined operations that must be implemented by the user. For example, the Java generator creates an interface with a corresponding java method signature. This interface has to be implemented by the client and passed to the statemachine, which is shown in the following code snippet:

 
DefaultSMStatemachine statemachine = new DefaultSMStatemachine(); 
 
SCInterfaceOperationCallback callback = new SCInterfaceOperationCallback() { 
 @Override 
 public int max(int param1, int param2) { 
  return Math.max(param1, param2) // Delegate the call to the Java Lib
 } 
};
statemachine.getSCInterface().setSCInterfaceOperationCallback(callback);

 
But what if you want to develop a statemachine for the Java platform only? This is where Statechart Domains come into play. Select the Java Domain in the 'New Model Wizard' and get access to Java native types. For example, import java.util.Math in your statechart and access all methods like max, min or random directly from the statechart model!


Statechart Tools for Java

This integration is illustrated by the following example, a simple statechart that receives events with a value and puts them onto a Stack for later processing.

in event inEvent : Message
var stack : java.util.Stack<Message>

Note the definition of the event inEvent. The special feature about this is that the type of the event is the complex type Message - a Plain Old Java Object defined in a local Java Project containing a timestamp of type java.util.Date and a message of type String. Those fields can be accessed as properties from the statechart.



The variable stack is of type java.util.Stack with the type argument Message - the Java Type Integration even supports generic types! All (public) methods provided by java.util.Stack can be accessed directly from the Statechart, as you can see in the State MessageReceived when the currentMessage is pushed onto the stack. With Statechart Tools for Java you can develop your state machine in an object-oriented way like you are used to it from Java!


Statechart Tools for Embedded C

When creating a new statemachine model for the C Domain, it is possible to import C-header files and get access to all of the declared functions.The following example is a simplyfied version of a header from our Yakindu Smart Car.

After importing the C-Header (import: car_api.h), all defined complex types like Structs, Unions and Enumerations as well as TypeDefs and Functions become accessible directly from the Statechart model. The generated code integrates well with the imported headers so there is no need to write a bunch of glue code anymore!

Current implementation state

This is work in progress and still a lot of work to do. Although it is already possible to edit C and Java specific statecharts and generate code out of it, we still have to adopt the simulation engine to handle complex types. We expect to have a first version available in Q3/2015.



Freitag, 31. Oktober 2014

Yakindu Statechart Tools 2.4.0 released!

We released Yakindu Statechart Tools 2.4.0 today! It contains a lot of bugfixes and improvements as well as new features.

Installation
This release requires Eclipse Luna. You can install SCT 2.4.0 from our update site:
http://updates.yakindu.org/sct/luna/releases/ or you can download a full Eclipse zip package from our dowload page.

New and Noteworthy

Here is a summary of the improvements worth mentioning:

1. Toggle documentation toolbar entry added
Menu-16We added a toolbar menu item  to the editors toolbar to toggle between documentation and formal expression language. This allows to toggle modes for the whole diagram.

2. C/C ++ generator feature for inner class function visibility added
innerFunctionVisibility (String, optional): This parameter is used to change the visibility of inner functions and variables. In default the „private” visibility is used. It can be changed to „protected” to allow function overriding for a class which inherits from the generated statemachine base class.
Example configuration:

feature GeneratorOptions {
 innerFunctionVisibility =  "protected"
}

3. C/C++ generator feature to change operation callback generation behavior added
staticOperationCallback (Boolean, optional): If set to ‚true’ the callback function declaration for statechart operations is static and the functions are called by the statemachine code statically.
Example configuration:
 
feature GeneratorOptions {
 staticOperationCallback = true
}

4. Model and Diagram compare feature
This release contains the beta version of a diff/merge viewer for statechart diagrams. It integrated seamlessly into the Eclipse Team API to allow diffing and merging of different revisions as well as comparing models with the local history.

compare_example

Note that this feature is still beta. There are currently two known problems (this and this) with the editor that will be fixed for the next release.

Bugfixes

We also fixed a bunch of bugs reported via our User Group:

Toggle subregion alignment does not work
Transition into substate does not recognize parents history context
Model can not be simulated if operations in named interfaces are used
C++ generator can generates allways actions with a lower-case letter in source file
CoreFunction methods for long types incomplete
Operations of other Statecharts (in the same folder) are in scope
Class cast exception if java provided custom operation is not executable
Simulating Operations With Custom Java Code is not working as intended
Diagram corrurpted when moving transition label

Thanks to all bug reporters!