Developing the Internet of Things: Myo-ity Report

Posted by on in Tutorial

Myoity-Report-SN-leadin

 

In todays post we will dive a little deeper into the exploration of gesture control as an emerging form of input. Learn how the Myo Gesture Control Armband from Thalmic Labs can be used with Appmethod to create new user experiences with traditional UI/UX components. Specifically, we will take a look at FireFlow, a photo gallery scroller that ships as a demo with Appmethod.

 

FireFlow, a Demo Photo Gallery

FireFlow is a sample demo that illustrates how to create a photo gallery with FireMonkey. FireFlow is bundled with the Appmethod download as a demo app. For more details on FireFlow please reference the Appmethod Docwiki article.

FireFlow + Thalmic Labs Myo Gesture Control Armband = Myo-ity Report!

Video not found or Youtube service not available
 

Today we all take one step closer to living out the Minority Report gesture-based user interface scene from 2002.  
 
The Myo-ity Report demo app is an example of extending existing UI/UX controls to create new user experiences with gesture input.  
 
Lets dive right in and examine how it works.

Appmethod and the Myo SDK

The first step is to load the Myo SDK into Appmethod.

If you have not yet integrated the Myo SDK with Appmethod, please refer to my previous post fore more deatils. The first half of following video will step you through integrating the SDK with Appmethod.

Video not found or Youtube service not available
 
 

 

Building Myo-ity Report

First, make a copy of the FileFlow demo project from the following location:

Start | Programs | Appmethod 1.15 | Samples -> CPP\FireMonkey Desktop\FireFlow  

or download directly from sourceforge.

New Class Properties

The following properties have been added to the datacollector class. scroll_left and scroll_right are flags which track the direction to scroll the photos. The set of const static ints determine the offsets from the calculated relative center position that represents the upper and lower bounds (left and right movement) for the Myo's positional location. The SCROLL_AMOUNT_*  refers to how fast the animation timer fires which controls how quickly the photo gallery will scroll.

yaw_w_ref represents the inital angle the Myo Armband starts at. yaw_w_rel is the calculated offset from that refrenced position.

1
2
3
4
5
6
7
8
9
     
int roll_w, pitch_w, yaw_w, yaw_w_ref, yaw_w_rel;
bool onArm, scroll_left, scroll_right;
const static int    SCROLL_LEFT_FAST = -4,
                        SCROLL_LEFT_SLOW = -2,
                        SCROLL_RIGHT_FAST = 3,
                        SCROLL_RIGHT_SLOW = 2,
                        SCROLL_AMOUNT_FAST = 250,
                        SCROLL_AMOUNT_SLOW = 750;

 

The DataCollector initialization list has been updated too:

1
2
3
4
5
DataCollector()
    : onArm(false), roll_w(0), pitch_w(0), yaw_w(0), currentPose(),
    scroll_left(false), scroll_right(false),  yaw_w_ref( -1 )
    {
    }

Processing Gesture Input

Within the datacollector class (please reference my previous Myo post for more detail) I have added a new method for the DataCollector class, read().

Firstly, It calculates the relative orientation of the Myo in regards to its start position. If the thumbToPinky or fingerSpread fists are detected then a new refrence orientation point is calculated against the current arm position in space. Doing this allows the wearer of the Myo to stand at pretty much any angle with respect to the Bluetooth USB dongle.

If the clenched fist is detected then the currently selected photo gets scaled out by a factor of 2 on X and Y and no further input is processed. This allows the person to 'grab' onto the current item and keep it selected regardless of any arm movements that may occur. This is an example of having to define new solutions for UX/UI experiences that are specific to gesture input.

Lastly, the relative yaw positional data is checked against upper and lower bounds which results in scrolling the images to the left and right.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
void read()
{
    TLayer3D *Cover = (TLayer3D *)FrmMain->Coverflow->Children->Items[ Round(FrmMain->TrackBar1->Value) ];
    if ( onArm ) {
        //- clear scrolling flags
        scroll_left  = false;
        scroll_right = false;
        //- check for current fist
        std::string poseString = currentPose.toString();
        //- recalibrate
        if ( poseString == "thumbToPinky" || poseString == "fingerSpread" ) {
            yaw_w_ref = yaw_w;
        }
        //- zoom selection on fist
        if ( poseString == "fist" ) {
           Cover->Scale->X = 2;
           Cover->Scale->Y = 2;
           //- the fist is being made, dont proccess movement
           return;
        }
        else {
           Cover->Scale->X = 1;
           Cover->Scale->Y = 1;
        }
        //- set refrence point
        if ( yaw_w_ref == -1 ) {
            yaw_w_ref = yaw_w;
        }
        //- set relative position of myo armband
        yaw_w_rel = yaw_w_ref - yaw_w;
        if ( yaw_w_rel <= SCROLL_LEFT_SLOW )
            scroll_right = true;
        else if ( yaw_w_rel >= SCROLL_RIGHT_SLOW  )
            scroll_left = true;
        if ( yaw_w_rel <= SCROLL_LEFT_FAST )
            FrmMain->Timer2->Interval = SCROLL_AMOUNT_FAST;
        else if ( yaw_w_rel <= SCROLL_LEFT_SLOW )
            FrmMain->Timer2->Interval = SCROLL_AMOUNT_SLOW;
        else if ( yaw_w_rel >= SCROLL_RIGHT_FAST )
            FrmMain->Timer2->Interval = SCROLL_AMOUNT_FAST;
        else if ( yaw_w_rel >= SCROLL_RIGHT_SLOW )
            FrmMain->Timer2->Interval = SCROLL_AMOUNT_SLOW;
    }
}

Adding Time

From the Tool Palette add two TTimer and a TMemo to the Form Designer

The first timer is set to a fast interval of 10ms. This timer is responsible for polling the Myo device for new data and outputting debug info to the Memo.  

The second timer is responsible for animating the photo gallery.  

Add the following code to their Timer Event handlers:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
void __fastcall TFrmMain::Timer1Timer(TObject *Sender)
{
    hub.run( 1000/50 );
    collector.read();
    Memo1->Lines->Clear();
    Memo1->Lines->Add( String().sprintf( L"scroll_left: %d",collector.scroll_left ) );
    Memo1->Lines->Add( String().sprintf( L"scroll_right: %d",collector.scroll_right ) );
    Memo1->Lines->Add( String().sprintf( L"yaw_w: %d", collector.yaw_w_ref - collector.yaw_w ) );
    Memo1->Lines->Add( String().sprintf( L"timer speed: %d", Timer2->Interval ) );
    std::string poseString = collector.currentPose.toString();
    Memo1->Lines->Add( poseString.c_str() );
}
//---------------------------------------------------------------------------
void __fastcall TFrmMain::Timer2Timer(TObject *Sender)
{
    if ( collector.scroll_left) {
        FrmMain->TrackBar1->Value--;
    }
    if ( collector.scroll_right ) {
        FrmMain->TrackBar1->Value++;
    }
}

 

 

Tags: Appmethod

Prev
Next

About
Gold User, Rank: 8, Points: 399
Brian Alexakis is a Product Marketing Manager at Embarcadero Technologies. He is focused on leveraging the connected world of technology to build new experiences for the Internet of Things.
http://embarcadero.com

Comments

Check out more tips and tricks in this development video: