[Fun] The MySphero Project
Last update: 19-Jan-2015
In Summer 2014 I finally got hold of a developer version of Myo, an armband that tracks for a few of your hand poses and your arm orientation [Link]. It's pretty neat and best of all, the company (Thalmics Labs) releases an SDK (Software Development Kit) so one can make their own applications by tapping into the signal sent from the armband. The SDK supports Windows, OSX, Android and iOS, some of the most popular computing platforms.
A few years back, another interesting product was released, called Sphero, a baseball-sized ball with the parts of an R/C car inside [Link]. By controlling that "car" inside one essentially rolls the ball in any desired direction. Again, the company released an SDK supporting a similar set of computing platforms.
In Summer 2014 I finally got hold of a developer version of Myo, an armband that tracks for a few of your hand poses and your arm orientation [Link]. It's pretty neat and best of all, the company (Thalmics Labs) releases an SDK (Software Development Kit) so one can make their own applications by tapping into the signal sent from the armband. The SDK supports Windows, OSX, Android and iOS, some of the most popular computing platforms.
A few years back, another interesting product was released, called Sphero, a baseball-sized ball with the parts of an R/C car inside [Link]. By controlling that "car" inside one essentially rolls the ball in any desired direction. Again, the company released an SDK supporting a similar set of computing platforms.
As a demonstration of using the Myo as a controller, a software engineer from the Thalmics Lab developed a "Myo-Sphero" application [Link]. However, this article mainly focused on the iterative design process for the control mechanism, rather than the application itself. The application itself wasn't available, nor its code.
Of course this wasn't something very difficult to achieve, afterall the SDK for both devices were released to the public. However, at the time of this post there wasn't any re-creation of the described application. The closest I found was a node.js script running off a computer (Myo-PC-Sphero). This is why I decided to write my own Android app, and here is a video of it in action (see if you can find the phone in the video :D).
Of course this wasn't something very difficult to achieve, afterall the SDK for both devices were released to the public. However, at the time of this post there wasn't any re-creation of the described application. The closest I found was a node.js script running off a computer (Myo-PC-Sphero). This is why I decided to write my own Android app, and here is a video of it in action (see if you can find the phone in the video :D).
The Android app was coded over a few evenings, and to save time I used the sample code from both SDKs. The way the app works is like the following:
- connect to a bluetooth-paired Sphero, using the provided discover UI
- connect to a bluetooth-paired Myo, using the provided discover UI
- map the data received from Myo (e.g., orientation) to Sphero's movement (e.g. direction), and relay it to Sphero
Hurdles (and how to overcome them)
- The first one I got was even though both SDKs come in a nice zip file and ready importable to Ellipse, using them in your own project is not as trivial as importing a few libraries. For example, the heart of the Myo SDK is called "MyoSdk" and it has to be referenced as a library. If you create a new project (or copy one of the sample projects) elsewhere you'll have to manually set it (Properties > Android > Library > Add..). Another thing about copying a sample project is you also copy the package name, and changing it results in all sorts of "Resource not found" problems (Manifest > package).
- As Myo only runs on Android 4.3+ with Bluetooth 4.0LE, it requires sdk-18 or above, which is higher than that of the Sphero. Make sure you set the Manifest and Project.property properly.
- I was using the UISample project from Sphero, and the way it shows different screens as views is very confusing, especially when the app goes to various lifecycles. By itself it works ok, but when combined with the Myo discovery code it starts to behave badly. On top of making the Myo discovery invisible I had to hide the Sphero's discovery UI and make sure it won't look for another Sphero again. This is done by calling
mSpheroConnectionView.setVisibility(View.GONE); mSpheroConnectionView.clearListeners();
after the invisible Myo discovery (Hub.getInstance().attachToAdjacentMyo();).
How to control
The ways to control is pretty much the same as what was mentioned in the My-Sphero article: Pitch for forward-backward, Roll for left-right, Keep fist for motion. And I skipped the Yaw for offset as I'll mostly be stationary. I also skipped the speed calculation based on the values of Pitch and Roll. You can add them back if you'd like to.
One thing that I don't understand, is how the author normalized the Roll. Based on the documentation of Myo, Roll ranges from -180 to 180 (by the way, it's not mentioned in the latest SDK... what's up with that??), but if you think about it a normal human being cannot turn their arm for a full 180 degrees. In fact, as seen in the video, you have to swing your arm to actually perform the Roll, as simply twisting your wrist won't rotate the armband at all. The author did mention about setting an active zone, but my guess is he also clipped the Roll to -90 to 90 before normalizing.
Overall an interesting pass-time for me to combine these 2 gadgets together, and I begin to realize the latency of using Bluetooth... so it might not be the best way to control things in a timely manner afterall (the bottleneck might be the design of control, or the Sphero... warrants further investigation).
The ways to control is pretty much the same as what was mentioned in the My-Sphero article: Pitch for forward-backward, Roll for left-right, Keep fist for motion. And I skipped the Yaw for offset as I'll mostly be stationary. I also skipped the speed calculation based on the values of Pitch and Roll. You can add them back if you'd like to.
One thing that I don't understand, is how the author normalized the Roll. Based on the documentation of Myo, Roll ranges from -180 to 180 (by the way, it's not mentioned in the latest SDK... what's up with that??), but if you think about it a normal human being cannot turn their arm for a full 180 degrees. In fact, as seen in the video, you have to swing your arm to actually perform the Roll, as simply twisting your wrist won't rotate the armband at all. The author did mention about setting an active zone, but my guess is he also clipped the Roll to -90 to 90 before normalizing.
Overall an interesting pass-time for me to combine these 2 gadgets together, and I begin to realize the latency of using Bluetooth... so it might not be the best way to control things in a timely manner afterall (the bottleneck might be the design of control, or the Sphero... warrants further investigation).
What's next?
I'd probably want to first clean up the code coming from the UISample project. There are too many views and I think replacing them with fragments will make things a bit easier to handle. The discovery process of both devices also need to be improved, and I think having 2 dedicated menu items (or at least 2 buttons) providing a more user-friendly interface to connect would be great.
The control is also quite sluggish. I'm not sure if it's the latency of the libraries, Bluetooth, or the code. But the Sphero could certainly be more responsive. Also the way to control is still awkward... to swing my arm just to turn seems to much work. A Yaw+Fist to go forward and Yaw+Fingers_spread might be a better option.
I'd probably want to first clean up the code coming from the UISample project. There are too many views and I think replacing them with fragments will make things a bit easier to handle. The discovery process of both devices also need to be improved, and I think having 2 dedicated menu items (or at least 2 buttons) providing a more user-friendly interface to connect would be great.
The control is also quite sluggish. I'm not sure if it's the latency of the libraries, Bluetooth, or the code. But the Sphero could certainly be more responsive. Also the way to control is still awkward... to swing my arm just to turn seems to much work. A Yaw+Fist to go forward and Yaw+Fingers_spread might be a better option.
The code
Ah the fun part. As mentioned to save space and let you have some development thrill, I'm just attaching the code block handling the Myo events. Start off with the UISample project from Sphero SDK, then follow the instructions in the Myo SDK reference page to set up the Myo (Hub). Finally, use this as the mListener object instead of the sample. Pay special attention to the Hurdles section above.
Ah the fun part. As mentioned to save space and let you have some development thrill, I'm just attaching the code block handling the Myo events. Start off with the UISample project from Sphero SDK, then follow the instructions in the Myo SDK reference page to set up the Myo (Hub). Finally, use this as the mListener object instead of the sample. Pay special attention to the Hurdles section above.
private boolean isMyoControlling = false;
private boolean isMyoConnected = false;
/** Listener class listening and responding to Myo events, use this in place of that used in the HelloWorld sample project **/
private DeviceListener mListener = new AbstractDeviceListener() {
//called whenever a Myo has been connected, set a button to green to indicate successful connection
@Override
public void onConnect(Myo myo, long timestamp) {
Button myoButton = (Button)findViewById(R.id.myo_button);
myoButton.setBackgroundColor(Color.GREEN);
isMyoConnected = true;
}
//called whenever a Myo has been disconnected, set a button to red to indicate disconnect
@Override
public void onDisconnect(Myo myo, long timestamp) {
Button myoButton = (Button)findViewById(R.id.myo_button);
myoButton.setBackgroundColor(Color.RED);
isMyoConnected = false;
}
//called whenever Myo has recognized a sync gesture
@Override
public void onArmSync(Myo myo, long timestamp, Arm arm, XDirection xDirection) { }
//called whenever Myo has detected itself being moved
@Override
public void onArmUnsync(Myo myo, long timestamp) { }
//called whenever a synced Myo has been unlocked. Under the standard locking
// policy, that means poses will now be delivered to the listener.
@Override
public void onUnlock(Myo myo, long timestamp) { }
//called whenever a synced Myo has been locked. Under the standard locking
// policy, that means poses will no longer be delivered to the listener.
@Override
public void onLock(Myo myo, long timestamp) { }
//called whenever a Myo provides its current orientation, represented as a quaternion.
@Override
public void onOrientationData(Myo myo, long timestamp, Quaternion rotation) {
// Calculate Euler angles (roll, pitch, and yaw) from the quaternion.
float roll = (float) Math.toDegrees(Quaternion.roll(rotation));
float pitch = (float) Math.toDegrees(Quaternion.pitch(rotation));
float yaw = (float) Math.toDegrees(Quaternion.yaw(rotation));
// Adjust roll and pitch for the orientation of the Myo on the arm.
if (myo.getXDirection() == XDirection.TOWARD_ELBOW) {
roll *= -1;
pitch *= -1;
}
// Calculate the angle (and optional speed) of the Sphero, referenced as mRobot
if(isMyoControlling && (pitch>5||pitch<-5)) {
float yAxis, xAxis; //the 2 parameters for the angle and speed
if(pitch<0) yAxis = -1*(float)Math.pow(pitch/90, 2);
else yAxis = (float)Math.pow(pitch/90, 2);
if(roll<0) xAxis = -1*(float)Math.pow(roll/180, 2);
else xAxis = (float)Math.pow(roll/180, 2);
float heading = (float)Math.toDegrees(Math.atan2(yAxis, xAxis));
float headingReal;
if(90-heading<0) headingReal = 90-heading+360;
else headingReal = 90-heading;
mRobot.drive(headingReal, 0.8f);
} else {
mRobot.stop();
}
}
//called whenever a Myo provides a new pose.
@Override
public void onPose(Myo myo, long timestamp, Pose pose) {
// Handle the cases of the Pose enumeration, and change the text of the text view
// based on the pose we receive. Right now just the FIST will do something. Add others as desired
switch (pose) {
case UNKNOWN:
break;
case REST:
isMyoControlling = false;
break;
case DOUBLE_TAP:
break;
case FIST:
isMyoControlling = true;
break;
case WAVE_IN:
break;
case WAVE_OUT:
break;
case FINGERS_SPREAD:
break;
}
/* shouldn't be required if the unlock policy is set to NONE
if (pose != Pose.UNKNOWN && pose != Pose.REST) {
// Tell the Myo to stay unlocked until told otherwise. We do that here so you can
// hold the poses without the Myo becoming locked.
myo.unlock(Myo.UnlockType.HOLD);
// Notify the Myo that the pose has resulted in an action, in this case changing
// the text on the screen. The Myo will vibrate.
myo.notifyUserAction();
} else {
// Tell the Myo to stay unlocked only for a short period. This allows the Myo to
// stay unlocked while poses are being performed, but lock after inactivity.
myo.unlock(Myo.UnlockType.TIMED);
}*/
}
};
/** more code **/
/** The button-click handler for discovering a nearby Myo **/
public void onMyoClick(View v) {
if(isMyoConnected) return; //skip the rest if a Myo is already connected to
Hub.getInstance().attachToAdjacentMyo();
//turn off the mSpheroConnectionView
mSpheroConnectionView.setVisibility(View.GONE);
mSpheroConnectionView.clearListeners();
}