/ README.md
README.md
1 # Programming the Meccanoid using the Arduino, Kinect SDK, and Visual Studio 2 3 <p align="center"> 4 <img src="https://github.com/lnmangione/meccanoid-imitate/blob/master/gifs/neck_right.gif"> 5 <img src="https://github.com/lnmangione/meccanoid-imitate/blob/master/gifs/arm_left.gif"> 6 <img src="https://github.com/lnmangione/meccanoid-imitate/blob/master/gifs/neck_and_arm.gif"> 7 </p> 8 9 ## Quick Intro 10 11 This guide outlines the process of interfacing the Meccanoid robot with Microsoft Kinect. The guide can be 12 used, however, simply for interfacing the Kinect with the Arduino, or the Arduino with the Meccanoid, or the 13 USB2Serial with the Arduino. The guide is split into clearly defined sections so that you can easily access just 14 what you need. 15 16 Interfacing the Kinect with the Meccanoid is a very doable process, but without a decent guide can take 17 weeks to figure out. There is no tutorial that goes through the entire process. There are tutorials online that 18 help with certain parts of the process, but most are difficult to find and many of them outdated. So that’s why 19 we made this. Note, this guide **requires a device running Windows.** 20 21 ## Using the Kinect Software 22 23 ### Step 1: Install the Kinect SDK 24 25 The version of the Kinect for Windows SDK that you install will depend on your computer’s operating system 26 and the type of Kinect sensor you are using. You should download the most recent (viable) version of the 27 Kinect SDK. 28 29 Note: If you are using the Kinect for Windows v2 or Kinect for Xbox One you MUST download the Kinect SDK 30 2+. Additionally, the Kinect SDK 2+ only supports Windows 8+. 31 32 Download SDK 1.8: https://www.microsoft.com/en-us/download/details.aspx?id= 33 34 Download SDK 2.0: https://www.microsoft.com/en-us/download/details.aspx?id= 35 36 ### Step 2: Install Microsoft Visual Studio 37 38 This is the IDE you will need to utilize the Kinect Software. You can download Visual Studio Community 2015 39 here: https://www.visualstudio.com/en-us/visual-studio-homepage-vs.aspx. 40 41 In case you already have another version of VS installed on your computer: Other versions of VS should also 42 work, but some are not supported depending on which version of the Kinect SDK you are using. For example, 43 SDK 2.0 requires VS 2012 or newer. Ensure your versions are compatible. 44 45 46 ### Step 3: Install the Kinect Drivers 47 48 Power the Kinect and plug it into your computer’s USB port. Your computer should automatically install the 49 Kinect drivers. Afterwards, check the light on the Kinect’s power-box. An orange light indicates a problem. 50 Otherwise, your Kinect is ready to go. 51 52 ### Step 4: Getting Started with Kinect Programming 53 54 Our guide uses C# within Visual Studio to utilize the Kinect software and create Windows applications. If you 55 are unfamiliar with C#, this website provides a quick but comprehensive reference: 56 https://learnxinyminutes.com/docs/csharp/ 57 58 This tutorial goes through the process of creating a basic WPF application with Kinect functionality: 59 [http://www.egr.msu.edu/classes/ece480/capstone/spring15/group02/assets/docs/dzappnote.pdf](http://www.egr.msu.edu/classes/ece480/capstone/spring15/group02/assets/docs/dzappnote.pdf) 60 61 Note that “reader” must be declared as a “DepthFrameReader” not a “MultiSourceFrameReader.” 62 63 ### Step 5: Advanced Use of the Kinect SDK 64 65 For more advanced usage of the Kinect SDK, see the documentation and programming examples that 66 Microsoft provides here: https://msdn.microsoft.com/en-us/library/dn799271.aspx 67 68 ## Using the Arduino to Control the Meccanoid 69 70 ### Step 1: Install the Meccanoid Arduino Library 71 72 Thankfully, Meccano provides a library that allows the Arduino to control the Meccanoid hardware. Download 73 it here: http://www.meccano.com/meccanoid-opensource 74 75 To install the library, open the Arduino IDE. Go to Sketch-->Include Library-->Add a .ZIP Library. Select the .zip 76 file you downloaded. 77 78 ### Step 2: Understanding the Hardware 79 80 For a brief understanding of the Meccanoid hardware, including smart servos and LED’s, see here: 81 [http://cdn.meccano.com/open-source/Meccano_SmartModuleProtocols_2015.pdf](http://cdn.meccano.com/open-source/Meccano_SmartModuleProtocols_2015.pdf) 82 83 NOTE: The smart servos are EXTREMELY BUGGY when daisy chained. Sometimes servos in the same chain 84 will “link” together. In our case for example, the 1st and 2nd servos on the left arm linked together, both 85 moving in response to data sent to the 1st servo. The 3rd servo would respond to data being sent to the 2nd 86 servo. We never determined the cause of this issue. You can, however, work around it by simply isolating the 87 servo that causes issues. In our case, we removed the 2nd servo from the chain, and connected it to the 88 Arduino as its own chain. 89 90 91 One last pro tip: The Arduino powered through the computer sometimes isn’t strong enough to lift the 92 Meccanoid’s arms. Powering the Arduino with an external battery (we used a 7.4V) can solve this issue. 93 94 ### Step 3: Programming the Meccanoid 95 96 The Meccanoid Open Source Programming page (the page from Step 1 with the Arduino library) also contains 97 an example project. Download it. The “Claw_Bot.ino” file provides a great example of how to use the library. 98 99 For further information on using the Arduino with the Meccanoid, see this post on the Arduino forum: 100 https://forum.arduino.cc/index.php?topic=368782.0 101 102 It’s not much, but it’s just about the only information out there. 103 104 ## Communicating between Arduino and Visual Studio 105 106 ### Step 1: Basic Serial Communication between Arduino and VS 107 108 This tutorial goes through the process of setting up a WPF application for basic serial communication with the 109 Arduino IDE: 110 [http://www.c-sharpcorner.com/UploadFile/e46423/arduino-control-using-a-windows-presentation-foundation-(](http://www.c-sharpcorner.com/UploadFile/e46423/arduino-control-using-a-windows-presentation-foundation-() 111 wpf/ 112 113 You can skip the part about using an on/off switch within VS, and just pre-program the WPF application to 114 send either ‘0’ or ‘1’. 115 116 ### Step 2: Advanced Serial Communication 117 118 The tutorial from Step 1 provides a great overview of setting up serial communication between the Arduino 119 and VS, but things get a little more complicated when you’re reading more than just a single char, which you 120 will likely be doing if you’re sending substantial amounts of data to the Arduino. 121 122 Read(), readString(), and readStringUntil() are all functions that Serial uses on the Arduino. They are very 123 different functions and it’s important to understand what they do if you plan on using them. Serial 124 communication is not as simple as you might assume. This blog provides a decent explanation of how it 125 works: https://hackingmajenkoblog.wordpress.com/2016/02/01/reading-serial-on-the-arduino/ 126 127 ### Extra: Potential Issues 128 129 We suffered extreme latency issues (of around 3-4 seconds) while communication between VS and the 130 Arduino. We fixed this issue by only sending data from VS every 10 frames. Additionally, we created a timer to 131 clear the serial port out buffer every second, however, we did not determine whether this timer was needed. 132 It may not be necessary. 133 134 135 ## Using the USB2Serial for Debugging 136 137 ### Purpose 138 139 A serial port on the Arduino cannot simultaneously read and write. So, if the serial port is receiving data from 140 VS for example, it cannot write to the serial monitor. This makes debugging very difficult, as print statements 141 become impossible. Using the USB2Serial, though, a second serial port can be opened for debugging 142 purposes. 143 144 ### Hardware Setup 145 146 Plug the RX of the USB2Serial into a TX (this CANNOT be TX0, use TX1+) on the Arduino. Plug the TX into the 147 corresponding RX. As an example, TX/RX 1 on an Arduino Mega are pins 18/19. 148 149 That’s it. The 5V, ground, and reset on the USB2Serial need not be used. 150 151 ### Software Setup 152 153 Now, you can program the serial port you used just like you normally program “Serial”. If you used TX/RX 1, for 154 example, you simply program using “Serial1”. 155 156 In order to change the output of the serial monitor to the proper serial port, go to Tools --> Port. Select the 157 port to which USB2Serial is connected. You can only do this AFTER uploading your sketch to the Arduino. 158 159 Created on 8/26/16 by Luigi Mangione 160 Questions? Contact me at lnmangione@gmail.com