Unity 3D Tutorials Augmented Reality using Unity 3D and FaceAPI
By Sriram.A.S. Date: 3rd Dec, 2010 sriramdasty7@gmail.com Originally Posted in: http://mypersonalsoft.blogspot.com/ (Keep checking the blog for updates)
About this Tutorial This tutorial is about implementing Augmented/Virtual Reality using Unity 3D and FaceAPI. Demo video about this tutorial is posted in YouTube. http://www.youtube.com/watch?v=yN_YEnDs2uk
Requirements Unity 3D (http://unity3d.com) FaceAPI Non Commercial Edition (http://www.seeingmachines.com/product/faceapi) 6dofstreamer (http://code.google.com/p/6dofstreamer) Download faceapistreamer___.zip file only The main script (as Unity Package) http://megaupload.com/?d=YKX2B382 A little knowledge of C# and Sockets is absolutely required Tutorials 1. Install the FaceAPI Non Commercial Edition, file named setup_FaceTrackingAPI_NC___.exe 2. Extract the 6dofstreamer zip file, something named faceapistreamerxxx.zip
3. Open Unity 3D and create a new project (FileNew Project) 4. Import the file by AssetsImport Package 5. This is what you see after that
As you see that, a new scene s1 is added and script ‘script’ is also added 6. Click the ‘camera’ element and you can notice that the script ‘script’, which is a C# script is attached to this camera element
7. Now that is all, we are going to open the script and understand that. The script may look a bit fuzzy, but believe me its very easy. Open the script ‘script’ (more preciously script.cs 8. This is the script and its explanation: using UnityEngine; using System.Collections; using System; using System.Net; using System.Net.Sockets; using System.Text;
public class script: MonoBehaviour { //This is the port that 6dofstreamer sends the face details. They use UDP //sockets. The string output from the streamer may look like //A B C D E F G //I donâ&#x20AC;&#x2122;t remember the format order much, check its documentation. But //it just sends three rotations and three translations in three axes //It is to be noted that we are interested in 4th and 5th words which //describes one rotation and one translation private const int listenPort=29129; UdpClient listener = new UdpClient(listenPort); IPEndPoint groupEP = new IPEndPoint(IPAddress.Any, listenPort); String op; float pz,ry,ry_old=0; Vector3 newpos; //Note the update function void Update () { byte[] receive_byte_array= listener.Receive(ref groupEP); //op contains the string received by the 6dofstreamer //Live streaming is done, so each time when the frame reloads, we want //our update function to get the fresh string from the streamer op=Encoding.ASCII.GetString(receive_byte_array, 0, receive_byte_array.Length); //We form an array by splitting the string string[] words=op.Split(' '); //We are in particular about the 4th and 5th as I told before //Now we work with a bit of mathematics //ry for rotation y and pz for translation x pz=(float)Convert.ToDouble(words[3]); ry=(float)Convert.ToDouble(words[4]); //This is my simple mathematical model, for much more smoother transition, I recommend you to use your own mathematical mapping of //the STRING FROM THE STREAMER TO ANY CHANGE IN ORIENTATION OF //OUR OBJECT if(pz<=-0.5 && pz>-0.4) pz=-50; else if(pz<=-0.4 && pz>-0.3) pz=-40; else if(pz<=-0.3 && pz>-0.2) pz=-30;
else if(pz<=-0.2 && pz>-0.1) pz=-20; else if(pz<=-0.1 && pz>0) pz=-10; else if(pz<=-0 && pz>=0) pz=0; else if(pz>0 && pz<=0.1) pz=10; else if(pz>0.1 && pz<=0.2) pz=20; else if(pz>0.2 && pz<=0.3) pz=30; else if(pz>0.3 && pz<=0.4) pz=40; else if(pz>0.4 && pz<=0.5) pz=50; pz=-pz; if(ry<=-0.5 && ry>-0.4) ry=-50; else if(ry<=-0.4 && ry>-0.3) ry=-40; else if(ry<=-0.3 && ry>-0.2) ry=-30; else if(ry<=-0.2 && ry>-0.1) ry=-20; else if(ry<=-0.1 && ry>0) ry=-10; else if(ry<=-0 && ry>=0) ry=0; else if(ry>0 && ry<=0.1) ry=10; else if(ry>0.1 && ry<=0.2) ry=20; else if(ry>0.2 && ry<=0.3) ry=30; else if(ry>0.3 && ry<=0.4) ry=40; else if(ry>0.4 && ry<=0.5) ry=50; if(ry_old!=0)
{ newpos.y=ry_old-ry; } ry_old=ry; //Thus the calculated new rotation and position from the streamer is //mapped to our game domain by some mathematics and then applied to //the game object transform.Rotate(newpos); //As you see only z position is affected gameObject.transform.position = new Vector3( gameObject.transform.position.x, gameObject.transform.position.y,pz); } }
9. Now just make sure that the streamer is running, and your face is detected, and Build and Run the game
You can notice that when you move back or forth or turn your neck right or left, the game camera position and angle changes corresponding.
Sorry friends, I was in a hurry, so couldnâ&#x20AC;&#x2122;t explain it in more detail. But still I think the script that I have released is enough so that you can interface this cool feature with your games.
Note If any hyperlink of the files that I have mentioned in this file is now working, please inform me to my email sriramdasty7@gmail.com or better you can post it in my blog http://mypersonalsoft.blogspot.com Especially the unity package which I have attached, I donâ&#x20AC;&#x2122;t have a web space of my own, so I rely on free file hosting. If somebody can host this file in your domain, I would be happy. And also, please keep checking my blog for updates.
Any more help, clarification etc., you can email me
!!ENJOY VIRTUAL REALITY!!