Setup Instructions

Set up Oculus headset

To be able to deploy apps from Unity, we need to set the headset into developer mode. This can only be done via the phone app.

  1. Download the Oculus app on your phone.
  2. The Meta account signed in as admin on the Oculus and the Meta account used in the app needs to be the same. If you are not signed in as admin on the oculus headset, you need to reset the device by:
    1. Power off the device
    2. Power on while holding volume (-)
    3. Choose factory reset
  3. Connect to your Oculus from your phone by choosing “pair your headset” in the app.
  4. Join or create a Meta dev organization to get developer rights for you Meta account.
  5. Select your meta quest device in the app -> Headset settings -> Developer Mode -> on
    6. Oculus developer mode
  6. Restart headset
  7. In the headset -> go to settings -> system -> developer -> USB connection dialog
    8. USB connectiont
  8. Now connect USB-c cable to headset and pc
  9. You will see the following dialog pop up-> press allow
    10. USB allow

Set up Unity on Ubuntu:

  1. Install Unity Hub 3.0.0 When following the instructions, make sure to install 3.0.0 i.e: sudo apt-get install unityhub=3.0.0 https://docs.unity3d.com/hub/manual/InstallHub.html#install-hub-linux
  2. In Unity hub install editor 2020.3.48f1
    3. Unity
  3. Select Android build support (Oculus quest 2 is an android device)
    4. unity-2

Setting up Occulus in Unity

  1. Make a new 3D Core template project in Unity
    2. unity-new
  2. Set up build options
    1. Go to file -> build settings and switch to Android -> switch platform
    2. Add a plane and cube to the scene
    3. Import Oculus integration package
      1. Add to your assets in unity package web page (must be logged in to you acc) https://assetstore.unity.com/packages/tools/integration/oculus-integration-82022
      2. Open package manager in the unity editor (window -> package manager)
      3. Choose “My Assets” in the drop down menu
        4. assets
      4. Select Oculus Integration and hit Download
      5. Click Import ... and wait a while. Unity-import If prompted to cleanup:
        1. Select show assets
          2. unity-dialoge-1
        2. Then select Upgrade
          3. unity-dialoge-2
        3. restart (save)
          4. unity-dialoge-3
        4. Sometimes it will find more to clean -> repeat until it starts without issues.
        5. If prompted to clean up -> do it.
          6. unity-dialoge-3
    4. Fix issues found be Unity
      1. Press oculus symbol in the bottom right if a red dot is present:
        2. Oculus fixes
      2. Press “fix all”
        3. Alt text
      3. Press fix all again.
      4. Press apply all for recommended actions ... wait a bit ...
        5. Oculus fixes again
      5. The red/orange dot should disappear from the symbol bottom right
    5. Drag OVRPlayerController.prefab into the scene
      6. Unity prefab
    6. Connect your Oculus to the PC (confirm usb dialog in the headset)
    7. Press on the red dot again if it appears and Fix all /apply all again.
    8. Open build settings -> file-> build settings
      9. Unity build
    9. Select Run Device -> Oculus quest 2. If the Oculus doesn’t show up under Run Device, make sure sure you have confirmed USB debugging in the headset when connecting the USB cable. If USB debug prompt does not show up try:
      1. Disable developer mode from phone
      2. Restart Oculus
      3. Enable developer mode from phone
      4. Restart Oculus.
      5. Connect USB cable.
      6. Click allow USB access in the headset.
      7. Confirm USB debugging in the headset.

  3. Press build and run (save apk file) ... wait a bit ...

    You might get some build errors or the Oculus just shows three dots... :

    • Check the oculus symbol in the bottom right and press fix until it disapiers. If errors consist:

    • If the error is related to the JDK, Android SDK & NDK (Unity installs some but for some Unity distros (like 18) it’s not working and has to be adjusted), we need to install the right version of these. You can set the JDK, SDK, NDK under “edit -> preferences -> External tools.

      Unity-preferences

    To install Android JDK, SDK, & NDK (Ubuntu 18.04):

    Install JDK

    1. install jdk sudo apt install openjdk-8-jdk
    2. Check you have correct version java -version
    3. Select it in unity.
    JDK

    Install SDK

    1. Download android sdk: wget https://dl.google.com/android/repository/sdk-tools-linux-4333796.zip Unzip: unzip sdk-tools-linux-4333796.zip -d androidSDK/
    2. Now we have to add it to the bashrc to be able to install some additional packages sudo nano ~/.bashrc Add this line to the file (replace username with your username (no “[]”)): export PATH=/home/[username]/androidSDK/tools/:/home/[username]/androidSDK/tools/bin:$PATH
    3. In a new terminal (or sources ./bashrc) go for: sdkmanager --list There should be bunch of stuff if not check the path again and make sure to have sourced or new terminal
    4. Now install 2 packages sdkmanager "build-tools;32.0.3" Accept the licence with y Install second one: sdkmanager "platform-tools"
    5. Select the androidSDK folder in unity: (mcw will be your username)
      6. SDKt

    Install NDK

    1. Download the NDK r19c from here: https://github.com/android/ndk/wiki/Unsupported-Downloads#r19c Direct link: https://dl.google.com/android/repository/android-ndk-r19c-linux-x86_64.zip From terminal: wget https://dl.google.com/android/repository/android-ndk-r19c-linux-x86_64.zip
    2. Unzip: unzip android-ndk-r19c-linux-x86_64 -d androidNDK/
    3. When selecting the folder in unity (android-ndk-r19c) you will see this error: NDK
      Manually change verision
      1. Navigate to the folder and open the file “source.properties”
      2. Change Pkg.Revision to 19.0.5232133
        3. NDK-2
    4. Now selection in unity will work
      5. NDK-3

Now you should be able to “Build and Run” your app on the Occulus Quest 2 I.e. you will see a white plane and cube and can walk around using the controllers and look around via headset movements

ROS to Unity connection

  1. Install ROS by following http://wiki.ros.org/melodic/Installation/Ubuntu (cahnge to your version if needed -> this guide continuous with melodic)
  2. Follow these steps to get the robotics tab in unity. Only install the ros tcp connector package! https://github.com/Unity-Technologies/Unity-Robotics-Hub/blob/main/tutorials/quick_setup.md
  3. It will look like this afterward (robotic tab appears)
    4. ROS in Unity
  4. Create a ROS workspace
    1. Create a workspace and src dir 
      2. mkdir -p ~/my_ws/src
    2. Init workspace
      3. 											catkin_init_workspace src
  5. Download the ros-tcp-endpoint package in your ros src folder:
    6. cd ~/my_ws/src
									git clone https://github.com/Unity-Technologies/ROS-TCP-Endpoint.git

  6. Clone the quest2_ros_msg package

    into your src.
    7. 											git clone https://github.com/vision-tactile-manip/quest2_ros_msg.git

  7. Build with catkin build

    cd ~/my_ws/
									catkin build
  8. Build ROS messaes in Unity. Go to robotics -> generate ROS Messages and set Ros message path to your quest2_ros_msg location. Press build 3 msgs
    ROS msgs
    You can see the unity message code thats been generated in your project:
    ROS msgs-2
    If you make a new message type do it in the ques2_ros_msg package and then build it in unity as described above

Moving a cube in VR over ros example

Now we will make a simple example that shows how to move a cube in vr mimicking your right controller movements when the “A” button is pressed over ROS

  1. Add scripts for ROS communication

    1. Get the teleop and cube_move unity scripts.
    2. Copy it into the assets folder
      unity-asset
      or, Make a new c# script in unity -> right click in assets create->c# script
    3. Draw the teleop script and the cube move script onto the plane:
      4. unity-script
    4. Name the cube in the scene “test_box”:
      5. unity-box
    5. Add ridgitbody to test_box (“Add component” -> “Rigidbody”) and disable gravity
      6. unity-box

  2. Configuring communication between ROS and headset:

    1. Make sure both devices (ROS pc and quest2) are in the same network
    2. Set rosIP Master etc:
      • In ROS terminal do: 
         export ROS_MASTER_URI=http://<pc that runs ros ip>:11311
											export ROS_IP=<pc that runs ros ip>
        Now start the ros tcp endpoint via
         roslaunch ros_tcp_endpoint endpoint.launch tcp_ip:=<pc that runs ros ip> tcp_port:=10000
      • In unity go to robotics->ROS settings and enter that ip in ros IP address:
        • ROS Unity connection
  3. Build and run the app. You can see if the connection is successful in the ros tcp terminal:
    4. ROS Unity connection
  4. Make python scrips executable i.e chmod +x filename
  5. Now in a new terminal set make the same ros_master and ros_IP settings and then run:
    rosrun quest2_ros_msg cube_move_ros.py
    The cube should move when pressing "A" and moving the right controller or pressing "X" and moving the left controller.

Haptic Feedback via Tactile sensors

  1. Get the haptic feedback unity scripts.
  2. Copy it into the assets folder or, Make a new c# script in unity -> right click in assets create->c# script
  3. Draw the haptic feedcack script onto the plane
  4. Rebulid the unity app on the Oculus
  5. Cancel the ROS cube move
  6. Connect the DIGIT sensors to your ROS PC
  7. Lunch digit_cams.launch (you might need to adjust the "video_device" parameter)
    8. roslaunch quest2_ros_msg digit_cams.launch
  8. start the haptic_feedback_via_tactile script in the ROS terminal (make sure the ROS_IP and ROS_MASTER_URI are correctly set + the ros tcp endpoint is running) and it's executable.
    9. rosrun quest2_ros_msg haptic_feedback_via_tactile.py
  9. Pressing the "B" button should start the reference frames collection (default 100 frames)
  10. Now use the touch sensors and recive the haptic feedback via the vibration of the controller