Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions .gitattributes
Original file line number Diff line number Diff line change
Expand Up @@ -118,3 +118,9 @@ Assets/Plugins/** linguist-vendored

# Exceptions for .asset files such as lightning pre-baking
LightingData.asset binary

# Pin the small development-only Quak release without another LFS download.
Unity-QuestVisionKit/Packages/com.xrdevrob.quak-0.4.0-alpha.1.tgz binary -filter

# Imported copy of the same pinned Quak development layer for fresh-clone builds.
Unity-QuestVisionKit/Assets/XR/APILayers~/AndroidLayers/arm64/libXrApiLayer_QUAK_injection.so binary -filter
13 changes: 13 additions & 0 deletions .github/workflows/check-repo.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
name: Repository checks
on: [push, pull_request]
permissions:
contents: read
jobs:
clone-integrity:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
lfs: true
- run: python3 scripts/check_repo.py
- run: git diff --check
5 changes: 5 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -118,3 +118,8 @@ Assets/.avatar_sdk_sample_assets_importer.txt
**/RuntimeActionBindings.json
**/RuntimeActionBindings.json.meta
.editorconfig

# Local Unity build/validation evidence is delivered separately from source.
/*.log
/scene-audit.txt
/evidence/
10 changes: 9 additions & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ If you’d like to see new functionality:

When submitting a pull request:
1. **Fork the Repository:** Create your branch off of the `main` branch.
2. **Follow the Code Style:** Maintain clarity and consistency with the existing code. We put a lot of emphazise on making the samples simple and easy-to-understand, so other devs can build cool things quickly!
2. **Follow the Code Style:** Maintain clarity and consistency with the existing code. We put a lot of emphasis on making the samples simple and easy-to-understand, so other devs can build cool things quickly!
3. **Test Your Changes:** Ensure your modifications work on a supported Meta Quest device (Quest 3 or 3s with HorizonOS v74 or later) and in Unity (recommended Unity 6).
4. **Write Clear Commit Messages:** Describe what you’re changing and why.
5. **Link Related Issues:** Reference any issues or feature requests that your PR addresses.
Expand All @@ -42,3 +42,11 @@ To set up QuestCameraKit on your machine:
1. **Clone the Repository:**
```bash
git clone https://github.com/xrdevrob/QuestCameraKit.git
cd QuestCameraKit
git lfs pull
python3 scripts/check_repo.py
```

2. Open `Unity-QuestVisionKit` with the Unity version in `ProjectSettings/ProjectVersion.txt` and let Package Manager finish.
3. Run the checks and the selected sample build described in [Testing](docs/testing.md).
4. Keep credentials out of scenes and prefabs. Use a private development key only for local ImageLLM testing; production apps should call an authenticated backend.
73 changes: 34 additions & 39 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,12 @@ QuestCameraKit is a collection of template and reference projects demonstrating
- [License](#license)
- [Contact](#contact)

# Running all samples

Build and Run now includes all six headset samples in one app. Press **Y** to open or close the menu, move the **left thumbstick** to choose, and press **X** to load a sample. In the Editor use **M**, **↑/↓**, and **Enter**. This menu currently uses Touch controllers; the samples retain their existing hand interactions. The desktop WebRTC receiver remains a separate scene.

**QR tracking (Meta native)** uses the MRUK package already required by this project. No separate decoder, NuGet installation, or QR scripting define is needed. Native tracking still depends on headset runtime support and spatial-data permission.

# PCA Samples

## 1. 🎨 Color Picker
Expand Down Expand Up @@ -41,7 +47,7 @@ QuestCameraKit is a collection of template and reference projects demonstrating
<summary>How to run this sample</summary>

- Open the `ObjectDetection` scene.
- Install [Unity AI Inference](https://docs.unity3d.com/Packages/com.unity.ai.inference@2.3/manual/get-started.html) (use `com.unity.ai.inference@2.3.0`).
- [Unity Inference Engine](https://docs.unity3d.com/Packages/com.unity.ai.inference@2.6/manual/index.html) is already pinned to `2.6.1` in the project.
- Select the labels you want to track. Leaving the list empty tracks all objects. <details>
<summary>Show all available labels</summary>
<table>
Expand All @@ -57,28 +63,29 @@ QuestCameraKit is a collection of template and reference projects demonstrating
<tr><td>refrigerator</td><td>book</td><td>clock</td><td>vase</td><td>scissors</td><td>teddy bear</td><td>hair drier</td><td>toothbrush</td></tr>
</table>
</details>
- Build and deploy to Quest. Use the trigger to scan the environment; markers will appear for detections above the configured confidence threshold.
- Build and deploy to Quest. Detection runs automatically; markers appear for the selected labels. Confidence filtering applies only when the model provides scores.

</details>

![Object Detection](https://media.githubusercontent.com/media/xrdevrob/QuestCameraKit/befec3b/Media/ObjectDetection.gif)

## 3. 📱 QR Code Tracking with ZXing
## 3. 📱 Native QR Code Tracking

- **Purpose:** Detect and track QR codes in real time. Open webviews or log-in to 3rd party services with ease.
- **Description:** Similarly to the object detection sample, get QR code coordinated and projects them into 3D space. Detect QR codes and call their URLs. You can select between a multiple or single QR code mode. The sample is running at around `70 fps` for multiple QR codes and a stable `72 fps` for a single code. Users are able to choose between CenterOnly and PerCorner raycasting modes via an enum in the inspector. This enables more accurate rotation tracking for use cases that require it (PerCorner), while preserving a faster fallback (CenterOnly).
- **Purpose:** Detect and track QRCodes without having to use a third-party library.
- **Description:** As this feature is still in experimental state, make sure to use experimental mode on your Quest3 when testing. Unity usually asks for enabling that, before building. You can activate it via command line command using Meta Quest Hub too. More information can be found here - [MR Utility Kit QRCode Detection](https://developers.meta.com/horizon/documentation/unity/unity-mr-utility-kit-qrcode-detection) and here - [Mobile Experimental Features](https://developers.meta.com/horizon/documentation/native/android/mobile-experimental-features/)

<details>
<summary>How to run this sample</summary>

- Open the `QRCodeTracking` scene.
- Ensure ZXing DLLs are present (the editor script auto-adds the `ZXING_ENABLED` define).
- Choose Single or Multiple detection mode and the raycast mode (CenterOnly vs PerCorner).
- Build to Quest, point the headset toward QR codes, and interact with the spawned markers.
- Install **TextMeshPro Essentials** if prompted.
- Enable Experimental Mode on your Quest 3/3S (headset UI or Quest Hub CLI).
- Open the **QRCodeDetection** scene.
- Build and deploy to your device, then use the controllers to interact with the UI (hand tracking is not yet supported).

</details>

![QR Code Tracking](https://media.githubusercontent.com/media/xrdevrob/QuestCameraKit/befec3b/Media/QRCodeTracking.gif)
![Native QR tracking](https://media.githubusercontent.com/media/xrdevrob/QuestCameraKit/befec3b/Media/QrCodeDetection.png)


## 4. 🪟 Shader Samples

Expand Down Expand Up @@ -210,19 +217,16 @@ https://github.com/user-attachments/assets/a4cfbfc2-0306-40dc-a9a3-cdccffa7afea
<details>
<summary>How to run this sample</summary>

- In **Package Manager**, click the **+** button → **Add package from git URL** and install, in order:
1. `https://github.com/endel/NativeWebSocket.git#upm`
2. `https://github.com/Unity-Technologies/com.unity.webrtc.git`
3. `https://github.com/FireDragonGameStudio/SimpleWebRTC.git?path=/Assets/SimpleWebRTC`
- Unity WebRTC, SimpleWebRTC and NativeWebSocket are pinned in `Packages/manifest.json` and install automatically.
- Open the **WebRTC-Quest** scene.
- On `[BuildingBlock] Camera Rig/TrackingSpace/CenterEyeAnchor/Client-STUNConnection`, set your WebSocket signaling server address (LAN or cloud).
- On `[BuildingBlock] Camera Rig/TrackingSpace/CenterEyeAnchor/Client-STUNConnection`, replace the placeholder with your own WebSocket signaling server address (LAN or cloud). Connections are disabled by default. After setting your server address on both peers, enable WebSocket Connection Active to connect to signaling.
- Build and deploy **WebRTC-Quest** to your Quest 3.
- Open the **WebRTC-SingleClient** scene in the editor (or deploy to another device) to act as the receiving peer.
- Launch both apps. Perform the **Start** gesture with your left hand (or press the menu button) on Quest to begin streaming.

**Troubleshooting**

- If you hit compiler errors, re-add the three git packages listed above.
- If package resolution fails, verify network/Git access and reopen the project with the Unity version in `ProjectSettings/ProjectVersion.txt`.
- Use **Tools ▸ Update WebRTC Define Symbol** after importing packages.
- Ensure your own WebSocket signaling server is running (tutorial [here](https://www.youtube.com/watch?v=-CwJTgt_Z3M)).
- For LAN streaming, leave the STUN address empty; otherwise keep the default.
Expand All @@ -236,24 +240,16 @@ https://github.com/user-attachments/assets/a4cfbfc2-0306-40dc-a9a3-cdccffa7afea

![PCA WebRTC](https://media.githubusercontent.com/media/xrdevrob/QuestCameraKit/befec3b/Media/PCA_WebRTC.gif)

## 7. 📅 Experimental QR Code detection

- **Purpose:** Detect and track QRCodes without having to use a third-party library.
- **Description:** As this feature is still in experimental state, make sure to use experimental mode on your Quest3 when testing. Unity usually asks for enabling that, before building. You can activate it via command line command using Meta Quest Hub too. More information can be found here - [MR Utility Kit QRCode Detection](https://developers.meta.com/horizon/documentation/unity/unity-mr-utility-kit-qrcode-detection) and here - [Mobile Experimental Features](https://developers.meta.com/horizon/documentation/native/android/mobile-experimental-features/)

<details>
<summary>How to run this sample</summary>

- Install **TextMeshPro Essentials** if prompted.
- Enable Experimental Mode on your Quest 3/3S (headset UI or Quest Hub CLI).
- Open the **QRCodeDetection** scene.
- Build and deploy to your device, then use the controllers to interact with the UI (hand tracking is not yet supported).
# Update Notes

</details>
## Maintenance update

![Shader Samples](https://media.githubusercontent.com/media/xrdevrob/QuestCameraKit/befec3b/Media/QrCodeDetection.png)
- Unity **6000.3.12f1**, Meta XR Core/MRUK **205.0.0**, Inference Engine **2.6.1**, OpenXR **1.18.0**, Meta OpenXR **2.6.1**, and URP **17.3.0**. Other direct dependencies are pinned in the manifest.
- QR tracking uses Meta MRUK; WebRTC dependencies install with the project. All seven sample scenes are checked for missing scripts.
- Camera/inference lifecycle fixes, capture-pose detection placement, reusable detection markers, safer voice requests, and corrected stereo WAV encoding.
- API credentials must be supplied privately for ImageLLM; never commit a real key to a prefab or scene.
- **Build and Run** defaults to ColorPicker. Use the per-sample build command and Quak device tests in [Testing](docs/testing.md) for the other scenes.

# Update Notes

## Shader sample consolidation

Expand All @@ -280,8 +276,8 @@ https://github.com/user-attachments/assets/a4cfbfc2-0306-40dc-a9a3-cdccffa7afea

## Prerequisites

- **Meta Quest Device:** Ensure you are runnning on a `Quest 3` or `Quest 3s` and your device is updated to `HorizonOS v74` or later.
- **Unity:** Recommended is `Unity 6`. Also runs on Unity `2022.3. LTS`.
- **Meta Quest Device:** Ensure you are running on a `Quest 3` or `Quest 3s` and your device is updated to `HorizonOS v74` or later.
- **Unity:** Use `6000.3.12f1` with Android Build Support. This upgraded project targets Unity 6.3; older editor versions are not supported by the current package set.
- **Camera Passthrough API does not work in the Editor or XR Simulator.**
- Get more information from the [Meta Quest Developer Documentation](https://developers.meta.com/horizon/documentation/unity/unity-pca-documentation)

Expand All @@ -292,16 +288,16 @@ https://github.com/user-attachments/assets/a4cfbfc2-0306-40dc-a9a3-cdccffa7afea
```

2. **Open the Project in Unity:**
Launch Unity and open the cloned project folder.
Launch Unity and open `Unity-QuestVisionKit` inside the cloned repository.

3. **Configure Dependencies:**
Follow the instructions in the section below to run one of the samples.
Allow Package Manager to resolve the pinned dependencies, then follow the selected sample instructions. See [Testing](docs/testing.md) for repeatable checks and builds.

# Troubleshooting & Known Issues

- For sample 7 QR Code Detection, make sure experimental mode is active. You can find more information about the necessary steps [here](https://developers.meta.com/horizon/documentation/unity/unity-mr-utility-kit-qrcode-detection).
- Object Detection uses `Unity.InferenceEngine` with `com.unity.ai.inference@2.3.0`.
- If switching betwenn Unity 6 and other versions such as 2023 or 2022 it can happen that your Android Manifest is getting modified and the app won't run anymore. Should this happen to you make sure to go to `Meta > Tools > Update AndroidManifest.xml` or `Meta > Tools > Create store-compatible AndroidManifest.xml`. After that make sure you add back the `horizonos.permission.HEADSET_CAMERA` manually into your manifest file.
- For sample 3 QR Code Detection, make sure experimental mode is active. You can find more information about the necessary steps [here](https://developers.meta.com/horizon/documentation/unity/unity-mr-utility-kit-qrcode-detection).
- Object Detection uses `Unity.InferenceEngine` with `com.unity.ai.inference@2.6.1`.
- If switching between Unity 6 and other versions such as 2023 or 2022 it can happen that your Android Manifest is getting modified and the app won't run anymore. Should this happen to you make sure to go to `Meta > Tools > Update AndroidManifest.xml` or `Meta > Tools > Create store-compatible AndroidManifest.xml`. After that make sure you add back the `horizonos.permission.HEADSET_CAMERA` manually into your manifest file.

# README Media Hosting

Expand Down Expand Up @@ -389,7 +385,6 @@ large files are removed from the current checkout. See

- Thanks to **Meta** for the Passthrough Camera API and [**Passthrough Camera API Samples**](https://github.com/oculus-samples/Unity-PassthroughCameraApiSamples/).
- Thanks to shader wizard [Daniel Ilett](https://www.youtube.com/@danielilett) for helping me in the shader samples.
- Thanks to **[Michael Jahn](https://github.com/micjahn/ZXing.Net/)** for the XZing.Net library used for the QR code tracking samples.
- Special thanks to [Markus Altenhofer](https://www.linkedin.com/in/markus-altenhofer-176453155/) from [FireDragonGameStudio](https://www.youtube.com/@firedragongamestudio) for contributing the WebRTC sample scene.
- Special thanks to [Thomas Ratliff](https://x.com/devtom7) for contributing his [shader samples](https://x.com/devtom7/status/1902033672041091453) to the repo.

Expand Down
15 changes: 14 additions & 1 deletion Unity-QuestVisionKit/.gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -127,4 +127,17 @@ Assets/Nuget.config
Assets/Nuget.config.meta
**/RuntimeActionBindings.json
**/RuntimeActionBindings.json.meta
.editorconfig
.editorconfig

# Machine-local SDK configuration and test/build output.
Assets/Resources/DevAgentSettings.asset
Assets/Resources/DevAgentSettings.asset.meta
Assets/Resources/OVRPlatformToolSettings.asset.meta
Assets/Resources/PerformanceTestRunInfo.json*
Assets/Resources/PerformanceTestRunSettings.json*
Assets/XR/Temp/
Assets/XR/Temp.meta
.quak/evidence/
.quak/machines.local.json
.quak/packages/
.quak/venv/
8 changes: 8 additions & 0 deletions Unity-QuestVisionKit/Assets/Diagnostics.meta

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading
Loading