Views
No views yet
face-mediapipe-pipeline.zipface-mediapipe-pipeline/ with this structure:1face-mediapipe-pipeline/
2├── manifest.json
3├── gltf/
4│ └── frame.gltf
5├── model/
6│ └── face_detector.tflite
7└── pipeline/
8 ├── face_detection_pipeline.json
9 └── face_display_pipeline.jsonmanifest.json: schema v2 package metadata. It declares the package id, pipeline entries, schema version, and Spatial runtime support.model/face_detector.tflite: MediaPipe Face detection model in TFLite format.pipeline/face_detection_pipeline.json: SpatialML pipeline definition for VST camera access, affine preprocessing, color conversion, LiteRT model inference, and MediaPipe SSD post-processing.pipeline/face_display_pipeline.json: SpatialML pipeline definition for consuming detection output, projecting the face center into 3D camera space, and producing a pose for the face-frame scene.gltf/frame.gltf: glTF asset rendered by the display pipeline.detection: reads rectified VST camera tensors, preprocesses the left camera image into a 256x256 RGB float input, runs the MediaPipe Face model, decodes the best SSD detection, and writes the post_det tensor.display: consumes post_det together with VST image, timestamp, and camera-matrix tensors, converts the detected image-space face center into 3D camera space, and outputs frame_pose plus frame_gltf for rendering.1"runtime": {
2 "supported_modes": [
3 "spatial"
4 ]
5}face-mediapipe-pipeline/ folder in your Android app assets. A Spatial SDK app can keep SpatialML assets under a namespaced assets path:app/src/main/assets/SpatialML/face-mediapipe-pipeline/1public fun SpatialMLSession.loadPipelinePackageFromAssets(
2 assetRoot: String,
3 externalGlobals: Map<String, GlobalTensor> = emptyMap(),
4): PipelinePackageBundlemanifest.json, validates that runtime.supported_modes includes spatial, builds each declared pipeline, creates or reuses global tensors for package inputs and outputs, and returns a PipelinePackageBundle containing:pipelines: package pipelines keyed by manifest id.globalTensors: materialized package tensors keyed by tensor name.manifest: parsed package metadata.1import com.pico.spatial.ml.securemr.SceneGraphProperty
2import com.pico.spatial.ml.securemr.SpatialMLInstance
3import com.pico.spatial.ml.securemr.SpatialMLSession
4import com.pico.spatial.ml.securemr.loadPipelinePackageFromAssets
5
6val instance = SpatialMLInstance.create(context)
7val session =
8 instance.createSession(
9 SpatialMLSession.InitInfo(
10 imageWidth = 580,
11 imageHeight = 326,
12 containerWidth = 1000,
13 containerHeight = 1000,
14 containerDepth = 10,
15 containerType = SpatialMLSession.ContainerType.VOLUMETRIC,
16 )
17 )!!
18
19val bundle = session.loadPipelinePackageFromAssets("SpatialML/face-mediapipe-pipeline")
20val detection = bundle.pipelines["detection"]!!
21val display = bundle.pipelines["display"]!!
22val frameScene = bundle.globalTensors["frame_gltf"]!!
23val framePose = bundle.globalTensors["frame_pose"]!!
24
25val initTask =
26 session.newPipeline().run {
27 // The full sample also sets an initial pose and scale before showing the scene.
28 switchSceneVisibility(frameScene, newLocalTensor(0xF.toByte()))
29 submit(emptyMap(), null, null)
30 }
31
32val framePipeline =
33 session.newPipeline().apply {
34 updateSceneGraphProperty(
35 frameScene,
36 "/",
37 SceneGraphProperty.CameraAnchor.Follow,
38 framePose,
39 )
40 }
41
42val detectionTask = detection.pipeline.submit(detection.submitBindings, null, initTask)
43val displayTask = display.pipeline.submit(display.submitBindings, null, detectionTask)
44framePipeline.submit(emptyMap(), null, displayTask)frame_pose to the packaged frame_gltf scene through SceneGraphProperty.CameraAnchor.Follow.post_det: decoded face detection output.vst_left_image, vst_right_image, vst_timestamp, vst_camera_matrix: VST tensors shared between the detection and display pipelines.frame_pose: transform matrix generated by the display pipeline.frame_gltf: packaged glTF scene rendered as the face frame..json, .gltf, and .tflite in androidResources.noCompress. Apps using VST camera tensors and spatial-data readback also need the appropriate app permissions, including com.picovr.permission.SPATIAL_DATA.SpatialMLSession.loadPipelinePackageFromAssets(...), create a volumetric SpatialML session, submit the detection pipeline first, then submit the display pipeline to update the face-frame pose and glTF output.