Class DeepAR

Main class for interacting with DeepAR SDK. To get a DeepAR object call initialize.

Hierarchy

DeepAR

Properties

callbacks

Callbacks property is used to add/remove/change callbacks called by the DeepAR SDK. See list of all callbacks at DeepARCallbacks.

Example: To add/change certain callback:

let deepAR = deepar.initialize({...});
deepAR.callbacks.onFaceVisibilityChanged = () => {
    // do something
};

To remove certain callback:

deepAR.callbacks.onFaceTracked = undefined;

ScriptingAPI

ScriptingAPI: ScriptingAPI

Scripting API property used to access all the scripting interop methods.

Example

let scriptingVariable = deepAR.ScriptingAPI.getStringVar('variableName');

Methods

switchEffect

switchEffect(effect: string | ArrayBuffer, effectOptions?: {
slot?: string;
face?: number;
}): Promise<void>

Switch the AR effect for preview.

Throws

SwitchEffectCanceled If the switch effect is canceled by something.

Example

// Switch filter 1.
await deepAR.switchEffect('url/path/to/filter1');
// Later switch fo filter 2.
await deepAR.switchEffect('url/path/to/filter2');

// Remove the current filter.
deepAR.clearEffect();

// Put two filters at the same time.
await deepAR.switchEffect('url/path/to/backgroundReplacement', {slot: 'background'});
await deepAR.switchEffect('url/path/to/glasses', {slot: 'faceMask'});
// Replace the glasses filter.
await deepAR.switchEffect('url/path/to/glasses2', {slot: 'faceMask'});
// Remove those filters.
deepAR.clearEffect('background');
deepAR.clearEffect('faceMask');

// Load filters for two people.
await deepAR.switchEffect('url/path/to/faceMask1', {face: 0, slot: 'mask1'});
await deepAR.switchEffect('url/path/to/faceMask2', {face: 1, slot: 'mask2'});
// Clear effect for the second person.
deepAR.clearEffect('mask2');

Parameters

effect: string | ArrayBuffer

A path (URL) to the AR effect file or an ArrayBuffer object of an already fetched AR effect file.
Optional effectOptions: {
slot?: string;
face?: number;
}

Effect options.
- Optional slot?: string
  
  Default value is "DEFAULT_SLOT" if slot is not given. Slot is a container for a given AR effect. When replacing the already loaded AR effect, call switchEffect with that same slot. To remove AR effect entirely, call clearEffect with the same slot.
- Optional face?: number
  
  If AR effect is face filter, select to which face to attach it to. The value should be between 0 and 3. Default value is 0.

Returns Promise<void>

clearEffect

clearEffect(slot?: string): void
Clears the given slot of any loaded effects.
Parameters
- Optional slot: string
  
  The effect slot name. Default is "DEFAULT_SLOT".
Returns void

takeScreenshot

takeScreenshot(): Promise<string>
Captures a screenshot of the current screen.

Returns
Data URL promise containing the image in data:image/png format.

Returns Promise<string>

startVideoRecording

startVideoRecording(options?: {
mimeType?: string;
}): void
Starts video recording of the canvas.
Parameters
- Optional options: {
  mimeType?: string;
  }
  
  Parameters that specify the format of recorded videos
  - Optional mimeType?: string
    
    A MIME type specifying the format for the resulting video, such as video/webm or video/mp4. Corresponds to the MIME type used by Blobobjects and MediaRecorderfrom the MediaStream Recording API. Note that video/mp4 may not be supported in all browsers.
Returns void

finishVideoRecording

finishVideoRecording(): Promise<Blob>
Stops the video recording and returns a video blob.

Returns
A promise resolving to Blob of a video.

Returns Promise<Blob>

startCamera

startCamera(cameraOptions?: {
    mirror?: boolean;
    mediaStreamConstraints?: MediaStreamConstraints;
    cameraPermissionAsked?: (() => void);
    cameraPermissionGranted?: (() => void);
}): Promise<void>
Starts the camera preview. By default, the camera will be user facing. The returned promise will resolve after the camera starts or it will reject if camera permission was denied.
Parameters
- Optional cameraOptions: {
      mirror?: boolean;
      mediaStreamConstraints?: MediaStreamConstraints;
      cameraPermissionAsked?: (() => void);
      cameraPermissionGranted?: (() => void);
  }
  
  Camera options.
  - Optional mirror?: boolean
    
    Mirror the camera horizontally. True by default.
  - Optional mediaStreamConstraints?: MediaStreamConstraints
    
    Options passed to MediaDevices.getUserMedia(). The default is the user facing camera.
  - Optional cameraPermissionAsked?: (() => void)
    - (): void
      
      Callback called when camera permission is asked.
      
      Returns void
  - Optional cameraPermissionGranted?: (() => void)
    - (): void
      
      Callback called when camera permission is granted.
      
      Returns void
Returns Promise<void>

stopCamera

stopCamera(): void
Stops the camera preview.

Returns void

stopVideo

stopVideo(): void
Stops the camera preview or custom video preview set by setVideoElement.

Returns void

setVideoElement

setVideoElement(videoElement: HTMLVideoElement, mirror: boolean): void
Used to pass the HTMLVideoElement to the DeepAR SDK. The SDK will use this video as camera source. This method should be used instead of startCamera when you want to handle getUserMedia outside the SDK or you need to apply the masks to any video stream. To disable automatic camera preview by DeepAR:
```
const deepAR = deepar.initialize({
    // ...
    additionalOptions: {
        cameraConfig: {
            disableDefaultCamera: true
        }
    }
});
```
Parameters
- videoElement: HTMLVideoElement
  
  Video element.
- mirror: boolean
  
  Mirror the video horizontally.
Returns void

shutdown

shutdown(): void
Shutdown the DeepAR SDK and release all resources associated with it. It is invalid to call any function from this DeepAR object after shutdown. After shutdown call, it is possible to call initialize again.

Returns void

processFrame

processFrame(frame: Uint8Array, width: number, height: number, mirror: boolean): void
Feed RGBA image to DeepAR as input instead of camera or video. Used for processing single image. Can be used instead of startCamera or setVideoElement. Can be called in a loop.
Parameters
- frame: Uint8Array
  
  Image.
- width: number
  
  Width of the image.
- height: number
  
  Height of the image.
- mirror: boolean
  
  Mirror frame horizontally.
Returns void

processImage

processImage(image: HTMLImageElement): void
If you want to apply DeepAR processing on a single image instead of a camera stream use this method. Can be used instead of startCamera or setVideoElement. See example usage here.
Parameters
- image: HTMLImageElement
Returns void

setPaused

setPaused(isPause: boolean): void
Pauses the DeepAR processing and rendering on canvas.
Parameters
- isPause: boolean
  
  If true, DeepAR will pause. Otherwise, it will resume processing and rendering.
Returns void

changeParameterFloat

changeParameterFloat(gameObject: string, component: string, parameter: string, value: number): void
Changes a node or component bool parameter of the currently loaded effect. For more details about changeParameter API read our docs here.
Parameters
- gameObject: string
  
  The name of the node from DeepAR Studio. If multiple nodes share the same name, only the first one will be affected.
- component: string
  
  The name of the component. If the name of the component is null or an empty string, the node itself will be affected.
- parameter: string
  
  The name of the parameter.
- value: number
  
  New parameter value.
Returns void

changeParameterBool

changeParameterBool(gameObject: string, component: string, parameter: string, value: boolean): void
Changes a node or component float parameter of the currently loaded effect. For more details about changeParameter API read our docs here.
Parameters
- gameObject: string
  
  The name of the node from DeepAR Studio. If multiple nodes share the same name, only the first one will be affected.
- component: string
  
  The name of the component. If the name of the component is null or an empty string, the node itself will be affected.
- parameter: string
  
  The name of the parameter.
- value: boolean
  
  New parameter value.
Returns void

changeParameterVector

changeParameterVector(gameObject: string, component: string, parameter: string, x: number, y: number, z: number, w: number): void
Changes a node or component vector parameter of the currently loaded effect. For more details about changeParameter API read our docs here.
Parameters
- gameObject: string
  
  The name of the node from DeepAR Studio. If multiple nodes share the same name, only the first one will be affected.
- component: string
  
  The name of the component. If the name of the component is null or an empty string, the node itself will be affected.
- parameter: string
  
  The name of the parameter.
- x: number
  
  X component of the new parameter vector.
- y: number
  
  Y component of the new parameter vector.
- z: number
  
  Z component of the new parameter vector.
- w: number
  
  W component of the new parameter vector.
Returns void

changeParameterTexture

changeParameterTexture(gameObject: string, component: string, parameter: string, textureUrl: string): void
Changes a node or component texture parameter of the currently loaded effect. For more details about changeParameter API read our docs here.
Parameters
- gameObject: string
  
  The name of the node from DeepAR Studio. If multiple nodes share the same name, only the first one will be affected.
- component: string
  
  The name of the component. If the name of the component is null or an empty string, the node itself will be affected.
- parameter: string
  
  The name of the parameter.
- textureUrl: string
  
  Url of the image that is going to be used as texture.
Returns void

fireTrigger

fireTrigger(trigger: string): void
This method allows the user to fire a custom animation trigger for model animations from code. To fire a custom trigger, the trigger string must match the custom trigger set in the Studio when creating the effect. Read more here.
Parameters
- trigger: string
  
  The name of the trigger.
Returns void

touchOccurred

touchOccurred(touchInfo: ARTouchInfo): void
Informs DeepAR that the specified touch event occurred.
Parameters
- touchInfo: ARTouchInfo
  
  Touch event information.
Returns void

setFaceDetectionSensitivity

setFaceDetectionSensitivity(sensitivity: number): void
Change face detection sensitivity
Parameters
- sensitivity: number
  
  0 .. 5 (0 is fast, 4,5 is slow but allows to find smaller faces)
Returns void

setOffscreenRenderingEnabled

setOffscreenRenderingEnabled(enable: boolean): void
Enable/disable forced rendering on the canvas. It is useful to enable offscreen rendering in scenarios when the browser does not call requestAnimationFrame() function. DeepAR internally uses requestAnimationFrame() for the rendering loop. For example, when the browser tab is not focused, the browser will not call requestAnimationFrame() and DeepAR will not render. If offscreen rendering is enabled, DeepAR will use its internal timer for the rendering loop. Note that offscreen rendering enabled will not have as good results in terms of FPS compared to offscreen rendering disabled.

If you need to use offscreen rendering. The best practice is to enable it only when needed - like when the browser tab is not focused. Otherwise, it is best to always disable offscreen rendering.
Parameters
- enable: boolean
  
  True - DeepAR will use its internal timer for the rendering loop. Rendering will work even when tab is not focused. False - DeepAR will use requestAnimationFrame() for the rendering loop.
Returns void

setFps

setFps(fps: number): void
Set the FPS of DeepAR rendering.
Parameters
- fps: number
  
  New FPS.
Returns void

initializeFootTracking

initializeFootTracking(): void
Initialize foot tracking.

Foot tracking is usually lazy loaded on demand when filter loaded with switchEffect requires it. But this method will start loading the foot tracking immediately. To start initializing foot tracking as soon as possible pass "footInit" hint in the initialize function (see DeepARParams).

If the foot tracking is already initialized it will do nothing. To check if foot tracking is initialized call isFootTrackingInitialized or wait onFootTrackingInitialized callback.

Returns void

isFootTrackingInitialized

isFootTrackingInitialized(): boolean
Check weather the foot tracking is initialized.

Returns boolean

initializeSegmentation

initializeSegmentation(): void
Initialize segmentation.

Segmentation is usually lazy loaded on demand when filter loaded with switchEffect requires it. But this method will start loading the segmentation immediately. To start initializing segmentation as soon as possible pass "segmentationInit" hint in the initialize function (see DeepARParams).

If the segmentation is already initialized it will do nothing. To check if segmentation is initialized call isSegmentationInitialized or wait onSegmentationInitialized callback.

Returns void

isSegmentationInitialized

isSegmentationInitialized(): boolean
Check weather the segmentation is initialized.

Returns boolean

isWristTrackingInitialized

isWristTrackingInitialized(): boolean
Check weather the wrist tracking is initialized.

Returns boolean

showStats

showStats(enabled: boolean): void
Display profiling metrics on preview.
Parameters
- enabled: boolean
Returns void

simulatePhysics

simulatePhysics(enabled: boolean): void
Enable or disable global physics simulation.
Parameters
- enabled: boolean
Returns void

moveGameObject

moveGameObject(selectedGameObject: string, targetGameObject: string): void
Moves the selected game object from its current position in a tree and sets it as a direct child of a target game object. This is equivalent to moving around a node in the node hierarchy in the DeepAR Studio.
Parameters
- selectedGameObject: string
  
  Node to move.
- targetGameObject: string
  
  New node parent.
Returns void

muteSound

muteSound(muteSound: boolean): void
Mutes or un-mutes all the sounds that are currently playing.
Parameters
- muteSound: boolean
  
  true if you want to mute all the sounds.
Returns void

backgroundBlur

backgroundBlur(enable: boolean, strength: number): Promise<void>
Enable background blur.

Background blur is usually used in video calling use cases.
Parameters
- enable: boolean
  
  Boolean indicating whether to enable or disable the background blur effect.
- strength: number
  
  Blur strength. Integer number in range 1-10.
Returns Promise<void>

backgroundReplacement

backgroundReplacement(enable: boolean, image: string): Promise<void>
Enable background replacement (also known as background removal or green screen effect).
Parameters
- enable: boolean
  
  Boolean indicating whether to enable or disable the background replacement effect.
- image: string
  
  The URL of the image to be used as the background.
Returns Promise<void>

setZoom

setZoom(zoomLevel: number): void
Sets the preview zoom.
Parameters
- zoomLevel: number
  
  Floating point number determining how much to zoom in. Value <= 1 will disable zoom. Example, value 2.0 puts 2x zoom.
Returns void

Class DeepAR

Hierarchy

Index

Properties

Methods

Properties

callbacks

ScriptingAPI

Example

Methods

switchEffect

Throws

Example

Parameters

effect: string | ArrayBuffer

Optional effectOptions: { slot?: string; face?: number; }

Optional slot?: string

Optional face?: number

Returns Promise<void>

clearEffect

Parameters

Optional slot: string

Returns void

takeScreenshot

Returns

Returns Promise<string>

startVideoRecording

Parameters

Optional options: { mimeType?: string; }

Optional mimeType?: string

Returns void

finishVideoRecording

Returns

Returns Promise<Blob>

startCamera

Parameters

Optional cameraOptions: { mirror?: boolean; mediaStreamConstraints?: MediaStreamConstraints; cameraPermissionAsked?: (() => void); cameraPermissionGranted?: (() => void); }

Optional mirror?: boolean

Optional mediaStreamConstraints?: MediaStreamConstraints

Optional cameraPermissionAsked?: (() => void)

Returns void

Optional cameraPermissionGranted?: (() => void)

Returns void

Returns Promise<void>

stopCamera

Returns void

stopVideo

Returns void

setVideoElement

Parameters

videoElement: HTMLVideoElement

mirror: boolean

Returns void

shutdown

Returns void

processFrame

Parameters

frame: Uint8Array

width: number

height: number

mirror: boolean

Returns void

processImage

Parameters

image: HTMLImageElement

Returns void

setPaused

Parameters

isPause: boolean

Returns void

changeParameterFloat

Parameters

gameObject: string

component: string

parameter: string

value: number

Returns void

changeParameterBool

Parameters

gameObject: string

`Optional` effectOptions: {
slot?: string;
face?: number;
}

`Optional` slot?: string

`Optional` face?: number

`Optional` slot: string

`Optional` options: {
mimeType?: string;
}

`Optional` mimeType?: string

`Optional` cameraOptions: {
mirror?: boolean;
mediaStreamConstraints?: MediaStreamConstraints;
cameraPermissionAsked?: (() => void);
cameraPermissionGranted?: (() => void);
}

`Optional` mirror?: boolean

`Optional` mediaStreamConstraints?: MediaStreamConstraints

`Optional` cameraPermissionAsked?: (() => void)

`Optional` cameraPermissionGranted?: (() => void)