Models

Models are trained to recognize what happens on screen, like a kill feed, a victory screen or a specific item. A widget reads the models attached to it and reacts to their predictions.

getModel

breezy.getModel(id: ID): ModelSnapshot | null

Returns the model with the given ID, or null if it is not attached to the widget.

ModelType

ModelType.ObjectDetection       // 'OBJECT_DETECTION'
ModelType.ObjectClassification  // 'OBJECT_CLASSIFICATION'

ObjectClassificationModelObserver

Watches object classification models and calls you back when a label is detected or released. It works like the browser's IntersectionObserver.

class ObjectClassificationModelObserver {
	constructor(callback: ObjectClassificationModelObserverCallback, options?: ObjectClassificationModelObserverOptions)
	observe(id?: ID): void
	unobserve(id: ID): void
	disconnect(): void
}

Method	Description
observe(id)	Starts observing a model. Without an ID, observes every model attached to the widget.
unobserve(id)	Stops observing a model.
disconnect()	Stops observing everything.

let observer = new ObjectClassificationModelObserver(entries => {
	for (let entry of entries) {
		if (entry.detected && entry.label.name == 'victory') {
			playCelebration()
		}
	}
})

observer.observe()

Options

type ObjectClassificationModelObserverOptions = {
	threshold?: number
	stability?: number
	continuous?: boolean
}

Option	Description
threshold	Minimum score for a label to count as detected. Defaults to the model's threshold, or 0.5.
stability	How long, in milliseconds, the score must stay above the threshold before the label is detected. Defaults to the model's stability, or 0.
continuous	When true, the callback receives every prediction. By default it only receives changes, when a label is detected or released.

A label is released as soon as its score drops below the threshold.

Entries

type ObjectClassificationModelObserverEntry = {
	model: ObjectClassificationModelSnapshot
	label: ObjectClassificationModelLabel
	score: number
	detected: boolean
	time: number
}

type ObjectClassificationModelObserverCallback = (
	entries: Array<ObjectClassificationModelObserverEntry>,
	observer: ObjectClassificationModelObserver
) => void

Model snapshots

type ModelSnapshot = ObjectDetectionModelSnapshot | ObjectClassificationModelSnapshot

type ObjectClassificationModelSnapshot = {
	id: ID
	title: string
	infos: string
	inferenceRate: number
	stability: number
	threshold: number
	files: Array<ObjectClassificationModelFile>
	labels: Array<ObjectClassificationModelLabel>
	version?: ObjectClassificationModelVersion | null
}

type ObjectDetectionModelSnapshot = {
	id: ID
	title: string
	infos: string
	inferenceRate: number
	files: Array<ObjectDetectionModelFile>
	images: Array<ObjectDetectionModelImage>
	labels: Array<ObjectDetectionModelLabel>
	version?: ObjectDetectionModelVersion | null
}

type ObjectClassificationModelLabel = {
	id: ID
	name: string
	title: string
	color: string
}

type ObjectDetectionModelLabel = {
	id: ID
	name: string
	title: string
	infos: string
	color: string
}

type ModelVersion = {
	id: ID
	label: string
	major: number
	minor: number
	patch: number
}