SNIFF is composed of two main components, a video tracking system and a game engine for real time graphics. The video tracking system is built in openFrameworks; for the game engine we chose is Unity3d.

The low-poly dog was modeled and animated in Blender3d . The model with attached animated actions was imported into Unity3d, without smoothing to retain its polygonal appearance.

The real-time flow of information in SNIFF looks a bit like this:

People on the sidewalk are monitored by an IR camera in openFrameworks. In oF each individual person is isolated and assigned a unique id for the duration of their interaction. Each persons’ position and gesture information is continually sent to Unity3d via OSC networking protocol. In Unity, an artificial intelligence system representing the dog forms relationships with the individuals. He chooses which person to pay attention to, is able to move towards them or back away, responds to their gestures and initiates gestures of his own. Based on the interaction he gets excited or bored, friendly or aggressive, which is reflected in his behavior.

openFrameworks

To detect people we are using the standard OpenCV blob extraction packaged with openFrameworks. Beyond the standard blob tracker, we have a ‘PersonTracker’ which assigns persistent blob ids between frames, and dampens the movement of the centroids to reduce jittery head tracking on the dog. We also run custom metrics to filter out false-positive detections, such as the glare from headlights that otherwise would register as people. Through blob detection we are given a good indications of the locations of people, but we are also interested in how they are behaving. During each frame of video we run optical flow to detect motion. Within each person’s bounding rectangle we sum all the motion to determine if that individual is making sudden movements.  With this method, we found that walking back and forth triggers a lot of optical motion yet shouldn’t be considered as gesturing.  To disfavor this false-positive we dampen the total motion for each person by the movement of their centroid.  This favors movements such as waving arms and clapping hands while standing in front of the dog. But if someone is to quickly rush towards the dog it is considered an aggressive gesture and he will react. But don’t think you can just be mean to Sniff, because we also run analysis on the shape of each silhouette to determine if the person is holding out their hand calmly. This is considered a friendly gesture, and Sniff may start to trust them.  Based on watching people interact with Sniff we intend to implement recognition of more complex behaviors that are taken from observation of natural tendencies people have when they encounter the dog.

Unity3d

The abstracted information for each individual is sent to Unity where the artificial intelligence and animation engine is contained. Communication between the two applications is handled by OSC messages containing person ids, location and gesture information. The Unity3d system works with an object oriented architecture, containing instances of a dog, people and relationships between them. SNIFF also contains a mood, which is determined by the combination of friendliness and enthusiasm metrics which change over time as people interact with him. If a lot of people are around gesturing at him he’ll go berserk. If he’s been left alone for a while or you aren’t interacting with him, he may just lay down or pay attention to someone else. He has the capacity to form opinions about people over time and will behave according to how he has been treated.