What is speaker tracking?

A 9:16 crop of a 16:9 video keeps about 32% of its width. On a talking head video with one person in the middle, that's enough. On a two-person podcast filmed wide, the frame has room for one face, not both, and speaker tracking decides which, moment by moment. In Clyper it works in four steps:

1. Split the clip at camera cuts. Each shot gets its own framing, changing exactly when the edit does.
2. Find and follow the faces. Faces are detected through each shot and linked into one track per person, so the tool knows who is who as they lean and turn.
3. Work out who's talking. When people sit too far apart to share the frame, it measures the movement around each person's mouth. This is the active speaker detection part.
4. Move a steady virtual camera. It holds still while the face stays near the middle and eases over when it moves away. When the speaker changes, it cuts to them rather than panning across the room.

Switching is the hard part. A frame that jumps to the listener on every “yeah” is worse than no tracking at all, so Clyper waits until the other person is clearly talking, then switches from the moment they started, so their first words aren't lost. People sitting close enough share the frame instead of being cut apart.

The words get used for each other, but they name different parts of the job:

• Face tracking follows a face through the video. With one person on camera, that's all you need. With two, it has to pick one, and it doesn't know who's talking.
• Active speaker detection decides which of the visible faces is talking at each moment. It's the decision, not the framing.
• Speaker tracking puts the two together: track the faces, pick the speaker, move the frame.
• Auto reframe is the whole job of turning a wide video into a vertical video, for any footage. Speaker tracking is the part of it built for people talking.

By hand, the same job is keyframing: cut at every camera change and every change of speaker, and move the crop each time. How editors do it, and their own auto reframe tools, are in how to convert a horizontal video to vertical.

Streams are where following the face is the wrong move: a small facecam in a corner, blown up to fill a vertical frame, crops the game away. So when Clyper sees a small face that stays put in a corner, it switches the clip to the stacked layout instead. To lay out one clip that way yourself, use the free split screen video maker.

• A voice with no face, like an off-camera host or a caller. Tracking can only frame the faces it can see.
• Crosstalk. When two people talk over each other, it holds on one rather than flicking between them.
• Turned heads and wide shots. A face in profile, or a small one at the back of a stage, is harder to read.
• The reaction is the moment. Sometimes the listener's face is what the clip is about, and tracking frames the speaker.

Before you post, skip through each clip to where the speaker changes. Clyper's editor trims, extends and fixes captions but doesn't move the crop, so if a clip is framed on the wrong person, recut that part with the free vertical video converter.