Chapter 5 Serial streak sampling

On the one hand, most theorists have included parallel position updating in their models. This seems to fit the data from MOT experiments with relatively-unconstrained object trajectories quite well - although few have compared the performance of their model with a serial model, so it is hard to know how diagnostic the modeled data are. On the other hand, the data from circular trajectories is hard to explain without positing that serial updating of position dominates. The accounts offered of this dichotomy seem unsatisfactory. However, there is an additional source of information that could assist tracking, one that would disabled by non-circular trajectories.

Long after most researchers considered the idea of serial updating of position to be disproven, Srimant Tripathy and his collaborators made a new case for the idea in a book chapter in 2011 (Tripathy, Ogmen, and Narasimhan 2011). Previous researchers had concluded that serial switching of attention would have to occur implausibly rapidly to account for tracking performance (Pylyshyn and Storm 1988; Yantis 1992; Oksama and Hyönä 2004). Tripathy, Ogmen, and Narasimhan (2011), however, pointed out that the demands on attention switching would be much less if the tracking process had the benefit of sensory memory traces left by the moving objects, also known as “motion streaks”. Motion streaks may be familiar from turning on a fan and watching it spin faster and faster until its blades blur perceptually into a complete circle - and in the photograph below, one sees the motion streaks created by a London bus, when photographed with a 200 ms exposure, which is likely comparable to the interval of motion streaks in our visual system.

A London bus, photographed with a 200 ms exposure, by EO1. CC BY-SA 2.0

Figure 5.1: A London bus, photographed with a 200 ms exposure, by EO1. CC BY-SA 2.0

At speeds like those used in MOT experiments, the moving objects leave trails of persisting activation in the visual cortex (e.g. Apthorp et al. 2013). Consider how the correspondence problem could be solved if the tracking process has access to these motion streaks, c. Rather than having access to only the most recently-registered position of each object, it would also “see” a short trail behind each object. If the rate of position sampling were frequent relative to the duration of visual persistence, then the motion streak would actually connect the last-registered position of an object with the current position sample. In most conditions, this should allow the tracker to avoid many of the confusions between targets and distractors that would otherwise occur. Recall from our explanation of the correspondence problem (1) that when only discrete positions are sampled, a distractor will be the closest object in the current sample to a target in the previous sample, causing the distractor to be mistakenly taken to be the target.

When motion streaks can be used, either the streak will literally connect the new sample to the old sample, or if the sampling happens less frequently relative to the targets’ speed, if the object hasn’t changed direction than the streaks will typically be collinear, whereas the distractors’ streaks will be positioned and oriented differently. The major exception to this would be when the target shared its trajectory with distractors, such as in the experiments of Alex O. Holcombe and Chen (2013).

The motion streaks created by billiard balls in a long exposure of the ball break. Public domain

Figure 5.2: The motion streaks created by billiard balls in a long exposure of the ball break. Public domain

Step by step, here’s how the motion streaks serial sampler could work, in an MOT experiment where the targets and distractors occupy different trajectories. When the sampler visits the last-known position of a target, that position may now be the site of the trailing end of the motion streak, which connects this last-known position of the target to its present position. The tracker can thus follow the streak to arrive at the new target position. The most up-to-date end of the streak is the one with the highest contrast (because it has had the least time to decay). If it has been so long since the sampler visited this target that there is no longer any activation at the last-visited location, there will typically be only a small gap between that location and the trailing end of the target’s current streak. The size of that gap will be the distance traveled by the object in the time between samples minus the duration of sensory persistence.

When, as in the Alex O. Holcombe and Chen (2013) experiments, the target moves about a trajectory that is shared by several distractors, the streaks will be much less helpful. When the serial sampler returns to the last known location of a target, if it does not return before the trailing distractor reaches that location, a new streak will be there, that of the distractor. Thus, this account predicts that in the trajectory-sharing displays, tracking will only succeed if the time between object visits to a location is longer than the time between samples. That is, success at tracking is now limited by serial switching time. This nicely converges with theorists’ predilection for proposing a different mechanism for trajectory-sharing displays, even if they have had the wrong idea for why such displays bear the signature of serial switching.

One might be skeptical that motion streaks are available to visual processes such as tracking - after all, we don’t usually perceive motion streaks when objects are moving at reasonable speeds. But there is good evidence that motion streaks do affect visual processing. Psychophysical studies have found evidence for perceptual adaptation to the streaks left by moving objects, and also that these streaks mask the visibility of static objects. It has been estimated that these streaks span about 100 ms of a moving object’s trajectory, and there is also plenty of physiological evidence for them - see Apthorp, Cass, and Alais (2011) for references. Moreover, recent evidence suggests that the elongated motion streaks created by saccades are used to solve the across-saccade object correspondence problem (Schweitzer and Rolfs 2020, 2021). As for why we don’t see motion streaks in many viewing conditions, this is not fully understood, but nearby objects appear to actively suppress their visibility (for a review, see Bedell, Tong, and Aydin 2010).

Tripathy’s group conducted a number of studies to investigate the possibility that motion streak information is available. This largely took the form of varying delays between stimulus presentation and cues, showing that participants could report motion direction and detect trajectory changes fairly precisely immediately after objects offset, but that this ability decayed very quickly with time (Narasimhan, Tripathy, and Barrett 2009; Shooner et al. 2010). The rapid decay was consistent with the time-course of the dissipation of sensory activation, as evident in studies of iconic memory.

In the previous chapter (4), when we inferred what the interval between samples needed to successfully track with the circular trajectory displays, we were not considering streaks, so we used the conventional nearest-neighbor interpretation of motion ambiguity, which implies that sampling must occur more frequently than the time it takes for a distractor to move halfway to a former target location (Alex O. Holcombe et al. 2005). But motion streaks connect the last-sampled target position to its new position, up until the distractor moves so far that it actually touches the former target location. This is simply the temporal frequency of the stimulus. For example, for the case of three targets where a temporal limit of less than 3 Hz is observed (Alex O. Holcombe and Chen 2013; Roudaia and Faubert 2017), we had estimated that samples must occur at least every 167 ms, but motion streaks mean that in principle, the samples need only occur every 332 milliseconds. However, a sizeable gap may be needed to resolve the difference between the front of the motion streak created by the distractor and the trailing end of the motion streak created by the target, so sampling more frequently may be necessary.

5.1 Taking stock

3 GROUPS OF RESERACHERS HAVE INDEPENDENTLY ENDED UP INVOKING SOMETHING FUNCTIONALLY RELATED TO STREAKS. Wang & Vul (2021) did it with motion direction being used to resolve correspondences. Lovett et al. (2019) did it with “motion history”. Tripathy and also Schweizer and Rolfs do it with motion streaks.

When Srimant Tripathy suggested the serial sampling of motion streaks idea, we had not yet discovered the decline with load of tracking’s temporal limit. At the time, I did not fully understand Srimant’s account, and did not appreciate that the account both predicts this sort of result and also predicts that a quite different result should be found with trajectories that do not highly overlap. It was only when grappling with the suggestion of Lovett, Bridewell, and Bello (2019) that when distractors and target share trajectories, tracking becomes more serial, that the connection to motion streaks became clear to me. The notion of serial position sampling in tracking converges with the ascendance of evidence for serial sampling in other behavioral and neuroscientific paradigms, and presents a strong prospect for resolving the serial-parallel debate.

The use of motion streaks provides a similar benefit as the use of velocity as a feature for correspondence matching, and studies that investigated the possible use of velocity do not seem to have been designed to discriminate between the use of velocity and the use of motion streaks (e.g., Wang and Vul (2021)). One difference is that streaks might provide information only about the orientation of an object’s trajectory but not which of the two directions along that orientation the object is moving in. If so, one result should be that participants should more frequently err in the opposite direction that a target is moving than in other directions. It does not appear that studies that collected participants’ awareness of direction information checked for this (e.g. Shooner et al. 2010; Horowitz and Cohen 2010).


Apthorp, Deborah, John Cass, and David Alais. 2011. “The Spatial Tuning of ‘Motion Streak’ Mechanisms Revealed by Masking and Adaptation.” Journal of Vision 11 (7): 17–17.
Apthorp, Deborah, D. Samuel Schwarzkopf, Christian Kaul, Bahador Bahrami, David Alais, and Geraint Rees. 2013. “Direct Evidence for Encoding of Motion Streaks in Human Visual Cortex.” Proceedings of the Royal Society B: Biological Sciences 280 (1752): 20122339.
Bedell, Harold E., Jianliang Tong, and Murat Aydin. 2010. “The Perception of Motion Smear During Eye and Head Movements.” Vision Research 50 (24): 2692–2701.
Holcombe, Alex O, and Wei-ying Chen. 2013. “Splitting Attention Reduces Temporal Resolution from 7 Hz for Tracking One Object to \(<\)3 Hz When Tracking Three.” Journal of Vision 13 (1): 1–19.
Holcombe, Alex O., Colin WG Clifford, David M. Eagleman, and Pooya Pakarian. 2005. “Illusory Motion Reversal in Tune with Motion Detectors.” Trends in Cognitive Sciences 9 (12): 559–59.
Horowitz, T. S., and M. A. Cohen. 2010. “Direction Information in Multiple Object Tracking Is Limited by a Graded Resource.” Attention, Perception & Psychophysics 72 (7): 1765–75.
Lovett, Andrew, Will Bridewell, and Paul Bello. 2019. “Selection Enables Enhancement: An Integrated Model of Object Tracking.” Journal of Vision 19 (14): 23.
Narasimhan, Sathyasri, Srimant P. Tripathy, and Brendan T. Barrett. 2009. “Loss of Positional Information When Tracking Multiple Moving Dots: The Role of Visual Memory.” Vision Research 49 (1): 10–27.
Oksama, Lauri, and Jukka Hyönä. 2004. “Is Multiple Object Tracking Carried Out Automatically by an Early Vision Mechanism Independent of Higher-Order Cognition? An Individual Difference Approach.” Visual Cognition 11 (5): 631–71.
Pylyshyn, Z W, and R W Storm. 1988. “Tracking Multiple Independent Targets: Evidence for a Parallel Tracking Mechanism.” Spatial Vision 3 (3): 179–97.
Roudaia, Eugenie, and Jocelyn Faubert. 2017. “Different Effects of Aging and Gender on the Temporal Resolution in Attentional Tracking.” Journal of Vision 17 (11): 1.
Schweitzer, Richard, and Martin Rolfs. 2020. “Intra-Saccadic Motion Streaks as Cues to Linking Object Locations Across Saccades.” Journal of Vision 20 (4): 17.
———. 2021. “Intrasaccadic Motion Streaks Jump-Start Gaze Correction.” Science Advances 7 (30): eabf2218.
Shooner, Christopher, Srimant P Tripathy, Harold E Bedell, and Haluk Ogmen. 2010. High-Capacity, Transient Retention of Direction-of-Motion Information for Multiple Moving Objects. Journal of Vision 10 (6): 8.
Tripathy, S P, H Ogmen, and S Narasimhan. 2011. “Multiple-Object Tracking: A Serial Attentional Process?” In Attention: Philosophical and Psychological Essays., edited by Declan Smithies Christopher Mole and Wayne Wu, 117–44. Oxford University Press.
Wang, Yang, and Edward Vul. 2021. “The Role of Kinematic Properties in Multiple Object Tracking.” Journal of Vision 21 (3): 22–22.
Yantis, S. 1992. Multielement Visual Tracking: Attention and Perceptual Organization. Cognitive Psychology 24 (3): 295–340.