People perform complex visual tasks. Airplane pilots land planes safely on the ground and baseball players swing bats at speeding fastballs. Drivers weave through traffic and sports fans skillfully track the movements of their favorite team. These are examples of visual search, the process of looking for something. Classic experiments have provided much information about characteristics affecting search efficiency (i.e., efficiency =display size/speed; Treisman & Gelade, 1980), but visual search literature is split on the underlying mechanisms involved in visual search. Visual search may be random (Wolfe, 2007), memory-driven (Zelinsky, 2008), or self-similar over time (Aks, Zelinsky, & Sprott, 2002). These standpoints assign memory at least some role in determining search behavior-the current work explores this possibility by looking for evidence of nonlinearity in visual search response times. Participants performed 250 visual search trials in one of three conditions, ascending-first, descending-first, or random. Ascending-first participants performed 125 searches increasing in difficulty, then 125 searches decreasing in difficulty. Descending-first participants completed 125 searches decreasing in difficulty, then 125 searches increasing in difficulty. Random participants completed 250 searches pseudo-randomly varying in difficulty. We constructed hysteresis plots for each condition and nonlinearity emerged in the data that does not fit traditional concepts of memory, practice, and fatigue. The findings suggest that the term memory may not be a useful concept for describing the visual search process. Hysteresis in visual behavior indicates history-dependence-we suggest the term history as a replacement for memory.