Remembering what we like: Toward an agent-based model of Web traffic

Bruno Gonçalves1, Mark R. Meiss1,2, José J. Ramasco3, Alessandro Flamini1 and Filippo Menczer1,3
1School of Informatics, Indiana University, Bloomington IN, USA.
2Advance Network Management Lab, Indiana University, Bloomington IN, USA.
3Complex Networks and Systems Lagrange Laboratory, CNLL, ISI Foundation, Turin, Italy.

(Jan 2009)

Analysis of aggregate Web traffic has shown that PageRank is a poor model of how people actually navigate the Web. Using the empirical traffic patterns generated by a thousand users over the course of two months, we characterize the properties of Web traffic that cannot be reproduced by Markovian models, in which destinations are independent of past decisions. In particular, we show that the diversity of sites visited by individual users is smaller and more broadly distributed than predicted by the PageRank model; that link traffic is more broadly distributed than predicted; and that the time between consecutive visits to the same site by a user is less broadly distributed than predicted. To account for these discrepancies, we introduce a more realistic navigation model in which agents maintain individual lists of bookmarks that are used as teleportation targets. The model can also account for branching, a traffic property caused by browser features such as tabs and the back button. The model reproduces aggregate traffic patterns such as site popularity, while also generating more accurate predictions of diversity, link traffic, and return time distributions. This model for the first time allows us to capture the extreme heterogeneity of aggregate traffic measurements while explaining the more narrowly focused browsing patterns of individual users.