Modeling Traffic on the Web Graph

Mark R. Meiss1,3, Bruno Gonçalves1,2,3, José J. Ramasco4, Alessandro Flamini1,2 and Filippo Menczer1,2,3,4
1School of Informatics, Indiana University, Bloomington IN, USA.
2Center for Complex Networks and Systems Research, Indiana University, Bloomington, USA
3Pervasive Technology Institute, Indiana University, Bloomington, USA
4Complex Networks and Systems Lagrange Laboratory, CNLL, ISI Foundation, Turin, Italy

(Sep 2010)

Analysis of aggregate and individual Web requests shows that Page- Rank is a poor predictor of traffic. We use empirical data to characterize properties of Web traffic not reproduced by Markovian models, including both aggregate statistics such as page and link traffic, and individual statistics such as entropy and session size. As no current model reconciles all of these observations, we present an agent-based model that explains them through realistic browsing behaviors: (1) revisiting bookmarked pages; (2) backtracking; and (3) seeking out novel pages of topical interest. The resulting model can reproduce the behaviors we observe in empirical data, especially heterogeneous session lengths, reconciling the narrowly focused browsing patterns of individual users with the extreme variance in aggregate traffic measurements. We can thereby identify a few salient features that are necessary and sufficient to interpret Web traffic data. Beyond the descriptive and explanatory power of our model, these results may lead to improvements in Web applications such as search and crawling.