Kamal Ali
ABSTRACT
Metrics such as click counts are vital to online businesses but their measurement has been problematic due to inclusion of high variance robot traffic. We posit that by applying statistical methods more rigorous than have been employed to date that we can build a robust model of thedistribution of clicks following which we can set probabilistically sound thresholds to address outliers and robots. Prior research in this domain has used inappropriate statistical methodology to model distributions and current industrial practice eschews this research for conservative ad-hoc click-level thresholds. Prevailing belief is that such distributions are scale-free power law distributions but using more rigorous statistical methods we find the best description of the data is instead provided by a scale-sensitive Zipf-Mandelbrot mixture distribution. Our results are based on ten data sets from various verticals in the Yahoo domain. Since mixture models can overfit the data we take care to use the BIC log-likelihood method which penalizes overly complex models. Using a mixture model in the web activity domain makes sense because there are likely multiple classes of users. In particular, we have noticed that there is a significantly large set of "users" that visit the Yahoo portal exactly once a day. We surmise these may be robots testing internet connectivity by pinging the Yahoo main website.
Backing up our quantitative analysis is graphical analysis in which empirical distributions are plotted against heoretical distributions in log-log space using robust cumulative distribution plots. This methodology has two advantages: plotting in log-log space allows one to visually differentiate the various exponential distributions and secondly, cumulative plots are much more robust to outliers. We plan to use the results of this work for applications for robot removal from web metrics business intelligence systems.
- G. Abdulla. Analysis and Modeling of World Wide Web Traffic. PhD thesis, Virginia Tech, 1998. Google Scholar
Digital Library
- A. Agresti. Categorical Data Analysis. Wiley Series in Probability and Statistics, 2002.Google Scholar
- L. Breslau, P. Cao, L. Fan, G. Phillips, and S. Shenker. Web caching and zipf-like distributions: Evidence and implications. In INFOCOM (1), pages 126--134, 1999.Google Scholar
- R. H. Byrd, J. Nocedal, and C. Zhu. A limited memory algorithm for bound constrained optimization. Journal of Scientific Computing (SIAM), 16:1190--1208, 1995. Google ScholarDigital Library
- G. Casella and R. L. Berger. Statistical Inference. Duxbury Press, 1990.Google Scholar
- U. Frisch and D. Sornette. Extreme deviation and applications. J. Phys. I France 7, 7:1155--1171, 1997.Google ScholarCross Ref
- S. Glassman. A caching relay for the World Wide Web. Computer Networks and ISDN Systems, 27(2):165--173, 1994. Google ScholarDigital Library
- D. C. Heilbron. Zero-altered and other regression models for count data with added zeroes. Biometrics, 36:531--547, 1994.Google ScholarCross Ref
- B. A. Huberman, P. L. T. Pirolli, J. E. Pitkow, and R. M. Lukose. Strong regularities in world wide web surfing. Science, 280:95--97, 1998.Google ScholarCross Ref
- J. Laherrere and D. Sornette. Stretched exponential distributions in nature and economy: "fat tails" with characteristic scales. The European Physical Journal B, 2:525, 1998.Google ScholarCross Ref
- D. Lambert. Zero-inflated poisson regression, with an application to defects in manufacturing. Technometrics, 34:1--14, 1992. Google ScholarDigital Library
- D. Lord, S. P. Washington, and J. N. Ivan. Poisson, poisson-gamma and zero-inflated regression models of motor vehicle crashes: balancing statistical fit and theory. Accident Analysis and Prevention, 37:35--46, 2005.Google ScholarCross Ref
- B. Mandelbrot. An informational theory of the statistical structure of language. In W. Jackson, editor, Communication Theory. Betterworths, 1953.Google Scholar
- S. M. Mwalili, E. Lesaffre, and D. Declerck. The zero-inflated negative binomial regression model with correction for misclassification: An example in caries research. Technical Report TR0462, IAP Statistics Network, 2005.Google Scholar
- R Development Core Team. R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria, 2004. ISBN 3-900051-00-3.Google Scholar
- J. A. Rice. Mathematical Statistics and Data Analysis. Wadsworth & Brooks/Cole, 1988.Google Scholar
- G. Schwarz. Estimating the dimension of a model. The Annals of Statistics, 6:461--464, 1978.Google ScholarCross Ref
- H. A. Simon. On a class of skew distribution functions. Biometrika, 42:425--440, 1955.Google ScholarCross Ref
- E. C. Titchmarsh. The Theory of the Riemann Zeta Function, 2nd ed. Oxford Science Publications, Clarendon Press, Oxford, 1986.Google Scholar
- D. G. Uitenbroek. SISA Pairwise tests. http://home.clara.net/sisa/pairwhlp.htm, 1997.Google Scholar
- D. von Seggern. CRC Standard Curves and Surfaces. CRC Press, 1993.Google Scholar
- J. R. Wilson. Logarithmic series distribution and its use in analyzing discrete data. In Proceedings of the Survey Research Methods Section, American Statistical Association, pages 275--280, 1988.Google Scholar
- G. K. Zipf. Human Behaviour and the Principle of Least-Effort. Addison-Wesley, Cambridge, MA, 1949.Google Scholar
Index Terms
- Robust methodologies for modeling web click distributions
Recommendations
Finding a simple probability distribution for human mobile speed
The speed of human mobility in a day is commonly known to have a representative probability distribution and previous studies have identified several useful distributions to represent it. In this study, we deduce conditions for a simple probability ...
Tail densities of skew-elliptical distributions
AbstractSkew-elliptical distributions constitute a large class of multivariate distributions that account for both skewness and a variety of tail properties. This class has simpler representations in terms of densities rather than cumulative ...
Fitting activity distributions using human partitioning and statistical calibration
Highlights- An extension to an existing calibration procedure for analysing empirical project data is presented.
AbstractMany project management and scheduling studies have modelled activity durations as a range of values to express the stochastic nature of projects in progress. A wide variety of simulation models have been proposed that all rely on pre-...
Comments