Building Scalable Data Collection

By mock from Victoria.pm
Date: Thursday, 30 August 2007 14:35
Duration: 20 minutes
Target audience: Any
Language:

Scaling the distribution of content to many users is mostly a well understood problem, but its opposite, scaling the collection of data from many users to a data warehouse has many challenges that have not been adequately solved using commodity hardware and software. This talk will show you some strategies for collecting data really fast (> 1000 entries per second) and how you can reuse some of the techniques and tools used to serve data, to collect it. Attention will be paid to the various event based frameworks and their performance, working with database partitioning, and archiving your data so that it can be easily mined later.