Partitioning techniques for fine-grained indexing

11 years 7 months ago
Partitioning techniques for fine-grained indexing
— Many data-intensive websites use databases that grow much faster than the rate that users access the data. Such growing datasets lead to ever-increasing space and performance overheads for maintaining and accessing indexes. Furthermore, there is often considerable skew with popular users and recent data accessed much more frequently. These observations led us to design Shinobi, a system which uses horizontal partitioning as a mechanism for improving query performance to cluster the physical data, and increasing insert performance by only indexing data that is frequently accessed. We present database design algorithms that optimally partition tables, drop indexes from partitions that are infrequently queried, and maintain these partitions as workloads change. We show a 60× performance improvement over traditionally indexed tables using a real-world query workload derived from a traffic monitoring application
Eugene Wu 0002, Samuel Madden
Added 21 Aug 2011
Updated 21 Aug 2011
Type Journal
Year 2011
Where ICDE
Authors Eugene Wu 0002, Samuel Madden
Comments (0)