(Translated by https://www.hiragana.jp/)
[2009.08150] Extensible Data Skipping
Skip to main content
We gratefully acknowledge support from the Simons Foundation, member institutions, and all contributors. Donate
arXiv logo
Cornell University Logo

Computer Science > Databases

arXiv:2009.08150 (cs)
[Submitted on 17 Sep 2020 (v1), last revised 15 Nov 2020 (this version, v2)]

Title:Extensible Data Skipping

Authors:Paula Ta-Shma, Guy Khazma, Gal Lushi, Oshrit Feder
View PDF
Abstract:Data skipping reduces I/O for SQL queries by skipping over irrelevant data objects (files) based on their metadata. We extend this notion by allowing developers to define their own data skipping metadata types and indexes using a flexible API. Our framework is the first to natively support data skipping for arbitrary data types (e.g. geospatial, logs) and queries with User Defined Functions (UDFs). We integrated our framework with Apache Spark and it is now deployed across multiple products/services at IBM. We present our extensible data skipping APIs, discuss index design, and implement various metadata indexes, requiring only around 30 lines of additional code per index. In particular we implement data skipping for a third party library with geospatial UDFs and demonstrate speedups of two orders of magnitude. Our centralized metadata approach provides a x3.6 speed up even when compared to queries which are rewritten to exploit Parquet min/max metadata. We demonstrate that extensible data skipping is applicable to broad class of applications, where user defined indexes achieve significant speedups and cost savings with very low development cost.
Subjects: Databases (cs.DB)
Cite as: arXiv:2009.08150 [cs.DB]
  (or arXiv:2009.08150v2 [cs.DB] for this version)
  https://doi.org/10.48550/arXiv.2009.08150

Submission history

From: Guy Khazma [view email]
[v1] Thu, 17 Sep 2020 08:34:51 UTC (1,196 KB)
[v2] Sun, 15 Nov 2020 14:16:02 UTC (1,135 KB)
Full-text links:

Access Paper:

view license
Current browse context:
cs.DB
< prev   |   next >
new | recent | 2020-09
Change to browse by:
cs

References & Citations

DBLP - CS Bibliography

listing | bibtex
Paula Ta-Shma
export BibTeX citation

Bookmark

BibSonomy logo Reddit logo

Bibliographic and Citation Tools

Bibliographic Explorer (What is the Explorer?)
Litmaps (What is Litmaps?)
scite Smart Citations (What are Smart Citations?)

Code, Data and Media Associated with this Article

CatalyzeX Code Finder for Papers (What is CatalyzeX?)
DagsHub (What is DagsHub?)
Gotit.pub (What is GotitPub?)
Papers with Code (What is Papers with Code?)
ScienceCast (What is ScienceCast?)

Demos

Replicate (What is Replicate?)
Hugging Face Spaces (What is Spaces?)
TXYZ.AI (What is TXYZ.AI?)

Recommenders and Search Tools

Influence Flower (What are Influence Flowers?)
Connected Papers (What is Connected Papers?)
CORE Recommender (What is CORE?)

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)