Glide: Easy ETL

Glide aims to be a simple, reusable, extensible approach to building ETL data pipelines. It provides a suite of nodes and pipelines out of the box that cover many common use cases, such as reading and writing data to/from SQL, URLs, local/remote files, and email.

Glide aims to have sane defaults and rely on standard, well-known Python libraries for data processing under the hood. It strives for familiar behavior out of the provided nodes while allowing for customization by passing arguments through to the underlying libraries in many cases. It’s also very easy to write completely custom nodes and pipelines.

Glide also tries to give you a lot for free, including simple parallel processing support, a variety of ways to manage node/pipeline context, and automatic CLI generation.

Glide encourages extensions that provide nodes and pipelines for interacting with various data sources and data processing modules. Check out the glide.extensions module to see some currently supported extensions, such as Dask for scalable analytics workflows.

• Quickstart
  • Introduction
  • Table of Contents
  • Installation
  • Primer
  • Basic Examples
    • Example: CSV Extract, Transform, and Load
    • Example: SQL Extract and Load
    • Example: SQL Transactions
    • Example: URL Extraction
  • Flow Control Examples
    • Example: Filters
    • Example: IterPush
    • Example: SplitPush
    • Example: SplitByNode
    • Example: Reduce
    • Example: Join
    • Example: Routers
    • Example: Window Processing
    • Example: Date Windows
    • Example: Return Values
  • Parallelization & Concurrency
    • Example: Parallel Transformation
    • Example: Parallel Pipelines via ParaGlider
    • Example: Parallel Branching
    • Example: Thread Reducers
    • Example: Asyncio
  • Utility Examples
    • Example: Templated Nodes and Pipelines
    • Example: Data Integrity Checks
    • Example: Debugging
    • Example: Profiling Pipelines
    • Example: Complex Pipelines
    • Example: Plotting Pipeline DAGs
  • CLI Generation
  • Extensions
    • Pandas
    • Dask - Experimental
    • Celery - Experimental
    • Redis Queue - Experimental
    • Swifter - Experimental
  • Documentation
  • How to Contribute
• Creating Nodes
• Node Context
• Runtime Context
• Config Context
• Cleaning Up
• Common Nodes
• Creating Pipelines
  • Complex Pipelines
• CLI Generation
  • Blacklisting Args
  • Custom Arguments
  • Argument Injection and Clean Up
  • Boolean Args
  • Parent CLIs
• Parallel Processing
• Common Pipelines
• Subpackages
  • glide.extensions package
    • Submodules
• Submodules
  • glide.core module
  • glide.extract module
  • glide.flow module
  • glide.filter module
  • glide.load module
  • glide.math module
  • glide.pipelines module
  • glide.sql module
  • glide.sql_utils module
  • glide.transform module
  • glide.utils module
  • glide.version module