glide.sql module

class glide.sql.AssertSQL(*args, **kwargs)

Bases: glide.sql.SQLNode

run(data, sql, conn, cursor=None, cursor_type=None, params=None, data_check=None, **kwargs)

Run a SQL query to check data.

Parameters

• data – Data to pass through on success

• sql (str) – SQL query to run. Should return a single row with a “assert” column to indicate success. Truthy values for “assert” will be considered successful, unless data_check is passed in which case it will be compared for equality to the result of that callable.

• conn – SQL connection object

• cursor (optional) – SQL connection cursor object

• cursor_type (optional) – SQL connection cursor type when creating a cursor is necessary

• params (tuple or dict, optional) – A tuple or dict of params to pass to the execute method

• data_check (callable, optional) – A callable that will be passed the node and data as arguments and is expected to return a value to be compared to the SQL result.

• **kwargs – Keyword arguments pushed to the execute method

class glide.sql.BaseSQLNode(*args, **kwargs)

Bases: glide.flow.SkipFalseNode

Base class for SQL-based nodes, checks for valid connection types on init

allowed_conn_types

A list or tuple of connection types that are allowed

Type

list or tuple

allowed_conn_types = None

begin()

check_conn(conn)

Check the database connection

commit(obj)

Commit any currently active transactions

create_like(conn, cursor, table, like_table, drop=False)

Create a table like another table, optionally trying to drop table first

drop_table(conn, cursor, table)

Drop tables all day long

execute(conn, cursor, sql, params=None, **kwargs)

Executes the sql statement and returns an object that can fetch results

Parameters

• conn – A SQL database connection object

• cursor – A SQL database cursor

• sql (str) – A sql query to execute

• params (tuple, optional) – A tuple of params to pass to the execute method of the conn or cursor

• **kwargs – kwargs passed through to execute()

Returns

cursor object that has executed but not fetched a query.

Return type

cursor

executemany(conn, cursor, sql, rows)

Bulk executes the sql statement and returns an object that can fetch results

Parameters

• conn – A SQL database connection object

• cursor – A SQL database cursor

• sql (str) – A sql query to execute

• rows – Rows of data to bulk execute

Returns

cursor object that has executed but not fetched a query.

Return type

cursor

get_bulk_statement(conn, stmt_type, table, rows, odku=False)

Get a bulk execution SQL statement

Parameters

• conn – A SQL database connection object

• stmt_type (str) – Type of SQL statement to use (REPLACE, INSERT, etc.)

• table (str) – name of a SQL table

• rows – An iterable of dict rows. The first row is used to determine column names.

• odku (bool or list, optional) – If true, add ON DUPLICATE KEY UPDATE clause for all columns. If a list then only add it for the specified columns. Note: Backend support for this varies.

Returns

Return type

A SQL bulk load query of the given stmt_type

get_sql_executor(conn, cursor_type=None)

Get the object that can execute queries

rename_tables(conn, cursor, renames)

Execute one or more table renames

rollback(obj)

Rollback any currently active transactions

transaction(conn, cursor=None)

Start a transaction. If conn is a SQLAlchemy conn return a reference to the transaction object, otherwise just return the conn which should have commit/rollback methods.

class glide.sql.SQLCursorPushMixin

Bases: object

Shared logic for SQL cursor-based nodes

do_push(cursor, chunksize=None)

Fetch data and push to the next node, obeying chunksize if passed

Parameters

• cursor – A cursor-like object with fetchmany and fetchall methods

• chunksize (int, optional) – If truthy the data will be fetched and pushed in chunks

class glide.sql.SQLExecute(*args, **kwargs)

Bases: glide.sql.SQLNode

run(sql, conn, cursor=None, cursor_type=None, params=None, commit=True, rollback=False, dry_run=False, **kwargs)

Perform a generic SQL query execution and push the cursor/execute response.

Parameters

• sql (str) – SQL query to run

• conn – SQL connection object

• cursor (optional) – SQL connection cursor object

• cursor_type (optional) – SQL connection cursor type when creating a cursor is necessary

• params (tuple or dict, optional) – A tuple or dict of params to pass to the execute method

• commit (bool, optional) – If true try to commit the transaction. If your connection autocommits this will have no effect. If this is a SQLAlchemy connection and you are in a transaction, it will try to get a reference to the current transaction and call commit on that.

• rollback (bool, optional) – If true try to rollback the transaction on exceptions. Behavior may vary by backend DB library if you are not currently in a transaction.

• **kwargs – Keyword arguments pushed to the execute method

class glide.sql.SQLFetch(name, _log=False, _debug=False, **default_context)

Bases: glide.core.Node, glide.sql.SQLCursorPushMixin

run(cursor, chunksize=None)

Fetch data from the cursor and push the result.

Parameters

• cursor – A cursor-like object that can fetch results

• chunksize (int, optional) – Fetch and push data in chunks of this size

class glide.sql.SQLNode(*args, **kwargs)

Bases: glide.sql.BaseSQLNode, glide.sql.SQLCursorPushMixin

A generic SQL node that will behave differently based on the connection type

allowed_conn_types = [<class 'object'>]

check_conn(conn)

Make sure the object is a valid SQL connection

class glide.sql.SQLTransaction(*args, **kwargs)

Bases: glide.sql.SQLNode

run(data, conn, cursor=None)

Begin a SQL transaction on the connection

Parameters

• data – Data being passed through the pipeline

• conn – Database connection to start the transaction on

• cursor (optional) – SQL connection cursor object