glide.load module

A home for common data load nodes

Nodes:

• CSVLoad

• ExcelLoad

• SQLLoad

• SQLTempLoad

• FileLoad

• URLLoad

• EmailLoad

• Print

• PrettyPrint

• LenPrint

• ReprPrint

• FormatPrint

class glide.load.CSVLoad(name, _log=False, _debug=False, **default_context)

Bases: glide.flow.SkipFalseNode

Load data into a CSV using DictWriter

begin()

Initialize state for CSV writing

end()

Reset state in case the node gets reused

run(rows, f, push_file=False, dry_run=False, **kwargs)

Use DictWriter to output dict rows to a CSV.

Parameters

• rows – Iterable of rows to load to a CSV

• f (file or buffer) – File to write rows to

• push_file (bool, optional) – If true, push the file forward instead of the data

• dry_run (bool, optional) – If true, skip actually loading the data

• **kwargs – Keyword arguments passed to csv.DictWriter

class glide.load.EmailLoad(name, _log=False, _debug=False, **default_context)

Bases: glide.core.Node

Load data to email via SMTP

run(data, frm=None, to=None, subject=None, body=None, html=None, attach_as='attachment', attachment_name=None, formatter=None, client=None, host=None, port=None, username=None, password=None, dry_run=False)

Load data to email via SMTP.

Parameters

• data – EmailMessage or data to send. If the latter, the message will be created from the other node arguments.

• frm (str, optional) – The from email address

• to (str or list, optional) – A str or list of destination email addresses

• subject (str, optional) – The email subject

• body (str, optional) – The email text body

• html (str, optional) – The email html body

• attach_as (str) – Where to put the data in the email message if building the message from node arguments. Options: attachment, body, html.

• attachment_name (str, optional) – The file name to write the data to when attaching data to the email. The file extension will be used to infer the mimetype of the attachment. This should not be a full path as a temp directory will be created for this.

• formatter (callable) – A function to format and return a string from the input data if attach_as is set to “body” or “html”.

• client (optional) – A connected smtplib.SMTP client

• host (str, optional) – The SMTP host to connect to if no client is provided

• port (int, optional) – The SMTP port to connect to if no client is provided

• username (str, optional) – The SMTP username for login if no client is provided

• password (str, optional) – The SMTP password for login if no client is provided

• dry_run (bool, optional) – If true, skip actually loading the data

class glide.load.ExcelLoad(name, _log=False, _debug=False, **default_context)

Bases: glide.flow.SkipFalseNode

Load data into an Excel file using pyexcel

run(rows, f, dict_rows=False, sheet_name='Sheet1', push_file=False, dry_run=False, **kwargs)

Use DictWriter to output dict rows to a CSV.

Parameters

• rows – Iterable of rows to load to an Excel file, or a dict of sheet_name->iterable for multi-sheet loads.

• f (file or buffer) – File to write rows to

• dict_rows (bool, optional) – If true the rows of each sheet will be converted from dicts to lists

• sheet_name (str, optional) – Sheet name to use if input is an iterable of rows. Unused otherwise.

• push_file (bool, optional) – If true, push the file forward instead of the data

• dry_run (bool, optional) – If true, skip actually loading the data

• **kwargs – Keyword arguments passed to pyexcel

class glide.load.FileLoad(name, _log=False, _debug=False, **default_context)

Bases: glide.core.Node

Load raw content to a file

run(data, f, open_flags='w', push_file=False, dry_run=False)

Load raw data to a file or buffer

Parameters

• data – Data to write to file

• f (file path or buffer) – File path or buffer to write

• open_flags (str, optional) – Flags to pass to open() if f is not already an opened buffer

• push_file (bool) – If true, push the file forward instead of the data

• dry_run (bool, optional) – If true, skip actually loading the data

class glide.load.FormatPrint(name, _log=False, _debug=False, **default_context)

Bases: glide.core.Node

Format and print the data

run(data, label=None, indent=None, color=None, autocolor=False, format_func='pf')

Format using tlx.format_msg, then print

Parameters

• data – The data to print

• **kwargs – Keyword arguments passed to tlx.format_msg

class glide.load.LenPrint(name, _log=False, _debug=False, **default_context)

Bases: glide.load.Print

Prints the length of the data

get_label()

Get a label for the print statement

print(data)

Print the data

class glide.load.PrettyPrint(name, _log=False, _debug=False, **default_context)

Bases: glide.load.Print

Pretty-prints the data

print(data)

Print the data

class glide.load.Print(name, _log=False, _debug=False, **default_context)

Bases: glide.core.Node

Print the data

get_label()

Get a label for the print statement

print(data)

Print the data

run(data, label=True)

Print the data with the printer function and push

class glide.load.ReprPrint(name, _log=False, _debug=False, **default_context)

Bases: glide.load.Print

Prints the reprlib.repr of the data

print(data)

Print the data

class glide.load.SQLLoad(*args, **kwargs)

Bases: glide.sql.SQLNode

Generic SQL loader

run(rows, conn, table, cursor=None, commit=True, rollback=False, stmt_type='REPLACE', odku=False, swap=False, keep_old=False, push_data=False, dry_run=False)

Form SQL statement and use bulk execute to write rows to table

Parameters

• rows – Iterable of rows to load to the table

• conn – Database connection

• table (str) – Name of a table to write the data to

• cursor (optional) – Database connection cursor

• commit (bool, optional) – If true try to commit the transaction. If your connection autocommits this will have no effect. If this is a SQLAlchemy connection and you are in a transaction, it will try to get a reference to the current transaction and call commit on that.

• rollback (bool, optional) – If true try to rollback the transaction on exceptions. Behavior may vary by backend DB library if you are not currently in a transaction.

• stmt_type (str, optional) – Type of SQL statement to use (REPLACE, INSERT, etc.). Note: Backend support for this varies.

• odku (bool or list, optional) – If true, add ON DUPLICATE KEY UPDATE clause for all columns. If a list then only add it for the specified columns. Note: Backend support for this varies.

• swap (bool, optional) – If true, load a table and then swap it into the target table via rename. Not supported with all database back ends.

• keep_old (bool, optional) – If true and swapping tables, keep the original table with a __old suffix added to the name

• push_data (bool, optional) – If true, push the data forward instead of the table name

• dry_run (bool, optional) – If true, skip actually loading the data

class glide.load.SQLTempLoad(*args, **kwargs)

Bases: glide.sql.SQLNode

Generic SQL temp table loader

run(rows, conn, cursor=None, schema=None, commit=True, rollback=False, dry_run=False)

Create and bulk load a temp table

Parameters

• rows – Iterable of rows to load to the table

• conn – Database connection

• cursor (optional) – Database connection cursor

• schema (str, optional) – Schema to create temp table in

• commit (bool, optional) – If true try to commit the transaction. If your connection autocommits this will have no effect. If this is a SQLAlchemy connection and you are in a transaction, it will try to get a reference to the current transaction and call commit on that.

• rollback (bool, optional) – If true try to rollback the transaction on exceptions. Behavior may vary by backend DB library if you are not currently in a transaction.

• dry_run (bool, optional) – If true, skip actually loading the data

class glide.load.URLLoad(name, _log=False, _debug=False, **default_context)

Bases: glide.core.Node

Load data to URL with requests

run(data, url, data_param='data', session=None, skip_raise=False, dry_run=False, **kwargs)

Load data to URL using requests and push response.content. The url maybe be a string (POST that url) or a dictionary of args to requests.request:

http://2.python-requests.org/en/master/api/?highlight=get#requests.request

Parameters

• data – Data to load to the URL

• url (str or dict) – If str, a URL to POST to. If a dict, args to requets.request

• data_param (str, optional) – parameter to stuff data in when calling requests methods

• session (optional) – A requests Session to use to make the request

• skip_raise (bool, optional) – if False, raise exceptions for bad response status

• dry_run (bool, optional) – If true, skip actually loading the data

• **kwargs – Keyword arguments to pass to the request method. If a dict is passed for the url parameter it overrides values here.