Utilities

We provide several useful utilities to help with common tasks

Async and sync queries

To execute a query synchronously run the following:

from dremio_client import init
import pandas as pd
query = 'select * from sys.options'
client = init(simple_client=True)
results = client.query(self, query)
pd.DataFrame(results)

This will submit the sql query to the sql endpoint then poll the job-status endpoint until the job completed. If the job has succeeded a list of dicts is returned else it throws an exception.

One can execute many queries at once via async querys also:

from dremio_client import init
from concurrent.futures.thread import ThreadPoolExecutor
from concurrent.futures import as_completed

queries = ['select * from table0',
           'select * from table1',
           'select * from table2',
           'select * from table3',
           'select * from table4',
           'select * from table5'
           ]
client = init(simple_client=True)

def execute(query):
    client.query(query, asynchronous=True)

# example shamelessly taken from https://docs.python.org/3/library/concurrent.futures.html
with ThreadPoolExecutor(max_workers=4) as executor:
    futures = {executor.submit(execute, query): query for query in queries}
    for future in as_completed(futures):
        query = futures[future]
        try:
            result = future.result()
        except Exception as exc:
            print('%r generated an exception: %s' % (url, exc))
        else:
            print('%r returned %d rows' % (url, len(results)))