Index

Module Index

Search Page

qnd.frontend module

Quick and Dirty, a high level file access wrapper.

The QnD interface looks like this:

f = format_specific_open(filename, mode)  # e.g. openh5
var = f.varname  # read var
var = f['varname']
var = f.get('varname', default)

f.var = var_value  # declare and write var
f['var'] = something
f.var = dtype, shape  # declare var without writing
f.update({...vars...}, another_var=value, ...)
f.grpname = {...vars...}  # declare a subgroup and some members

if name in f: do_something
varnames = list(f)
for name in f: do_something
for name, var in f.items(): do_something

g = f.grpname
f = g.root()  # Use the root method to get the top-level QGroup.
f.close()  # important if you have written to f

Generally, a QnD QGroup like f in the example behaves like a dict. However, you may also reference variables or subgroups as if they were attributes. Use attributes to access variables when you know the variable name. In short, use square brackets when the variable name is the value of an expression. (QnD will remove a single trailing underscore from any attribute reference, so you can use f.yield_ for f['yield'] or f.items_ for f['items'].) The adict module has an ADict class and a redict function to produce ordinary in-memory dict objects with their items accessible as attributes with the same rules. You can read a whole file (or a whole subgroup) like this:

ff = f(2)

The optional 2 argument is the auto-read mode flag. By default, the auto-read mode flag is set to 1, which causes f.varname to read an array variable and return its value, but to simply return a QGroup object (like f) if the name refers to a subgroup. When the auto flag equals 2, any subgroups are read recursively, and their values become ADict instances. (QnD also supports QList variables, and auto=2 mode returns those as python list instances.)

The items() method also accepts an optional auto argument to temporarily change auto-read mode used for the iteration.

You can turn auto-read mode off by setting the auto flag to 0. In this mode, referencing a variable returns a QLeaf instance without reading it. This enables you to query a variable without reading it. You can also do that by retrieving the attributes object:

with f.push():  # Use f as a context manager to temporarily change modes.
    f.auto(0)  # Turn off auto-read mode.
    v = f.varname
value = v()  # Read a QLeaf by calling it...
value = v[:]  # ...or by indexing it.
v(value)  # Write a QLeaf by calling it with an argument...
v[:] = value  # ...or by setting a slice.
v.dtype, v.shape, v.size, v.ndim  # properties of the QLeaf v
# An alternate method which pays no attention to auto mode:
va = f.attrs.varname  # Get attributes of varname.
va.dtype, va.shape, va.size, va.ndim  # Built-in pseudo-attributes.
# You can use va to get or set real attributes of varname as well:
units = va.units  # retrieve units attribute
va.centering = 1  # set centering attribute

When you call a QGroup like f as a function, you may also pass it a list of variable names to read only that subset of variables. With auto-read mode turned off, this results in a sort of “casual subgroup”:

g = f(0, 'vname1', 'vname2', ...)
h = f(1, 'vname1', 'vname2', ...)
ff = f(2, 'vname1', 'vname2', ...)

Here, g is an ADict containing QLeaf and QGroup objects, with nothing at all read from the file, while h is and ADict containing ndarray and QGroup objects, while ff is an ADict containing ndarray and ADict objects, with no references at all to f.

If you want to use f as a context manager in the manner of other python file handles, so that the file is closed when you exit the with statement, just do it:

with openh5(filename, "a") as f:
    do_something(f)
# f has been properly flushed and closed on exit from the with.

---

QnD also supports old netCDF style UNLIMITED dimensions, and their equivalents in HDF5. Unlike the netCDF or HDF5 interface, in QnD the first (slowest varying) dimension of these arrays maps to a python list, so we regard the entire collected variable as a list of ndarrays. The netCDF record number is the index into the list, while any faster varying dimensions are real ndarray dimensions. This subtle difference in approach is more consistent with the way these variables are stored, and also generalizes to the fairly common case that the array dimensions – often mesh dimensions – change from one record to the next.

To write records using QnD, turn on “recording mode”:

f.recording(1)  # 0 for off, 2 for generalized records
f.time = 0.
f.x = x = arange(10)
f.time = 0.5
f.x = x**2

Ordinarily, when you set the value of f.time or f.x, any previous value will be overwritten. But in recording mode, each time you write a variable, you create a new record, saving the new value without overwriting the previous value. If you want all record variables to have the same number of records, you need to be sure you write them each the same number of times. One way to do that is to use the update function rather than setting them one at a time:

record = ADict()
record.time, record.x = 0., arange(10)
f.recording(1)
f.update(record)
record.time, record.x = 0.5, record.x**2
f.update(record)

You cannot change a variable from not having records to having records (or from recording mode 1 to recording mode 2); the recording mode in force when a variable was first declared determines if and how all future write operations behave.

Reading back record variables introduces “goto mode”. Initially, goto mode is off or None, so that reading a record variable gets the whole collection of values as a QList, or as an ordinary python list if auto mode is on:

f.goto(None)  # explicitly turn off goto mode
f.auto(2)
times = f.time  # python list of f.time values
xs = f.x  # python list of f.x arrays
f.auto(0)
time = f.time  # QList for the collection of time values
nrecords = len(time)

On the other hand, with goto mode turned on, the fact that time and x are record variables disappears, so that your view of f.time and f.x matches what it was when you recorded them. You use the goto function to set the record:

f.goto(0)  # first record is 0, like any python list
t = f.time  # == 0.
f.goto(1)  # set to second record
t = f.time  # == 0.5
x = f.x  # == arange(10)**2
f.goto(-1)  # final record, negative index works like any python list
# You can also pass a keyword to goto, which can be the name of any
# scalar record variable, to go to the record nearest that value.
f.goto(time=0.1)  # will select record 0 here
current_record = f.goto()  # goto() returns current record number

for r in f.gotoit(): do_something  # f.goto(r) is set automatically

Note the gotoit() method returns an iterator over all records, yielding the record number for each pass, and setting the goto record for each pass automatically. You can use f.push() in a with statement to temporarily move to a different record.

If you set the recording mode to 2, the record variables need not have the same shape or same type from one record to the next (indeed, they can be a subgroup on one record and an array on another). This cannot be represented as an UNLIMITED array dimension in an HDF5 or netCDF file, so the QList variable in QnD will become an HDF5 group in this case, where variable names in the group are _0, _1, _2, and so on for QList element 0, 1, 2, and so on (plus a hidden element _ which identifies this group as a list when it is empty). You can create a QList of this general type without using recording or goto mode at all:

f.recording(0)  # Turn off recording and goto modes.
f.goto(None)
f.varname = list  # Make an empty QList
ql = f.varname
ql.append(value0)
ql.extend([value1, value2, ...])
var = ql[1]  # retrieves value1
nelements = len(ql)  # current number of elements (also works for QGroup)
ql.auto(0)  # a QList has auto mode just like a QGroup
for var in ql: do_something  # var depends on ql auto mode setting

---

class qnd.frontend.QAttributes(parent, vname=None)

Bases: qnd.adict.ItemsAreAttrs

Attributes for a QGroup and its members.

Usage:

qa = qgroup.attrs()
qa0 = qa.vname  # for variables in this group, or qa['vname']
qa1 = qa._  # or qa[''] for attributes of this group
value = qa0.aname  # or qa0['aname'], None if no such attribute
qa0.aname = value  # or qa0['aname'] = value
qa0.aname = dtype, shape, value
if 'aname' in qa0: do_something
for aname in qa0: do_something
for aname, value in qa0.items(): do_something

class qnd.frontend.QGroup(item=None, state=None, auto=None, recording=None, goto=None)

Bases: qnd.adict.ItemsAreAttrs

Group of subgroups, lists, and ndarrays.

You reference QGroup items by name, either as qg['name'] like a dict item, or equivalently as qg.name like an object attribute. Use [] when the item name is an expression or the contents of a variable; use . when you know the name of the item. You can use [] or . to both get and set items in the QGroup. To read the entire group into a ADict, call it like a function, qg(); you may supply a list of names to read only a subset of items. A QGroup acts like a dict in many ways:

if 'name' in qg: do_something
for name in qg: do_something
item_names = list(qg)  # qg.keys() exists but is never necessary
for name, item in qg.items(): do_something
qg.update({name0: val0, ...}, [(name1, val1), ...], name2=val2, ...)
value = qg.get('name', default)

A QGroup has several possible states or modes:

1. Recording mode, turned on by qg.recording(1) and off by qg.recording(0), affects what happens when you set group items. With recording mode off, setting an item to an array creates the item as an array if its name has not been used, or otherwise writes its new value, requiring it be compatible with the dtype and shape of the previous declaration. With recording mode on, setting an item for the first time creates a QList and sets its first element to the given value, and subsequently setting that item appends the given value to the existing QList. There is also a recording mode qg.recording(2) in which subsequent values need not match the dtype or shape of the first item. You may not switch recording modes for a given item; the mode in effect when an item is first created governs the behavior of that item.
2. Goto mode, in which you set a current record with qg.goto(rec). Any item you retrieve or query which is a QList retrieves or queries the element with 0-origin index rec instead of the whole QList. You turn off goto mode with qg.goto(None). There is also a qg.gotoit() function which returns an iterator over all the records (generally the longest QList in qg).
3. Auto mode, turned on by qg.auto(1) and off by qg.auto(0), in which getting any item reads and returns its value, rather than a QLeaf object. There is also a qg.auto(2) mode in which the auto-read feature applies to any QGroup or QList (if goto mode is off) items recursively.

A QGroup has push and drop methods which can be used to save and restore all its modes. The drop method is called implicitly upon exit from a with statement, so you can use the QGroup as a context manager:

with openh5('myfile.h5', 'a') as qg:
    do_something(qg)
    with qg.push():
        qg.goto(rec)
        do_something_else(qg)
    # qg restored to goto mode state before with.
    do_even_more(qg)
# qg flushed and closed upon exit from with clause that has no
# no corresponding push

islist

isleaf

Always 0.

isgroup

Always 1.

dtype

Always dict, the builtin python type.

shape

ndim

size

sshape

Always None.

attrs()

Return attribute tree for variables in this group.

auto(recurse)

Set the auto-read mode for this QGroup.

In auto-read mode, getting an item returns its value, rather than a QLeaf. If the item is a QGroup or QList, that is returned if the recurse value is 1, whereas if recurse is 2, the QGroup or QList variables will be read recursively. Setting recurse to 0 turns off auto-read mode entirely.

Note that you can temporarily set auto mode using a with clause.

close()

Close associated file.

drop(nlevels=None, close=False)

Restore previous recording, goto, and auto mode settings.

Default drop() drops one pushed state, drop(n) drops n, drop('all') drops all pushed states. By default, drop is a no-op if no pushed states to drop, drop(close=1) closes the file if no pushed states to drop, which is called implicitly on exit from a with suite.

dtype

alias of builtins.dict

flush()

Flush associated file.

get(key, default=None)

like dict.get method

goto(record=<object object>, **kwargs)

Set the current record for this QGroup, or turn off goto mode.

Pass record of None to turn off goto mode, so that QList variables appear as the whole QList. Setting an integer record makes any QList variable appear to be the specified single element. A record value may be negative, with the usual python interpretation for a negative sequence index. If different QList variables have different lengths, the current record may be out of range for some variables but not for others. (Hence using goto mode may be confusing in such situations.)

Note that you can temporarily set goto mode using a with clause.

This goto method also accepts a keyword argument instead of a record number. The keyword name must match the name of a QList variable in this QGroup, whose vaules are scalars. This will set record to the record where that variable is nearest the keyword value. Thus, goto(time=t) selects the record nearest time t.

As a special case, you can get the current record number by calling goto with neither a record nor a keyword:

current_record = qg.goto()

gotoit(name=None)

Iterate over goto records, yielding current record.

Optional name argument is the name of a goto method keyword, which may implicitly remove records corresponding to non-monotonic changes of that variable. If name is a decreasing variable, the record order will be reversed.

As a side effect, the current record of this QGroup will be set during each pass. If the loop completes, the original goto state will be restored, but breaking out of the loop will leave the goto record set.

items(auto=None)

like dict.items method (iteritems in python2)

push()

Push current recording, goto, and auto mode onto state stack.

recording(flag)

Change recording mode for this QGroup.

With recording mode off, writing to a variable overwrites that variable. With recording mode on, new variables are declared as a QList and subsequent write operations append a new element to this QList instead of overwriting any previously stored values. In netCDF parlance, variables declared in recording mode are record variables. Writing to a variable declared when recording mode was off will always overwrite it; once declared, you cannot convert a variable to a QList simply by turning on recording mode.

See goto mode for handling record variable read operations.

A flag value of 0 turns off recording mode. A flag of 1 turns on recording mode, utilizing a trailing UNLIMITED array dimension in netCDF or HDF5 parlance, which promises that all values written will have the same dtype and shape. A flag of 2 places no restrictions on the dtype or shape of the QList elements; such an unrestricted QList resembles an anonymous QGroup.

root()

Return root QGroup for this item.

class qnd.frontend.QLeaf(item)

Bases: object

An ndarray or None stored in a file.

You can read the data by calling the leaf instance ql(), or by indexing it ql[:], which also provides a means for partial reads. A QLeaf has dtype, shape, ndim, and size properties with the same meanings as an ndarray (except None has all these properties equal None). Additionally, the sshape property may return a symbolic shape with optional strings in the tuple representing dimension names.

You can write data by calling ql(value), or by setting a slice, which provides a means for partial writes.

isgroup

islist

Always 0.

isleaf

Always 1.

dtype

The numpy dtype of this ndarray, or None if this leaf is None. This is the dtype in memory, not necessarily as stored.

shape

ndim

size

The numpy ndarray properties, or None if this leaf is None.

sshape

A symbolic shape tuple, like shape except dimension lengths may be type str instead of int.

root()

Return root QGroup for this item.

class qnd.frontend.QList(item=None, auto=0)

Bases: object

List of subgroups, lists, and ndarrays.

You reference QList elements by index or slice, like ordinary list elements, including the python convention for negative index values. To read the entire list, call it like a function, ql(), which is equivalent to ql[:]. A QList has __iter__, append, and extend:

for element in ql: do_something
ql.append(value)
ql.extend(iterable)

In general, the elements of a QList are unrelated to one another; it’s like an anonymous QGroup. However, a common use case is to represent a so-called UNLIMITED dimension in netCDF or HDF5. In this case, every element will have the same dtype and shape. The islist method returns 1 for this special restricted case, while it returns 2 for an unrestricted QList. Whether this makes any difference depends on the underlying file format. The QGroup recording and goto methods allow you to access QList items in the group transparently, as if they were individual elements at a current record or index.

isgroup

isleaf

Always 0.

islist

This is 1 if this QList is a record array declared in recording mode 1, and 2 if it was declared in any other way (including as a record array in recording mode 2).

dtype

Always list, the builtin python type.

shape

ndim

size

sshape

Always None.

append(value)

append a new element to this QList

auto(recurse)

Set auto read mode, analogous to QGroup.auto method.

dtype

alias of builtins.list

extend(iterable)

append multiple new elements to this QList

root()

Return root QGroup for this item.

class qnd.frontend.QState(recording=0, goto=None, auto=0)

Bases: list

State information for a QGroup.

class qnd.frontend.QnDList(parent, empty=None)

Bases: object

Implmentation of a low level QList type using QGroup.

A backend which has no direct support for QList objects can use this to produce a pseudo-list, which is a group with member names _ (None or a single signed or unsigned byte, value never read) and names _0, _1, _2, etc.

This implementation will handle both UNLIMITED index-style lists made with recording = 1 (that is group.declare with unlim flag) and general lists. If UNLIMITED dimensions are supported, pass the QnDLeaf to this constructor:

item = QnDList(QnDLeaf)  # if at least one record exists
item = QnDList(QnDLeaf, 1)  # if no records yet exist

Use the fromgroup constructor to check if a QnDGroup is a pseudo-list:

item = QnDList.fromgroup(QnDGroup)