options

options helps encapsulate options and configuration data using a layered stacking model (a.k.a. nested contexts).

For most functions and many classes, options is overkill. Python’s already-flexible function arguments, *args, **kwargs, and inheritance patterns are elegant and sufficient for 99.9% of all development situations. options is for the other 0.1%:

  • highly functional classes,
  • with many different features and options,
  • which might be adjusted or overriden at any time,
  • yet that need “reasonable” or “intelligent” defaults, and
  • that yearn for a simple, unobtrusive API.

In those cases, Python’s simpler built-in, inheritance-based model adds complexity as non-trivial options and argument-management code spreads through many individual methods. This is where options‘s delegation-based approach begins to shine.

http://d.pr/i/6JI4+

For more backstory, see this StackOverflow.com discussion of how to combat “configuration sprawl”. For examples of options in use, see say and show. (If you’re using options in your own package, drop me a line!)

Usage

In a typical use of options, your highly-functional class defines default option values. Subclasses can add, remove, or override options. Instances use class defaults, but they can be overridden when each instance is created. For any option an instance doesn’t override, the class default “shines through.”

So far, this isn’t very different from a typical use of Python’s standard instance and class variables. The next step is where options gets interesting.

Individual method calls can similarly override instance and class defaults. The options stated in each method call obtain only for the duration of the method’s execution. If the call doesn’t set a value, the instance value applies. If the instance didn’t set a value, its class default applies (and so on, to its superclasses, if any).

One step further, Python’s with statement can be used to set option values for just a specific duration. As soon as the with block exists, the option values automagically fall back to what they were before the with block. (In general, if any option is unset, its value falls back to what it was in the next higher layer.)

To recap: Python handles class, subclass, and instance settings. options handles these as well, but also adds method and transient settings. By default when Python overrides a setting, it’s destructive; the value cannot be “unset” without additional code. When a program using options overrides a setting, it does so non-destructively, layering the new settings atop the previous ones. When attributes are unset, they immediately fall back to their prior value (at whatever higher level it was last set).

An Example

Unfortunately, because this is a capability designed for high-end, edge-case situations, it’s hard to demonstrate its virtues with simple code. But we’ll give it a shot.

from options import Options, attrs

class Shape(object):

    options = Options(
        name   = None,
        color  = 'white',
        height = 10,
        width  = 10,
    )

    def __init__(self, **kwargs):
        self.options = Shape.options.push(kwargs)

    def draw(self, **kwargs):
        opts = self.options.push(kwargs)
        print attrs(opts)

one = Shape(name='one')
one.draw()
one.draw(color='red')
one.draw(color='green', width=22)

yielding:

color='white', width=10, name='one', height=10
color='red', width=10, name='one', height=10
color='green', width=22, name='one', height=10

So far we could do this with instance variables and standard arguments. It might look a bit like this:

class ClassicShape(object):

    def __init__(self, name=None, color='white', height=10, width=10):
        self.name   = name
        self.color  = color
        self.height = height
        self.width  = width

but when we got to the draw method, things would be quite a bit messier.:

def draw(self, **kwargs):
    name   = kwargs.get('name',   self.name)
    color  = kwargs.get('color',  self.color)
    height = kwargs.get('height', self.height)
    width  = kwargs.get('width',  self.width)
    print "color={0!r}, width={1}, name={2!r}, height={3}".format(color, width, name, height)

One problem here is that we broke apart the values provided to __init__() into separate instance variables, now we need to re-assemble them into something unified. And we need to explicitly choose between the **kwargs and the instance variables. It gets repetitive, and is not pretty. Another classic alternative, using native keyword arguments, is no better:

def draw2(self, name=None, color=None, height=None, width=None):
    name   = name   or self.name
    color  = color  or self.color
    height = height or self.height
    width  = width  or self.width
    print "color={0!r}, width={1}, name={2!r}, height={3}".format(color, width, name, height)

If we add just a few more instance variables, we have the Mr. Creosote of class design on our hands. For every instance variable that might be overridden in a method call, that method needs one line of code to decide whether the override is, in fact, in effect. Suddenly dealing with parameters starts to be a full-time job, as every possible setting has to be managed in every method. That’s neither elegant nor scalable. Pretty soon we’re in “just one more wafer-thin mint...” territory.

But with options, it’s easy. No matter how many configuration variables there are to be managed, each method needs just one line of code to manage them:

opts = self.options.push(kwargs)

Changing things works simply and logically:

Shape.options.set(color='blue')
one.draw()
one.options.set(color='red')
one.draw(height=100)
one.draw(height=44, color='yellow')

yields:

color='blue', width=10, name='one', height=10
color='red', width=10, name='one', height=100
color='yellow', width=10, name='one', height=44

In one line, we reset the default color for all Shape objects. That’s visible in the next call to one.draw(). Then we set the instance color of one (all other Shape instances remain blue). Finally, We call one with a temprary override of the color.

A common pattern makes this even easier:

class Shape(OptionsClass):
    ...

The OptionsClass base class will provide a set() method so that you don’t need Shape.options.set(). Shape.set() does the same thing, resulting in an even simpler API. The set() method is a “combomethod” that will take either a class or an instance and “do the right thing.” OptionsClass also provides a settings() method to easily handle transient with contexts (more on this in a minute), and a __repr__() method so that it prints nicely.

The more options and settings a class has, the more unwieldy the class and instance variable approach becomes, and the more desirable the delegation alternative. Inheritance is a great software pattern for many kinds of data and program structures–but it’s not a particularly good pattern for complex option and configuration handling.

For richly-featured classes, options‘s delegation pattern is simpler. As the number of options grows, almost no additional code is required. More options impose no additional complexity and introduce no additional failure modes. Consolidating options into one place, and providing neat attribute-style access, keeps everything tidy. We can add new options or methods with confidence:

def is_tall(self, **kwargs):
    opts = self.options.push(kwargs)
    return opts.height > 100

Under the covers, options uses a variation on the ChainMap data structure (a multi-layer dictionary) to provide option stacking. Every option set is stacked on top of previously set option sets, with lower-level values shining through if they’re not set at higher levels. This stacking or overlay model resembles how local and global variables are managed in many programming languages.

This makes advanced use cases, such as temporary value changes, easy:

with one.settings(height=200, color='purple'):
    one.draw()
    if is_tall(one):
        ...         # it is, but only within the ``with`` context

if is_tall(one):    # nope, not here!
    ...

Note

You will still need to do some housekeeping in your class’s __init__() method, including creating a new options layer. If you don’t wish to inherit from OptionsClass, you can manually add set() and settings() methods to your own class. See the OptionsClass source code for details.

As one final feature, consider “magical” parameters. Add the following code to your class description:

options.magic(
    height = lambda v, cur: cur.height + int(v) if isinstance(v, str) else v,
    width  = lambda v, cur: cur.width  + int(v) if isinstance(v, str) else v
)

Now, in addition to absolute height and width parameters specified with int (integer/numeric) values, your module auto-magically supports relative parameters for height and width given as string parametrs.:

one.draw(width='+200')

yields:

color='blue', width=210, name='one', height=10

Neat, huh?

For more, see this StackOverflow.com discussion of how to combat “configuration sprawl”. For examples of options in use, see say and show.

Design Considerations

options is not intened to replace every class’s or method’s parameter passing mechanisms–just the one or few most highly-optioned ones that multiplex a package’s functionality to a range of use cases. These are generally the highest-level, most outward-facing classes/objects. They will generally have at least five configuration variables (e.g. kwargs used to create, configure, and define each instance).

In general, classes will define a set of methods that are “outwards facing”–methods called by external code when consuming the class’s functionality. Those methods should generally expose their options through **kwargs, creating a local variable (say opts) that represents the sum of all options in use–the full stack of call, instance, and class options, including any present magical interpretations.

Internal class methods–the sort that are not generally called by external code, and that by Python convention are often prefixed by an underscore (_)–these generally do not need **kwargs. They should accept their options as a single variable (say opts again) that the externally-facing methods will provide.

For example, if options didn’t provide the nice formatting function attrs, we might have designed our own:

def _attrs(self, opts):
    nicekeys = [ k for k in opts.keys() if not k.startswith('_') ]
    return ', '.join([ "{}={}".format(k, repr(opts[k])) for k in nicekeys ])

def draw(self, **kwargs):
    opts = self.options.push(kwargs)
    print self._attrs(opts)

draw(), being the outward-facing API, accepts general arguments and manages their stacking (by push``ing ``kwargs onto the instance options). When the internal _attrs() method is called, it is handed a pre-digested opts package of options.

A nice side-effect of making this distinction: Whenever you see a method with **kwargs, you know it’s outward-facing. When you see a method with just opts, you know it’s internal.

Objects defined with options make excellent “callables.” Define the __call__ method, and you have a very nice analog of function calls.

options has broad utility, but it’s not for every class or module. It best suits high-level front-end APIs that multiplex lots of potential functionality, and wish/need to do it in a clean/simple way. Classes for which the set of instance variables is small, or functions/methods for which the set of known/possible parameters is limited–these work just fine with classic Python calling conventions. For those, options is overkill. “Horses for courses.”

Setting and Unsetting

Using options, objects often become “entry points” that represent both a set of capabilities and a set of configurations for how that functionality will be used. As a result, you may want to be able to set the object’s values directly, without referencing their underlying options. It’s convenient to add a set() method, then use it, as follows:

def set(self, **kwargs):
    self.options.set(**kwargs)

one.set(width='*10', color='orange')
one.draw()

yields:

color='orange', width=100, name='one', height=10

one.set() is now the equivalent of one.options.set(). Or continue using the options attribute explicitly, if you prefer that.

Values can also be unset.:

from options import Unset

one.set(color=Unset)
one.draw()

yields:

color='blue', width=100, name='one', height=10

Because 'blue' was the color to which Shape had be most recently set. With the color of the instance unset, the color of the class shines through.

Note

While options are ideally accessed with an attribute notion, the preferred of setting options is through method calls: set() if accessing directly, or push() if stacking values as part of a method call. These perform the interpretation and unsetting magic; straight assignment does not. In the future, options may provide an equivalent __setattr__() method to allow assignment–but not yet.

Leftovers

options expects you to define all feasible and legitimate options at the class level, and to give them reasonable defaults.

None of the initial settings ever have magic applied. Much of the expected interpretation “magic” will be relative settings, and relative settings require a baseline value. The top level is expected and demanded to provide a reasonable baseline.

Any options set “further down” such as when an instance is created or a method called should set keys that were already-defined at the class level.

However, there are cases where “extra” **kwargs values may be provided and make sense. Your object might be a very high level entry point, for example, representing very large buckets of functionality, with many options. Some of those options are relevant to the current instance, while others are intended as pass-throughs for lower-level modules/objects. This may seem a doubly rarefied case–and it is, relatively speaking. But it does happen–and when you need multi-level processing, it’s really, really super amazingly handy to have it.

options supports this in its core push() method by taking the values that are known to be part of the class’s options, and deleting those from kwargs. Any values left over in the kwargs dict are either errors, or intended for other recipients.

As yet, there is no automatic check for leftovers.

Magic Parameters

Python’s *args variable-number of arguments and **kwargs keyword arguments are sometimes called “magic” arguments. options takes this up a notch, allowing setters much like Python’s property function or @property decorator. This allows arguments to be interpreted on the fly. This is useful, for instance, to provide relative rather than just absolute values. As an example, say that we added this code after Shape.options was defined:

options.magic(
    height = lambda v, cur: cur.height + int(v) if isinstance(v, str) else v,
    width  = lambda v, cur: cur.width  + int(v) if isinstance(v, str) else v
)

Now, in addition to absolute height and width parameters which are provided by specifying int (integer/numeric) values, your module auto-magically supports relative parameters for height and width.:

one.draw(width='+200')

yields:

color='blue', width=210, name='one', height=10

This can be as fancy as you like, defining an entire domain-specific expression language. But even small functions can give you a great bump in expressive power. For example, add this and you get full relative arithmetic capability (+, -, *, and /):

def relmath(value, currently):
    if isinstance(value, str):
        if value.startswith('*'):
            return currently * int(value[1:])
        elif value.startswith('/'):
            return currently / int(value[1:])
        else:
            return currently + int(value)
    else:
        return value

...

options.magic(
    height = lambda v, cur: relmath(v, cur.height),
    width  = lambda v, cur: relmath(v, cur.width)
)

Then:

one.draw(width='*4', height='/2')

yields:

color='blue', width=40, name='one', height=5

Magically interpreted parameters are the sort of thing that one doesn’t need very often or for every parameter–but when they’re useful, they’re enormously useful and highly leveraged, leading to much simpler, much higher function APIs.

We call them ‘magical’ here because of the “auto-magical” interpretation, but they are really just analogs of Python object properties. The magic function is basically a “setter” for a dictionary element.

The Magic APIs

The callables (usually functions, lambda expressions, static methods, or methods) called to preform magical interpretation can be called with 1, 2, or 3 parameters. options inquires as to how many parameters the callable accepts. If it accepts only 1, it will be the value passed in. Cleanups like “convert to upper case” can be done, but no relative interpretation. If it accepts 2 arguments, it will be called with the value and the current option mapping, in that order. (NB this order reverses the way you may think logical. Caution advised.) If the callable requires 3 parameters, it will be None, value, current mapping. This supports method calls, though has the defect of not really passing in the current instance.

A decorator form, magical() is also supported. It must be given the name of the key exactly:

@options.magical('name')
def capitalize_name(self, v, cur):
    return ' '.join(w.capitalize() for w in v.split())

The net is that you can provide just about any kind of callable. But the meta-programming of the magic interpretation API could use a little work.

Subclassing

Subclass options may differ from superclass options. Usually they will share many options, but some may be added, and others removed. To modify the set of available options, the subclass defines its options with the add() method to the superclass options. This creates a layered effect, just like push() for an instance. The difference is, push() does not allow new options (keys) to be defined; add() does. It is also possible to assign the special null object Prohibited, which will disallow instances of the subclass from setting those values.:

options = Superclass.options.add(
    func   = None,
    prefix = Prohibited,  # was available in superclass, but not here
    suffix = Prohibited,  # ditto
)

Because some of the “additions” can be prohibitions (i.e. removing particular options from being set or used), this is “adding to” the superclass’s options in the sense of “adding a layer onto” rather than strict “adding options.”

An alternative is to copy (or restate) the superclass’s options. That suits “unlinked” cases–where the subclass is highly independent, and where changes to the superclass’s options should not effect the subclass’s options. With add(), they remain linked in the same way as instances and classes are.

Transients and Internal Options

Some options do not make sense as permanent values–they are needed only as transient settings in the context of individual method calls. The special null value Transient can be assigned as an option value to signal this.

Other options are useful, but only internal to your class. They are not meant to be exposed as part of the external API. In this case, they can be signified by prefixing with an underscore, such as _cached_value. This is consistent with Python naming practice.

Flat Arguments

Sometimes it’s more elegant to provide some arguments as flat, sequential values rather than by keyword. In this case, use the addflat() method:

def __init__(self, *args, **kwargs):
    self.options = Quoter.options.push(kwargs)
    self.options.addflat(args, ['prefix', 'suffix'])

to consume optional prefix and suffix flat arguments. This makes the following equivalent:

q1 = Quoter('[', ']')
q2 = Quoter(prefix='[', suffix=']')

An explicit addflat() method is provided not as much for Zen of Python reasons (“Explicit is better than implicit.”), but because flat arguments are commonly combined with abbreviation/shorthand conventions, which may require some logic to implement. For example, if only a prefix is given as a flat argument, you may want to use the same value to implicitly set the suffix. To this end, addflat returns the set of keys that it consumed:

if args:
    used = self.options.addflat(args, ['prefix', 'suffix'])
    if 'suffix' not in used:
        self.options.suffix = self.options.prefix

Choosing Option Names

You can choose pretty much any option name that is a legitimate Python keyword argument. The exceptions: Names that are already defined by methods of Options or OptionsChain. To wit: add, addflat, clear, copy, fromkeys, get, items, iteritems, iterkeys, itervalues, keys, magic, magical, new_child, parents, pop, popitem, push, read, set, setdefault, update, values, and write are off-limits.

If you try to define options with verboten names, a BadOptionName exception will be raised. This will save you grief down the road; getting back a wrong thing at runtime is vastly harder to debug than fielding an early exception.

Notes

  • This is a work in progress. The underlying techniques have been successfully used in multiple projects, but it remains in active development. The API may change over time. Swim at your own risk.
  • options undergoes frequent automated multi-version testing with pytest and tox. It is successfully packaged for, and tested against, Python 2.6, 2.7, 3.2, and 3.3. It additionally runs under, and is tested against, PyPy 2.1 (based on 2.7.3) though it has to work around a few bugs in the underlying stuf module to do so.
  • The author, Jonathan Eunice or @jeunice on Twitter welcomes your comments and suggestions.

Installation

pip install options

To easy_install under a specific Python version (3.3 in this example):

python3.3 -m easy_install options

(You may need to prefix these with “sudo ” to authorize installation.)