1. Phil - Python-based hierarchical interchange language 

Links

The primary home of this document is:: https://freephil.readthedocs.io
The source code examples below are available as one file here:: https://github.com/cctbx/cctbx_project/blob/master/iotbx/examples/libtbx_phil_examples.py
freephil is available on pypi and conda-forge: https://pypi.org/project/freephil

1.1. Phil overview 

Phil (Python-based hierarchical interchange language) is a module for the management of application parameters and, to some degree, inputs. Many applications use command-line options as a user interface (e.g. based on Python’s optparse, a.k.a Optik). This approach works well for small applications, but has it limitations for more complex applications with a large set of parameters.

A simple Phil file as presented to the user may look like this:

minimization.input {
  file_name = experiment.dat
  label = set2
}
minimization.output {
  model_file = final.mdl
  plot_file = None
}
minimization.parameters {
  method = *bfgs conjugate_gradient
  max_iterations = 10
}

Phil is designed with a minimal syntax. An important goal is to enable users to get started just by looking at defaults and examples, without having to read documentation.

The Phil syntax has only two main elements, freephil.definition (e.g. max_iterations = 10 and freephil.scope (e.g. minimization.input { }). To make this syntax as user-friendly as possible, strings do not have to be quoted and, unlike Python, indentation is not syntactically significant. E.g. this:

minimization.input {
file_name="experiment.dat"
labels="set2"
}

is equivalent to the corresponding definitions above.

Scopes can be nested recursively. The number of nesting levels is limited only by Python’s recursion limit (default 1000). To maximize convenience, nested scopes can be defined in two equivalent ways. For example:

minimization {
  input {
  }
}

is equivalent to:

minimization.input {
}

1.2. Beyond syntax 

Phil is much more than just a parser for a very simple, user-friendly syntax. Major Phil features are:

The concepts of base files and user files. The syntax for the two types of Phil files is identical, but the processed Phil files are used in different ways. I.e. the concepts exist only at the semantical level. The “look and feel” of the files is uniform.

Interpretation of command-line arguments as Phil definitions.

Merging of (multiple) Phil files and (multiple) Phil definitions derived from command-line arguments.

Automatic conversion of Phil files to Python objects which are essentially independent of the Phil system. I.e. core algorithms using Phil-derived parameter objects do not actually have to depend on Phil.

The reverse conversion of (potentially modified) Python objects back to Phil files. This could also be viewed as a Phil pretty printer.

Shell-like variable substitution using $var and ${var} syntax.

include syntax to merge Phil files at the parser level, or to import Phil objects from other Python scripts.

1.3. Base files 

Base files are written by the software developer and include “attributes” for each parameter, such as the type (integer, floating-point, string, etc.) and support information for graphical interfaces. For example:

minimization.parameters
  .help = "Selection and tuning of minimization algorithm."
  .expert_level = 0
{
  method = *bfgs conjugate_gradient
    .type = choice
  max_iterations = 10
    .type = int
    .input_size = 8
}

The is the last part of the output of this command:

freephil --show-some-attributes example.params

Run this command with --show-all-attributes to see the full set of definition and scope attributes. This output tends to get very long, but end-users don’t have to be aware of this, and even programmers only have to deal with the attributes they want to change.

1.4. User files 

User files are typically generated by the application. For example:

minimization.quick --show_defaults

will process its base file and show only the most relevant parameters, classified by the software developer as .expert_level = 0 (default). E.g. the minimization.parameters scope in the example above is not shown. The attributes are also not shown. Therefore the output is much shorter compared to the freephil --show-some-attributes output above:

minimization.parameters {
  method = *bfgs conjugate_gradient
  max_iterations = 10
}

1.5. Command-line arguments 

In theory, the user could save and edit the generated parameter files. However, in many practical situations this manual step can be avoided. Phil is designed with the idea that the application inspects all input files and uses the information found to fill in the blanks automatically. This is not only convenient, but also eliminates the possiblity of typing errors. In addition, the user can specify parameters directly on the command line, and this information is also use to fill in the blanks.

Command-line arguments that are not file names or options prefixed with -- (like --show_defaults above) should be given to Phil for examination. E.g., this is a possible command:

minimization.quick experiment.dat output.plot_file=plot.pdf

First the application should check if an argument is the name of a file that can be opened. Assume this succeeds for the first argument, so the processing of this argument is finished. Assume further that a file with the name output.plot_file=plot.pdf does not exist. This argument will therefore be interpreted with Phil. The next sections presents an examples.

Note

Checking for application-compatible files is not provided by FreePHIL. However, parsing arguments mimicking Phil syntax is provided.

Note

In the syntax of command line arguments, = signs cannot be separated with spaces from neither parameter name nor value.

1.6. Phil object (scope)

Phil files are converted to uniform Phil objects, type freephil.scope or freephil.definition. It holds all the information of the original Phil file (or string), and brings all the facilities needed to further work with the Phil objects.

1.6.1. parse: Creating of Phil objects 

The Phil parser converts base files, user files (or equivalent strings) to uniform Phil objects. The input can be either a string (input_string=) or a file (file_name=). Useful feature is source_info= which can hold information on the source of the Phil object. Example:

from freephil import parse

base_phil = parse("""
  minimization.input {
    file_name = None
      .type = path
    label = None
      .type = str
  }
  """,
  source="base PHIL")

freephil.parse(input_string=None, source_info=None, file_name=None, converter_registry=None, process_includes=False, include_stack=None)[source]

Creates Phil object from a string or a file

Parameters:

input_string – String to be parsed
source_info – Description of the source. Defaults to file_name
file_name – Parse from a file
converter_registry – Custom converters (see Extending Phil)
process_includes – Enables processing include statement
include_stack –

Returns:

Phil object

Return type:

freephil.scope

1.6.2. Parsing command line arguments 

Having to type in fully qualified parameter names (e.g. minimization.input.labels) can be very inconvenient. Therefore Phil includes support for matching parameter names of command-line arguments as substrings to the parameter names in the base files:

## extract code begin: freephil_examples.py

argument_interpreter = base_phil.command_line_argument_interpreter(
  home_scope="minimization")

command_line_phil = argument_interpreter.process(
  arg="minimization.input.label=set2")

## extract code end

This works even if the user writes just label=set2 or even put.lab=x1 x2. The only requirement is that the substring leads to a unique match in the base file. Otherwise Phil produces a helpful error message. For example:

argument_interpreter.process("a=set2")

leads to:

Sorry: Ambiguous parameter definition: a = set2
Best matches:
  minimization.input.file_name
  minimization.input.label

The user can cut-and-paste the desired parameter into the command line for another trial to run the application.

freephil.scope.command_line_argument_interpreter(self, home_scope=None, argument_description=None)

Creates an interpreter of command line arguments for the scope

Parameters:

home_scope (freephil.scope) – Parse only within sub-scope
argument_description (str) – Description of arguments source. Defaults “command line”

Returns:

Command line interpreter

Return type:

freephil.command_line.argument_interpreter

freephil.command_line.argument_interpreter.process(self, arg=None, args=None, custom_processor=None)

Process string as command line argument.

Parameters:

arg (str) – One argument
args (list of str) – Arguments to be processed
custom_processor – Use custom Phil processor.

Returns:

Phil object

Return type:

freephil.scope

1.6.3. fetch: merging of Phil objects 

Phil objects can be merged to generate a combined set of “working” parameters used in running the application. We demonstrate this by way of a simple, self-contained Python script with embedded Phil syntax:

## extract code begin: freephil_examples.py

from freephil import parse

base_phil = parse("""
  minimization.input {
    file_name = None
      .type = path
    label = None
      .type = str
  }
  """)

user_phil = parse("""
  minimization.input {
    file_name = experiment.dat
  }
  """)

command_line_phil = parse(
  "minimization.input.label=set2")

working_phil = base_phil.fetch(
  sources=[user_phil, command_line_phil])
working_phil.show()

## extract code end

base_phil defines all available parameters including the type information. user_phil overrides the default file_name assignment but leaves the labels undefined. These are defined by a (fake) command-line argument. All inputs are merged via base_phil.fetch(). working_phil.show() produces:

minimization.input {
  file_name = experiment.dat
  label = set2
}

freephil.scope.fetch(self, source=None, sources=None, track_unused_definitions=False, diff=False, skip_incompatible_objects=False)

Combine multiple Phil objects using the base Phil (self). Returns full Phil object with changes from sources applied. If an arguments occurs multiple times in different sources, the first from the list is used. For more details see fetch: merging of Phil objects.

Parameters:

source (freephil.scope) – Input Phil object
sources (list of freephil.scope) – Multiple input Phil objects
track_unused_definitions (bool) – If True, the function returns a tuple, where second member contains entries not used in base Phil object (see: fetch option: track_unused_definitions)
diff (bool) – If True, equivalent to fetch_diff()
skip_incompatible_objects (bool) – Skip incompatible object types

Returns:

Phil object, or Phil object and object with unprocessed data

Return type:

freephil.scope or tuple(freephil.scope, list of freephil.object_locator)

1.6.4. fetch_diff: difference between base_phil and working_phil 

The .fetch() method introduced above produces a complete copy of the Phil base with all user definitions and scopes merged in. If the Phil base is large, the output of working_phil.show() will therefore also be large. It may be difficult to see which definitions still have default values, and which definitions are changed. To get just the difference between the base and the working Phil objects, the .fetch_diff() method is available. For example:

## extract code begin: freephil_examples.py

base_phil = parse("""
  minimization.parameters {
    method = *bfgs conjugate_gradient
      .type = choice
    max_iterations = 10
      .type = int
  }
  """)

user_phil = parse("""
  minimization.parameters {
    method = bfgs *conjugate_gradient
  }
  """)

working_phil = base_phil.fetch(source=user_phil)
diff_phil = base_phil.fetch_diff(source=working_phil)
diff_phil.show()

## extract code end

Output:

minimization.parameters {
  method = bfgs *conjugate_gradient
}

Here the minimization method was changed from bfgs to conjugate_gradient but the number of iterations is unchanged. Therefore the latter does not appear in the output. .fetch_diff() is completely general and works for any combination of definitions and scopes with .multiple = False or .multiple = True.

freephil.scope.fetch_diff(self, source=None, sources=None, track_unused_definitions=False, skip_incompatible_objects=False)

Creates difference Phil object containing only items, which differ between the base Phil object and source(s).

Parameters:

source (freephil.scope) – Input Phil object
sources (list of freephil.scope) – Multiple input Phil objects
track_unused_definitions (bool) – If True, the function returns a tuple, where second member contains entries not used in base Phil object (see: fetch option: track_unused_definitions)
diff (bool) – If True, equivalent to fetch_diff()
skip_incompatible_objects (bool) – Skip incompatible object types

Returns:

Phil object, or Phil object and object with unprocessed data

Return type:

freephil.scope or tuple(freephil.scope, list of freephil.object_locator)

1.7. Python object (scope_extract)

The Phil parser produces objects that preserve most information generated in the parsing process, such as line numbers and parameter attributes. While this information is very useful for pretty printing (e.g. to archive the working parameters) and the automatic generation of graphical user interfaces, it is only a burden in the context of core algorithms. Therefore Phil supports “extraction” of light-weight Python objects from the Phil objects.

The python object is of the type freephil.scope_extract. The object contains items with the values, or nested freephil.scope_extract.

1.7.1. extract: conversion of Phil objects to Python objects 

Based on the example above, the extraction can be achieved with just one line:

## extract code begin: freephil_examples.py

working_params = working_phil.extract()

## extract code end

We can now use the extracted objects in the context of Python:

## extract code begin: freephil_examples.py

print working_params.minimization.input.file_name
print working_params.minimization.input.label

## extract code end

Output:

experiment.dat
set2

file_name and label are now a simple Python strings.

freephil.scope.extract(self, parent=None)

Extracts the Phil object into Python object.

Parameters:: parent (freephil.scope) – Set parent Phil object
Returns:: Python object
Return type:: freephil.scope_extract

1.7.2. format: conversion of Python objects to Phil objects 

Phil also supports the reverse conversion compared to the previous section, from Python objects to Phil objects. For example, to change the label:

## extract code begin: freephil_examples.py

working_params.minimization.input.label = "set3"
modified_phil = base_phil.format(python_object=working_params)
modified_phil.show()

## extract code end

Output:

minimization.input {
  file_name = "experiment.dat"
  label = "set3"
}

We need to bring in base_phil again because all the meta information was lost in the working_phil.extract() step that produced working_params. A type-specific converter is used to produce a string for each Python object (see the Extending Phil section below).

freephil.scope.format(self, python_object)

Converts Python object into Phil object. It has to be called as a member function of the base Phil object to recover Phil metadata.

Parameters:: python_object (freephil.scope_extract) – Python object to be converted
Returns:: Phil object
Return type:: freephil.scope

1.7.3. Special features 

The Python object (freephil.scope_extract) includes a few additional, special features that are worth knowing:

.__phil_path__()

Safety guard

1.7.3.1. .__phil_path__()

The first special feature is the freephil.scope_extract.__phil_path__() method:

## extract code begin: freephil_examples.py

print working_params.minimization.input.__phil_path__()
print working_params.minimization.parameters.__phil_path__()

## extract code end

Output:

minimization.input
minimization.parameters

This feature is most useful for formatting informative error messages without having to hard-wire the fully-qualified parameter names. Use .__phil_path__() to ensure that the names are automatically correct even if the base file is changed in major ways. Note that the .__phil_path__() method is available only for extracted scopes, not for extracted definitions since it would be very cumbersome to implement. However, the fully-qualified name of a definition can be obtained via .__phil_path__(object_name="max_iterations"); usually the object_name is readily available in the contexts in which the fully-qualified name is needed.

There is also freephil.scope_extract.__phil_path_and_value__() which returns a 2-tuple of the fully-qualified path and the extracted value, ready to be used for formatting error messages.

1.7.3.2. Safety guard 

The next important feature is a safety guard: assignment to a non-existing attribute leads to an exception. For example, if the attribute is mis-spelled:

working_params.minimization.input.filename = "other.dat"

Result:

AttributeError: Assignment to non-existing attribute "minimization.input.filename"
  Please correct the attribute name, or to create
  a new attribute use: obj.__inject__(name, value)

In addition to trivial spelling errors, the safety guard traps overlooked dependencies related to changes in the base file.

In some (unusual) situations it may be useful to attach attributes to an extracted scope that have no correspondence in the base file. Use the freephil.scope_extract.__inject__() method for this purpose to by-pass the safety-guard. As a side-effect of this design, injected attributes are easily pin-pointed in the source code (simply search for __inject__), which can be a big help in maintaining a large code base.

1.8. More on Phil syntax 

1.8.1. Available attributes 

Possible attributes in base phil files are listed in description of phil.definition and phil.scope.

1.8.2. .multiple = True 

Both phil.definition and phil.scope support the .multiple = True attribute. For the sake of simplicity, in the following “multiple definition” and “multiple scope” means a base definition or scope with .multiple = True. Please note the distinction between this and multiple values given in a user file. For example, this is a multiple definition in a base file:

## extract code begin: freephil_examples.py

base_phil = parse("""
  minimization.input {
    file_name = None
      .type = path
      .multiple = True
  }
  """)

## extract code end

And these are multiple values for this definition in a user file:

## extract code begin: freephil_examples.py

user_phil = parse("""
  minimization.input {
    file_name = experiment1.dat
    file_name = experiment2.dat
    file_name = experiment3.dat
  }
  """)

## extract code end

I.e. multiple values are simply specified by repeated definitions. Without the .multiple = True in the base file, .fetch() retains only the last definition found in the base and all user files or command-line arguments. .multiple = True directs Phil to keep all values. .extract() then returns a list of all these values converted to Python objects. For example, given the user file above:

## extract code begin: freephil_examples.py

working_params = base_phil.fetch(source=user_phil).extract()
print working_params.minimization.input.file_name

## extract code end

will show this Python list:

['experiment1.dat', 'experiment2.dat', 'experiment3.dat']

Multiple scopes work similarly, for example:

## extract code begin: freephil_examples.py

base_phil = parse("""
  minimization {
    input
      .multiple = True
    {
      file_name = None
        .type = path
      label = None
        .type = str
    }
  }
  """)

## extract code end

A corresponding user file may look like this:

## extract code begin: freephil_examples.py

user_phil = parse("""
  minimization {
    input {
      file_name = experiment1.dat
      label = set2
    }
    input {
      file_name = experiment2.dat
      label = set1
    }
  }
  """)

## extract code end

The result of the usual fetch-extract sequence is:

## extract code begin: freephil_examples.py

working_params = base_phil.fetch(source=user_phil).extract()
for input in working_params.minimization.input:
  print input.file_name
  print input.label

## extract code end

Output:

experiment1.dat
set2
experiment2.dat
set1

Definitions and scopes may be nested with any combination of .multiple = False or .multiple = True. For example, this would be a plausible base file:

## extract code begin: freephil_examples.py

base_phil = parse("""
  minimization {
    input
      .multiple = True
    {
      file_name = None
        .type = path
      label = None
        .type = str
        .multiple = True
    }
  }
  """)

## extract code end

This is a possible corresponding user file:

## extract code begin: freephil_examples.py

user_phil = parse("""
  minimization {
    input {
      file_name = experiment1.dat
      label = set1
      label = set2
      label = set3
    }
    input {
      file_name = experiment2.dat
      label = set2
      label = set3
    }
  }
  """)

## extract code end

The fetch-extract sequence is the same as before:

## extract code begin: freephil_examples.py

working_params = base_phil.fetch(source=user_phil).extract()
for input in working_params.minimization.input:
  print input.file_name
  print input.label

## extract code end

but the output shows lists of strings for label instead of just one Python string:

experiment1.dat
['set1', 'set2', 'set3']
experiment2.dat
['set2', 'set3']

1.8.3. Multiple definitions and scopes 

All Phil attributes of multiple definitions or scopes are determined by the first occurrence in the base file. All following instances in the base file are defaults. Any instances in user files (merged via .fetch()) are added to the default instances in the base file. For example:

## extract code begin: freephil_examples.py

base_phil = parse("""
  plot
    .multiple = True
  {
    style = line bar pie_chart
      .type=choice
    title = None
      .type = str
  }
  plot {
    style = line
    title = Line plot (default in base)
  }
  """)

user_phil = parse("""
  plot {
    style = bar
    title = Bar plot (provided by user)
  }
  """)

working_phil = base_phil.fetch(source=user_phil)
working_phil.show()

## extract code end

Output:

plot {
  style = *line bar pie_chart
  title = Line plot (default in base)
}
plot {
  style = line *bar pie_chart
  title = Bar plot (provided by user)
}

.extract() will produce a list with two elements:

## extract code begin: freephil_examples.py

working_params = working_phil.extract()
print working_params.plot

## extract code end

Output:

[<freephil.scope_extract object at 0x2b1ccb5b1910>,
 <freephil.scope_extract object at 0x2b1ccb5b1c10>]

Note that the first (i.e. base) occurrence of the scope is not extracted. In practice this is usually the desired behavior, but it can be changed by setting the plot scope attribute .optional = False. For example:

## extract code begin: freephil_examples.py

base_phil = parse("""
  plot
    .multiple = True
    .optional = False
  {
    style = line bar pie_chart
      .type=choice
    title = None
      .type = str
  }
  plot {
    style = line
    title = Line plot (default in base)
  }
  """)

## extract code end

With the user_phil as before, .show() and .extract() now produce three entries each:

## extract code begin: freephil_examples.py

working_phil = base_phil.fetch(source=user_phil)
working_phil.show()
print working_phil.extract().plot

## extract code end

Output:

plot {
  style = line bar pie_chart
  title = None
}
plot {
  style = *line bar pie_chart
  title = Line plot (default in base)
}
plot {
  style = line *bar pie_chart
  title = Bar plot (provided by user)
}
[<freephil.scope_extract object at 0x2af4c307bcd0>,
 <freephil.scope_extract object at 0x2af4c307bd50>,
 <freephil.scope_extract object at 0x2af4c307be10>]

With .optional = True, the base of a multiple definition or scope is never extracted. With .optional = False, it is always extracted, and always first in the list.

The “always first in the list” rule for multiple base objects is special. Other instances of multiple scopes are shown and extracted in the order in which they appear in the base file and the merged user file(s), with all exact duplicates removed. If duplicates are detected, the earlier copy is removed, unless it is the base.

These rules are designed to produce easily predictable results in situations where multiple Phil files are merged (via .fetch()), including complete copies of the base file.

1.8.4. Available .type 

FreePHIL specifies following optinos for attribute .type:

.type =
  words     retains the "words" produced by the parser
  strings   list of Python strings (also used for .type = None)
  str       combines all words into one string
  path      path name (same as str_converters)
  key       database key (same as str_converters)
  bool      Python bool
  int       Python int
  float     Python float
  choice    string selected from a pre-defined list

See Extending Phil for instructions, how to add custom options.

1.8.5. .type = ints and .type = floats 

The built-in ints and floats converters handle lists of integer and floating point numbers, respectively. For example:

## extract code begin: freephil_examples.py

base_phil = parse("""
  random_integers = None
    .type = ints
  euler_angles = None
    .type = floats(size=3)
  unit_cell_parameters = None
    .type = floats(size_min=1, size_max=6)
  rotation_part = None
    .type = ints(size=9, value_min=-1, value_max=1)
  """)

user_phil = parse("""
  random_integers = 3 18 5
  euler_angles = 10 -20 30
  unit_cell_parameters = 10,20,30
  rotation_part = "1,0,0;0,-1,0;0,0,-1"
  """)

working_phil = base_phil.fetch(source=user_phil)
working_phil.show()
print
working_params = working_phil.extract()
print working_params.random_integers
print working_params.euler_angles
print working_params.unit_cell_parameters
print working_params.rotation_part
print
working_phil = base_phil.format(python_object=working_params)
working_phil.show()

## extract code end

Output:

random_integers = 3 18 5
euler_angles = 10 -20 30
unit_cell_parameters = 10,20,30
rotation_part = "1,0,0;0,-1,0;0,0,-1"

[3, 18, 5]
[10.0, -20.0, 30.0]
[10.0, 20.0, 30.0]
[1, 0, 0, 0, -1, 0, 0, 0, -1]

random_integers = 3 18 5
euler_angles = 10 -20 30
unit_cell_parameters = 10 20 30
rotation_part = 1 0 0 0 -1 0 0 0 -1

The list of random_integers can have arbitrary size and arbitrary values.

For euler_angles, exactly three values must be given.

For unit_cell_parameters, one to six values are acceptable.

The list of values for rotation_part must have nine integer elements, with values {-1,0,1}.

All keywords are optional and can be used in any combination, except if size is given, size_min and size_max cannot also be given.

Lists of values can optionally use commas or semicolons as separators between values. In this context, both characters are equivalent to a white-space. .format() always uses spaces as separators, i.e. commas and semicolons are not preserved in an .extract()-.format() cycle.

Note

Lists using semicolons as separators must be quoted; see the Semicolon syntax section below.

1.8.6. .type = choice 

The built-in choice converters support single and multi choices. Here are two examples, a single choice gender and a multi choice favorite_sweets:

## extract code begin: freephil_examples.py

base_phil = parse("""
  gender = male female
    .type = choice
  favorite_sweets = ice_cream chocolate candy_cane cookies
    .type = choice(multi=True)
  """)

jims_choices = parse("""
  gender = *male female
  favorite_sweets = *ice_cream chocolate candy_cane *cookies
  """)

jims_phil = base_phil.fetch(source=jims_choices)
jims_phil.show()
jims_params = jims_phil.extract()
print jims_params.gender, jims_params.favorite_sweets

## extract code end

Selected items are marked with a star *. The .extract() method returns either a string with the selected value (single choice) or a list of strings with all selected values (multi choice). The output of the example is:

gender = *male female
favorite_sweets = *ice_cream chocolate candy_cane *cookies
male ['ice_cream', 'cookies']

To maximize convenience, especially for choices specified via the command-line, the * is optional if only one value is given. For example, the following two definitions are equivalent:

gender = female
gender = male *female

If the .optional attribute is not defined, it defaults to True and this is possible:

## extract code begin: freephil_examples.py

ignorant_choices = parse("""
  gender = male female
  favorite_sweets = ice_cream chocolate candy_cane cookies
  """)

ignorant_params = base_phil.fetch(source=ignorant_choices).extract()
print ignorant_params.gender, ignorant_params.favorite_sweets

## extract code end

Output:

None []

In this case the application has to deal with the None and the empty list. If .optional = False, .extract() will lead to informative error messages. The application will never receive None or an empty list.

If a value in the user file is not a possible choice, .extract() leads to an error message listing all possible choices, for example:

Sorry: Not a possible choice for favorite_sweets: icecream
  Possible choices are:
    ice_cream
    chocolate
    candy_cane
    cookies

This message is designed to aid users in recovering from mis-spelled choices typed in at the command-line. Command-line choices are further supported by this syntax:

## extract code begin: freephil_examples.py

greedy_choices = parse("""
  favorite_sweets=ice_cream+chocolate+cookies
  """)

greedy_params = base_phil.fetch(source=greedy_choices).extract()
print greedy_params.favorite_sweets

## extract code end

Ouput:

['ice_cream', 'chocolate', 'cookies']

Finally, if the .optional attribute is not specified or True, None can be assigned:

## extract code begin: freephil_examples.py

no_thanks_choices = parse("""
  favorite_sweets=None
  """)

no_thanks_params = base_phil.fetch(source=no_thanks_choices).extract()
print no_thanks_params.favorite_sweets

## extract code end

Output:

[]

1.8.7. Includes 

Phil also supports merging of files at the parsing level. For example:

include file general.params

minimization.parameters {
  include file specific.params
}

Another option for building base files from a library of building blocks is based on Python’s import mechanism. For example:

include file general.params

minimization.parameters {
  include scope app.module.base_phil
}

When encountering the include scope, the Phil parser automatically imports app.module (equivalent to import app.module in a Python script). The base_phil object in the imported module must be a pre-parsed Phil scope or a plain Phil string. The content of the base_phil scope is inserted into the scope of the include scope statement.

include directives enable hierarchical building of base files without the need to copy-and-paste large fragments explicitly. Duplication appears only in automatically generated user files. I.e. the programmer is well served because a system of base files can be kept free of large-scale redundancies that are difficult to maintain. At the same time the end user is well served because the indirections are resolved automatically and all parameters are presented in one uniform view.

Note

Parsing of the include statements has to be explicitly enabled when calling freephil.parse() with process_includes = True.

1.8.8. Variable substitution 

Phil supports variable substitution using $var and $(var) syntax. A few examples say more than many words:

## extract code begin: freephil_examples.py

var_phil = parse("""
  root_name = peak
  file_name = $root_name.mtz
  full_path = $HOME/$file_name
  related_file_name = $(root_name)_data.mtz
  message = "Reading $file_name"
  as_is = ' $file_name '
  """)
var_phil.fetch(source=var_phil).show()

## extract code end

Output:

root_name = peak
file_name = "peak.mtz"
full_path = "/net/cci/rwgk/peak.mtz"
related_file_name = "peak_data.mtz"
message = "Reading peak.mtz"
as_is = ' $file_name '

Note that the variable substitution does not happen during parsing. The output of params.show is identical to the input. In the example above, variables are substituted by the .fetch() method that we introduced earlier to merge user files given a base file.

1.8.9. Semicolon syntax 

In all the examples above, line breaks act as syntactical elements delineating the end of definitions. This is most obvious, but for convenience, Phil also supports using the semicolon ; instead. For example:

## extract code begin: freephil_examples.py

phil_scope = parse("""
   quick .multiple=true;.optional=false{and=very;.type=str;dirty=use only on command-lines, please!;.type=str}
   """)

phil_scope.show(attributes_level=2)

## extract code end

Clearly, the output looks much nicer:

quick
  .optional = False
  .multiple = True
{
  and = very
    .type = str
  dirty = use only on command-lines, please!
    .type = str
}

Base files generally shouldn’t make use of the semicolon syntax, even though it is possible. In user files it is more acceptable, but the main purpose is to support passing parameters from the command line.

Note that the Phil output methods (.show(), .as_str()) never make use of the semicolon syntax.

1.8.10. Phil comments 

Phil supports two types of comments:

Simple one-line comments starting with a hash #. All following characters through the end of the line are ignored.

Syntax-aware comments starting with an exclamation mark !.

The exclamation mark can be used to easily comment out entire syntactical constructs, for example a complete scope including all attributes:

## extract code begin: freephil_examples.py

base_phil = parse("""
  !input {
    file_name = None
      .type = path
      .multiple = True
  }
  """)
base_phil.show()

## extract code end

Output:

!input {
  file_name = None
}

As is evident from the output, Phil keeps the content “in mind”, but the scope is not actually used by .fetch():

## extract code begin: freephil_examples.py

user_phil = parse("""
  input.file_name = experiment.dat
  """)
print len(base_phil.fetch(source=user_phil).as_str())

## extract code end

Output:

I.e. the .fetch() method ignored the user definition because the corresponding base is commented out.

1.8.11. Quotes and backslash 

Similar to Python, Phil supports single quotes, double quotes, and triple quotes (three single or three double quotes). Unlike Python, quotes can often be omitted, and single quotes and double quotes have different semantics, similar to that of Unix shells: $ variables are expanded if embedded in double quotes, but not if embedded in single quotes. See the variable substitution section above for examples.

The backslash can be used in the usual way (Python, Unix shells) to “escape” line breaks, quotes, and a second backslash.

For convenience, a line starting with quotes is automatically treated as a continuation of a definition on the previous line(s). The trailing backslash on the previous line may be omitted.

The exact rules for quoting and backslash escapes are intricate. A significant effort was made to mimic the familiar behavior of Python and Unix shells where possible, but nested constructs of quotes and backslashes are still prone to cause surprises. In unusual situations, probably the fastest method to obtain the desired result is trial and error (as opposed to studying the intricate rules).

1.9. More on coding usage 

1.9.1. scope.show 

In this document, the freephil.scope.show() method is used extensively in the examples. With the defaults for the method parameters, it only shows the Phil scope or definition names and associated values. It is also possible to include some or all Phil scope or definition attributes in the .show() output, as directed by the attributes_level parameter:

attributes_level=0: shows only names and values
                 1: also shows the .help attribute
                 2: shows all attributes which are not None
                 3: shows all attributes

scope.show(attributes_level=2) can be used to pretty-print base files without any loss of information. attributes_level=3 is useful to obtain a full listing of all available attributes, but all information is preserved with the usually much less verbose attributes_level=2. This is illustrated by the following example:

## extract code begin: freephil_examples.py

base_phil = parse("""
  minimization {
    input
      .help = "File names and data labels."
      .multiple = True
    {
      file_name = None
        .type = path
      label = None
        .help = "A unique substring of the data label is sufficient."
        .type = str
    }
  }
  """)

for attributes_level in range(4):
  base_phil.show(attributes_level=attributes_level)

## extract code end

Output with attributes_level=0 (the default):

minimization {
  input {
    file_name = None
    label = None
  }
}

Output with attributes_level=1:

minimization {
  input
    .help = "File names and data labels."
  {
    file_name = None
    label = None
      .help = "A unique substring of the data label is sufficient."
  }
}

Output with attributes_level=2:

minimization {
  input
    .help = "File names and data labels."
    .multiple = True
  {
    file_name = None
      .type = path
    label = None
      .help = "A unique substring of the data label is sufficient."
      .type = str
  }
}

Output with attributes_level=3:

minimization
  .style = None
  .help = None
  .caption = None
  .short_caption = None
  .optional = None
  .call = None
  .multiple = None
  .sequential_format = None
  .disable_add = None
  .disable_delete = None
  .expert_level = None
{
  input
    .style = None
    .help = "File names and data labels."
    .caption = None
    .short_caption = None
    .optional = None
    .call = None
    .multiple = True
    .sequential_format = None
    .disable_add = None
    .disable_delete = None
    .expert_level = None
  {
    file_name = None
      .help = None
      .caption = None
      .short_caption = None
      .optional = None
      .type = path
      .multiple = None
      .input_size = None
      .expert_level = None
    label = None
      .help = "A unique substring of the data label is sufficient."
      .caption = None
      .short_caption = None
      .optional = None
      .type = str
      .multiple = None
      .input_size = None
      .expert_level = None
  }
}

1.9.2. fetch option: track_unused_definitions 

The default behavior of freephil.scope.fetch() is to simply ignore user definitions that don’t match anything in the base file. It it is possible to request a complete list of all user definitions ignored by .fetch(). For example:

## extract code begin: freephil_examples.py

base_phil = parse("""
  input {
    file_name = None
      .type = path
  }
  """)

user_phil = parse("""
  input {
    file_name = experiment.dat
    label = set1
    lable = set2
  }
  """)

working_phil, unused = base_phil.fetch(
  source=user_phil, track_unused_definitions=True)
working_phil.show()
for object_locator in unused:
  print "unused:", object_locator

## extract code end

Output:

input {
  file_name = experiment.dat
}
unused: input.label (input line 4)
unused: input.lable (input line 5)

To catch spelling errors, or to alert users to changes in the base file, it is good practice to set track_unused_definitions=True and to show warnings or errors.

1.10. Extending Phil 

Phil comes with a number of predefined converters used by .extract() and .format() to convert to and from pure Python objects. These are:

.type =
  words     retains the "words" produced by the parser
  strings   list of Python strings (also used for .type = None)
  str       combines all words into one string
  path      path name (same as str_converters)
  key       database key (same as str_converters)
  bool      Python bool
  int       Python int
  float     Python float
  choice    string selected from a pre-defined list

It is possible to extend Phil with user-defined converters. For example:

## extract code begin: freephil_examples.py

import freephil
from freephil import tokenizer

class upper_converters:

  phil_type = "upper"

  def __str__(self): return self.phil_type

  def from_words(self, words, master):
    s = freephil.str_from_words(words=words)
    if (s is None): return None
    return s.upper()

  def as_words(self, python_object, master):
    if (python_object is None):
      return [tokenizer.word(value="None")]
    return [tokenizer.word(value=python_object.upper())]

converter_registry = freephil.extended_converter_registry(
  additional_converters=[upper_converters])

## extract code end

The extended converter_registry is passed as an additional argument to Phil’s parse function:

## extract code begin: freephil_examples.py

base_phil = parse("""
  value = None
    .type = upper
  """,
    converter_registry=converter_registry)
user_phil = parse("value = extracted")
working_params = base_phil.fetch(source=user_phil).extract()
print working_params.value

## extract code end

The print statement at the end writes “EXTRACTED”. It also goes the other way, starting with a lower-case Python value:

## extract code begin: freephil_examples.py

working_params.value = "formatted"
working_phil = base_phil.format(python_object=working_params)
working_phil.show()

## extract code end

The output of the .show() call is “value = FORMATTED”.

Arbitrary new types can be added to Phil by defining similar converters. If desired, the pre-defined converters for the basic types can even be replaced. All converters have to have __str__(), from_words() and as_words() methods. More complex converters may optionally have a non-trivial __init__() method (an example is the choice_converters class in freephil/__init__.py).

Additional domain-specific converters are best defined in a separate module, along with a corresponding parse() function using the extended converter registry as the default. See, for example, iotbx/phil.py in the same cctbx project that also hosts libtbx.