API documentation¶
Python bindings for yMMSL.
This package contains all the classes needed to represent a yMMSL file, as well as to read and write yMMSL files.
-
class
CheckpointRule
¶ Defines a checkpoint rule.
There are two flavors of rules:
CheckpointRangeRule
andCheckpointAtRule
. Do not use this class directly.
-
class
CheckpointRangeRule
(start: Union[float, int, None] = None, stop: Union[float, int, None] = None, every: Union[float, int] = 0)¶ Defines a range of checkpoint moments.
If
start
is supplied, this rule specifies a checkpoint at:start
,start + every
,start + 2*every
, etc., for as long asstart + n*every <= stop
, withn
a whole number. If stop is not given, the range continues indefinitely.Start may be omitted, in which case a checkpoint is defined for
0
,every
,2*every
, etc. Note that in this case the range also extends to negative numbers (-every
,-2*every
, etc.), as simulated time may be negative (e.g. in rocket launch,t=0
is generally taken as lift-off time, but events already take place before that moment).-
start
¶ Start of the range.
-
stop
¶ Stopping criterium of the range.
-
every
¶ Step size of the range, must be positive.
-
-
class
CheckpointAtRule
(at: Optional[List[Union[float, int]]])¶ Defines an “at” checkpoint rule.
An “at” checkpoint rule creates a snapshot at the specified moments.
-
at
¶ List of checkpoints.
-
-
class
Checkpoints
(at_end: bool = False, wallclock_time: Optional[List[CheckpointRule]] = None, simulation_time: Optional[List[CheckpointRule]] = None)¶ Defines checkpoints in a configuration.
There are three checkpoint triggers: at_end, wallclock_time and simulation_time. The at_end trigger specifies that a checkpoint should be created just before the workflow finishes. The wallclock_time trigger is based on the elapsed real time since starting the muscle_manager in seconds. The simulation_time trigger is based on the time in the simulation as reported by the instances.
Note that the simulation_time trigger assumes a shared concept of time among all components of the model.
-
at_end
¶ Whether a checkpoint should be created just before ending the workflow.
-
wallclock_time
¶ Checkpoint rules for the wallclock_time trigger.
-
simulation_time
¶ Checkpoint rules for the simulation_time trigger.
-
update
(overlay: Checkpoints) → None¶ Update this checkpoints with the given overlay.
Sets at_end to True if it is set in the overlay, otherwise at_end remains as is. Updates the checkpoint rules for wallclock time and simulation time. See
CheckpointRules.update()
.
-
-
class
Component
(name: str, implementation: Optional[str] = None, multiplicity: Union[None, int, List[int]] = None, ports: Optional[Ports] = None)¶ An object declaring a simulation component.
Simulation components are things like submodels, scale bridges, proxies, and any other program that makes up a model. This class represents a declaration of a set of instances of a simulation component, and it’s used to describe which instances are needed to perform a certain simulation.
-
name
¶ The name of this component.
Type: ymmsl.Reference
-
implementation
¶ A reference to the implementation to use.
Type: ymmsl.Reference
-
multiplicity
¶ The shape of the array of instances that execute simultaneously.
Type: List[int]
-
ports
¶ The ports of this component, organised by operator. None if not specified.
Type: Optional[Ports]
-
instances
() → List[Reference]¶ Creates a list of instances needed.
Returns: A list with one Reference for each instance of this component.
-
-
class
Conduit
(sender: str, receiver: str)¶ A conduit transports data between simulation components.
A conduit has two endpoints, which are references to a port on a simulation component. These references must be of one of the following forms:
- component.port
- namespace.component.port (or several namespace prefixes)
-
sender
¶ The sending port that this conduit is connected to.
-
receiver
¶ The receiving port that this conduit is connected to.
-
receiving_component
() → Reference¶ Returns a reference to the receiving component.
-
receiving_port
() → Identifier¶ Returns the identity of the receiving port.
-
receiving_slot
() → List[int]¶ Returns the slot on the receiving port.
If no slot was given, an empty list is returned.
Returns: A list of slot indexes.
-
sending_component
() → Reference¶ Returns a reference to the sending component.
-
sending_port
() → Identifier¶ Returns the identity of the sending port.
-
sending_slot
() → List[int]¶ Returns the slot on the sending port.
If no slot was given, an empty list is returned.
Returns: A list of slot indexes.
-
class
Configuration
(model: Model, settings: Optional[Settings] = None, implementations: Union[List[Implementation], Dict[Reference, Implementation]] = [], resources: Union[Sequence[ResourceRequirements], MutableMapping[Reference, ResourceRequirements]] = [], description: Optional[str] = None, checkpoints: Optional[Checkpoints] = None, resume: Optional[Dict[Reference, pathlib.Path]] = None)¶ Configuration that includes all information for a simulation.
PartialConfiguration has some optional attributes, because we want to allow configuration files which only contain some of the information needed to run a simulation. At some point however, you need all the bits, and this class requires them.
When loading a yMMSL file, you will automatically get an object of this class if all the required components are there; if the file is incomplete, you’ll get a PartialConfiguration instead.
-
model
¶ A model to run.
-
settings
¶ Settings to run the model with.
-
implementations
¶ Implementations to use to run the model. Dictionary mapping implementation names (as References) to Implementation objects.
-
resources
¶ Resources to allocate for the model components. Dictionary mapping component names to Resources objects.
-
description
¶ A human-readable description of the configuration.
-
checkpoints
¶ Defines when each model component should create a snapshot
-
resume
¶ Defines what snapshot each model component should resume from
-
check_consistent
() → None¶ Checks that the configuration is internally consistent.
This checks whether all conduits are connected to existing components, that there’s an implementation for every component, and that resources have been requested for each component.
If any of these requirements is false, this function will raise a RuntimeError with an explanation of the problem.
-
-
dump
(config: PartialConfiguration) → str¶ Converts a yMMSL configuration to a string containing YAML.
Parameters: config – The configuration to be saved to yMMSL. Returns: A yMMSL YAML description of the given document.
-
class
ExecutionModel
¶ Describes how to start a model component.
-
DIRECT
= 1¶ Start directly on the allocated core(s), without MPI.
-
INTELMPI
= 3¶ Start using Intel MPI’s mpirun.
-
OPENMPI
= 2¶ Start using OpenMPI’s mpirun.
-
SRUNMPI
= 4¶ Start MPI implementation using srun.
-
-
class
Identifier
(seq: Any)¶ A custom string type that represents an identifier.
An identifier may consist of upper- and lowercase characters, digits, and underscores.
-
class
Implementation
(name: Reference, modules: Union[str, List[str], None] = None, virtual_env: Optional[pathlib.Path] = None, env: Optional[Dict[str, str]] = None, execution_model: ExecutionModel = <ExecutionModel.DIRECT: 1>, executable: Optional[pathlib.Path] = None, args: Union[str, List[str], None] = None, script: Union[str, List[str], None] = None, can_share_resources: bool = True, keeps_state_for_next_use: KeepsStateForNextUse = <KeepsStateForNextUse.NECESSARY: 1>)¶ Describes an installed implementation.
An Implementation normally has an
executable
and any other needed attributes, withscript
set to None. You should specify a script only as a last resort, probably after getting some help from the authors of this library. If a script is specified, all other attributes except for the name must beNone
.For
execution_model
, the following values are supported:- direct
- The program will be executed directly. Use this for non-MPI programs.
- openmpi
- For MPI programs that should be started using OpenMPI’s mpirun.
- intelmpi
- For MPI programs that should be started using Intel MPI’s mpirun.
The
can_share_resources
attribute describes whether this implementation can share resources (cores) with other components in a macro-micro coupling. Set this toFalse
if the implementation does significant computing inside of its time update loop after having sent messages on its O_I port(s) but before receiving messages on its S port(s). In the unlikely case that it’s doing significant computing before receiving for F_INIT or after sending its O_F messages, likewise set this toFalse
.Setting this to
False
unnecessarily will waste core hours, setting it toTrue
incorrectly will slow down your simulation.-
name
¶ Name of the implementation
-
modules
¶ HPC software modules to load
-
virtual_env
¶ Path to a virtual env to activate
-
env
¶ Environment variables to set
-
execution_model
¶ How to start the executable
-
executable
¶ Full path to executable to run
-
args
¶ Arguments to pass to the executable
-
script
¶ A script that starts the implementation
Whether this implementation can share resources (cores) with other components or not
-
keeps_state_for_next_use
¶ Does this implementation keep state for the next iteration of the reuse loop. See
ImplementationState
.
-
class
KeepsStateForNextUse
¶ Describes whether an implementation keeps internal state between iterations of the reuse loop.
See also Keeps state for next use.
-
HELPFUL
= 3¶ The implementation has an internal state, though this could be regenerated. Doing so may be expensive.
-
NECESSARY
= 1¶ The implementation has an internal state that is necessary for continuing the implementation.
-
NO
= 2¶ The implementation has no internal state.
-
-
load
(source: Union[str, pathlib.Path, IO[Any]]) → PartialConfiguration¶ Loads a yMMSL document from a string or a file.
Parameters: source – A string containing yMMSL data, a pathlib Path to a file containing yMMSL data, or an open file-like object containing from which yMMSL data can be read. Returns: A PartialConfiguration object corresponding to the input data.
-
class
Model
(name: str, components: List[Component], conduits: Optional[List[Union[Conduit, MulticastConduit]]] = None)¶ Describes a simulation model.
A model consists of a number of components connected by conduits.
Note that there may be no conduits, if there is only a single component. In that case, the conduits argument may be omitted when constructing the object, and also from the YAML file; the conduits attribute will then be set to an empty list.
-
name
¶ The name by which this simulation model is known to the system.
-
components
¶ A list of components making up the model.
-
conduits
¶ A list of conduits connecting the components.
-
check_consistent
() → None¶ Checks that the model is internally consistent.
This checks whether all conduits are connected to existing components, and will raise a RuntimeError with an explanation if one is not.
-
update
(overlay: Model) → None¶ Overlay another model definition on top of this one.
This updates the object with the name, components and conduits given in the argument. The name is overwritten, and components are overwritten if they have the same name as an existing argument or else added.
Conduits are added. If a receiving port was already connected, the old conduit is removed. If a sending port was already connected, the new conduit is added and the sending port acts as a multicast port.
Parameters: overlay – A Model definition to overlay on top of this one.
-
-
class
ModelReference
(name: str)¶ Describes a reference (by name) to a model.
-
name
¶ The name of the simulation model this refers to.
-
-
class
MPICoresResReq
(name: Reference, mpi_processes: int, threads_per_mpi_process: int = 1)¶ Describes resources for simple MPI implementations.
This allocates individual cores or sets of cores on the same node for a given number of MPI processes per instance.
-
name
¶ Name of the component to configure.
-
mpi_processes
¶ Number of MPI processes to start.
-
threads_per_mpi_process
¶ Number of threads/cores per process.
-
-
class
MPINodesResReq
(name: Reference, nodes: int, mpi_processes_per_node: int, threads_per_mpi_process: int = 1)¶ Describes resources for node based MPI implementations.
This allocates resources for an MPI process in terms of nodes and cores, processes and threads on them.
-
name
¶ Name of the component to configure.
-
nodes
¶ Number of nodes to reserve.
-
mpi_processes_per_node
¶ Number of MPI processes to start on each node.
-
threads_per_mpi_process
¶ Number of threads/cores per process.
-
-
class
Operator
¶ An operator of a component.
This is a combination of the Submodel Execution Loop operators, and operators for other components such as mappers.
-
F_INIT
= 1¶ Initialisation phase, before start of the SEL
-
NONE
= 0¶ No operator
-
O_F
= 5¶ Observation of final state, after the SEL
-
O_I
= 2¶ State observation within the model’s main loop
-
S
= 3¶ State update in the model’s main loop
-
allows_receiving
() → bool¶ Whether ports on this operator can receive.
-
allows_sending
() → bool¶ Whether ports on this operator can send.
-
-
class
PartialConfiguration
(model: Optional[ModelReference] = None, settings: Optional[Settings] = None, implementations: Union[List[Implementation], Dict[Reference, Implementation], None] = None, resources: Union[Sequence[ResourceRequirements], MutableMapping[Reference, ResourceRequirements], None] = None, description: Optional[str] = None, checkpoints: Optional[Checkpoints] = None, resume: Optional[Dict[Reference, pathlib.Path]] = None)¶ Top-level class for all information in a yMMSL file.
-
model
¶ A model to run.
-
settings
¶ Settings to run the model with.
-
implementations
¶ Implementations to use to run the model. Dictionary mapping implementation names (as References) to Implementation objects.
-
resources
¶ Resources to allocate for the model components. Dictionary mapping component names to ResourceRequirements objects.
-
description
¶ A human-readable description of the configuration.
-
checkpoints
¶ Defines when each model component should create a snapshot
-
resume
¶ Defines what snapshot each model component should resume from
-
as_configuration
() → Configuration¶ Converts to a full Configuration object.
This checks that this PartialConfiguration has all the pieces needed to run a simulation, and if so converts it to a Configuration object.
Note that this doesn’t check references, just that there is a model, implementations and resources. For the more extensive check, see
Configuration.check_consistent()
.Returns: A corresponding Configuration. Raises: ValueError
– If this configuration isn’t complete.
-
update
(overlay: PartialConfiguration) → None¶ Update this configuration with the given overlay.
This will update the model according to
Model.update()
, copy settings from overlay on top of the current settings, overwrite implementations with the same name and add implementations with a new name, and likewise for resources and resume. The description of the overlay is appended to the current description. Checkpoints are updated according toCheckpoints.update()
.Parameters: overlay – A configuration to overlay onto this one.
-
-
class
Port
(name: Identifier, operator: Operator)¶ A port on a component.
Ports are used by component to send or receive messages on. They are connected by conduits to enable communication between components.
-
name
¶ The name of the port.
-
operator
¶ The MMSL operator in which this port is used.
-
-
class
Ports
(f_init: Union[None, str, List[str]] = None, o_i: Union[None, str, List[str]] = None, s: Union[None, str, List[str]] = None, o_f: Union[None, str, List[str]] = None)¶ Ports declaration for a component.
Ports objects compare for equality by value. The names may be specified as a list of strings, or separated by spaces in a single string. If a particular operator has no associated ports, it may be omitted. For example:
ports: f_init: # list of names - a - b o_i: c d # on one line, space-separated s: e # single port # o_f omitted as it has no ports
-
f_init
¶ The ports associated with the F_INIT operator.
-
o_i
¶ The ports associated with the O_I operator
-
s
¶ The ports associated with the S operator.
-
o_f
¶ The ports associated with the O_F operator
-
all_ports
() → Iterable[Port]¶ Returns an iterable containing all ports.
-
operator
(port_name: Identifier) → Operator¶ Looks up the operator for a given port.
Parameters: port_name – Name of the port to look up. Returns: The operator for that port. Raises: KeyError
– If no port with this name was found.
-
port_names
() → Iterable[Identifier]¶ Returns an iterable containing the names of all ports.
-
-
class
Reference
(parts: Union[str, List[Union[Identifier, int]]])¶ A reference to an object in the MMSL execution model.
References in string form are written as either:
- an Identifier,
- a Reference followed by a period and an Identifier, or
- a Reference followed by an integer enclosed in square brackets.
In object form, they consist of a list of Identifiers and ints. The first list item is always an Identifier. For the rest of the list, an Identifier represents a period operator with that argument, while an int represents the indexing operator with that argument.
Reference objects act like a list of Identifiers and ints, you can get their length using len(), iterate through the parts using a loop, and get sublists or individual items using []. Note that the sublist has to be a valid Reference, so it cannot start with an int.
References can be compared for equality to each other or to a plain string, and they can be used as dictionary keys. Reference objects are immutable (or they’re supposed to be anyway), so do not try to change any of the elements. Instead, make a new Reference. Especially References that are used as dictionary keys must not be modified, this will get your dictionary in a very confused state.
-
without_trailing_ints
() → Reference¶ Returns a copy of this Reference with trailing ints removed.
Examples
a.b.c[1][2] -> a.b.c a[1].b.c -> a[1].b.c a.b.c -> a.b.c a[1].b.c[2] -> a[1].b.c
-
class
ResourceRequirements
(name: Reference)¶ Describes resources to allocate for components.
-
name
¶ Name of the component to configure.
-
-
save
(config: PartialConfiguration, target: Union[str, pathlib.Path, IO[Any]]) → None¶ Saves a yMMSL configuration to a file.
Parameters: - config – The configuration to save to yMMSL.
- target – The file to save to, either as a string containing a path, as a pathlib Path object, or as an open file-like object.
-
class
Settings
(settings: Optional[Dict[str, Union[str, int, float, bool, List[float], List[List[float]], yatiml.util.bool_union_fix]]] = None)¶ Settings for doing an experiment.
An experiment is done by running a model with particular settings, for the submodel scales, model parameters and any other configuration.
-
as_ordered_dict
() → collections.OrderedDict¶ Represent as a dictionary of plain built-in types.
- Returns: A dictionary that uses only built-in types, containing
- the configuration.
-
copy
() → Settings¶ Makes a shallow copy of these settings and returns it.
-
ordered_items
() → List[Tuple[Reference, Union[str, int, float, bool, List[float], List[List[float]], yatiml.util.bool_union_fix]]]¶ Return settings as a list of tuples.
-
-
class
ThreadedResReq
(name: Reference, threads: int)¶ Describes resources for threaded implementations.
This includes singlethreaded and multithreaded implementations that do not support MPI. As many cores as specified will be allocated on a single node, for each instance.
-
name
¶ Name of the component to configure.
-
threads
¶ Number of threads/cores per instance.
-