topsim.core.cluster.Cluster

class topsim.core.cluster.Cluster(env, config)

A class used to represent the Cluster, the abstract representation of computing resources in the Science Data Processor

The Cluster runs on a per-timestep capacity through its run() function in the same way other actors do; however, it’s runtime work is minimal.

The main purpose of the Cluster is to provide access methods for requesting machine data (either aggregate or individual), and for requesting allocations to schedule. The majority of the Cluster is therefore ‘read-only’ from a user perspective. The only situations in which a user will change the system on the cluster is by using a non-default resource provisioning policy (see Notes below).

Parameters:

• env (simpy.Environment) – The environment for the current simulation.

• config (Config) – The configuration object for the simulation. See Simulation for more details.

Notes

TopSim defaults to a ‘free-for-all’ style of resource allocation; unless otherwise stated, a resource that is marked as ‘available’ may be used for any task (provided capacity restrictions are met etc.).

If a SLURM-type resource provisioning approach is wanted, where a portion of resources are allocated to a specific workflow for the duration of that workflow, it is possible to use the provision_batch_resources() class method in your (online) Scheduling algorithm. This associates a set of machines for your workflow based on a provisioning scheme of your design.

The clean-up of resources is completed by the Scheduler once all Task objects in the WorkflowPlan have finished running, and requires no additional code on behalf of the user.

Attributes

finished_tasks	Returns the tasks that are finished on the cluster
env	Simulation Environment object
system_bandwidth	list of Machine objects

Methods

__init__(env, config)	Initialising a Cluster object requires only the Simpy environment and a Config object.
allocate_task_to_cluster(task, machine[, ...])	Receive task from scheduler for allocation to specified machine
check_ingest_capacity(pipeline_demand, ...)	Check if the Cluster has the machine capacity to process the observation Ingest pipeline
clean_up_ingest([c])	Once we finished 'provision ingest', we want to update the cluster status before starting the new timestep
current_available_resources()	Produce a list of current available resources
finished_task_time_data()	Each task in the 'finished' component of 'self.clusters' has the estimated start times, end times, and the actual start and finish times.
get_available_resources([c])
get_finished_tasks([c])	Return a list of the tasks that are finished :param c:
get_idle_resources(observation[, c])	List the resources that are currently provisioned for the specified observation.
get_machine_from_id(id[, c])
is_idle()	Check to see if anything is running on the cluster
is_observation_provisioned(observation[, c])
is_occupied(machine[, observation, c])	Check if the machine is occupied
is_task_finished(task[, c])	Check if task is finished or still running on the cluster
provision_batch_resources(size, name[, c])	Mark a machine on the cluster as being allocated to a workflow, without the allocation place just yet.
provision_ingest_resources(demand, observation)	Based on the requirements of the pipeline, provision a certain number of resources for ingest
release_batch_resources(observation[, c])	For a given observation, release these observations from the idle-resources section and add them to the available resources 'pile'
run()	Start the runtime loop for the cluster, and manage checks on the machine
to_df()