aztk.models package

class aztk.models.ApplicationLog(name: str, cluster_id: str, log: str, total_bytes: int, application_state: azure.batch.models.batch_service_client_enums.TaskState, exit_code: int)

Bases: object

class aztk.models.Cluster(pool: azure.batch.models.cloud_pool_py3.CloudPool, nodes: azure.batch.models.compute_node_paged.ComputeNodePaged = None)

Bases: object

class aztk.models.ClusterConfiguration(*args, **kwargs)

Bases: aztk.core.models.model.Model

Cluster configuration model

Parameters:

• cluster_id (str) – Id of the Aztk cluster
• toolkit (aztk.models.Toolkit) – Toolkit to be used for this cluster
• size (int) – Number of dedicated nodes for this cluster
• size_low_priority (int) – Number of low priority nodes for this cluster
• vm_size (int) – Azure Vm size to be used for each node
• subnet_id (str) – Full resource id of the subnet to be used(Required for mixed mode clusters)
• plugins (List[aztk.models.plugins.PluginConfiguration]) – List of plugins to be used
• file_shares (List[aztk.models.FileShare]) – List of File shares to be used
• user_configuration (aztk.models.UserConfiguration) – Configuration of the user to be created on the master node to ssh into.

cluster_id

Model String field

toolkit

Field is another model

Parameters:

• model (aztk.core.models.Model) – Model object that field should be
• merge_strategy (ModelMergeStrategy) – When merging models how should the nested model be merged. Default: ModelMergeStrategy.merge

size

Model Integer field

size_low_priority

Model Integer field

vm_size

Model String field

subnet_id

Model String field

plugins

Field that should be a list

file_shares

Field that should be a list

user_configuration

Field is another model

Parameters:

• model (aztk.core.models.Model) – Model object that field should be
• merge_strategy (ModelMergeStrategy) – When merging models how should the nested model be merged. Default: ModelMergeStrategy.merge

scheduling_target

Field that should be an enum

mixed_mode() → bool

Returns:if the pool is using mixed mode(Both dedicated and low priority nodes)

gpu_enabled()

get_docker_repo()

get_docker_run_options() → str

class aztk.models.ClusterState

Bases: enum.Enum

An enumeration.

deleting = 'deleting'

resizing = 'resizing'

steady = 'steady'

stopping_resize = 'stopping'

class aztk.models.DockerConfiguration(**kwargs)

Bases: aztk.core.models.model.Model

Configuration for connecting to private docker

Parameters:

• endpoint (str) – Which docker endpoint to use. Default to docker hub.
• username (str) – Docker endpoint username
• password (str) – Docker endpoint password

endpoint

Model String field

username

Model String field

password

Model String field

class aztk.models.Enum

Bases: object

Generic enumeration.

Derive from this class to define new enumerations.

class aztk.models.File(name: str, payload: _io.StringIO)

Bases: object

class aztk.models.FileShare(**kwargs)

Bases: aztk.core.models.model.Model

storage_account_name

Model String field

storage_account_key

Model String field

file_share_path

Model String field

mount_path

Model String field

class aztk.models.Model(**kwargs)

Bases: object

Base class for all aztk models

To implement model wide validation implement __validate__ method

__getstate__()

For pickle serialization. This return the state of the model

__setstate__(state)

For pickle serialization. This update the current model with the given state

validate()

Validate the entire model

merge(other)

classmethod from_dict(val: dict)

to_dict()

class aztk.models.NodeOutput(id: str, output: Union[tempfile.SpooledTemporaryFile, str] = None, error: Exception = None)

Bases: object

class aztk.models.PluginConfiguration(**kwargs)

Bases: aztk.core.models.model.Model

Plugin manifest that should be returned in the main.py of your plugin

Args

name: Name of the plugin. Used to reference the plugin runOn: Where the plugin should run execute: Path to the file to execute(This must match the target of one of the files) files: List of files to upload args: List of arguments to pass to the executing script env: Dict of environment variables to pass to the script

name

Model String field

files

Field that should be a list

execute

Model String field

args

Field that should be a list

env

Field that should be a list

target

Field that should be an enum

target_role

Field that should be an enum

ports

Field that should be a list

has_arg(name: str)

class aztk.models.PluginFile(target: str = None, local_path: str = None)

Bases: aztk.core.models.model.Model

Reference to a file for a plugin.

target

Model String field

local_path

Model String field

content()

class aztk.models.PluginPort(**kwargs)

Bases: aztk.core.models.model.Model

Definition for a port that should be opened on node :param internal: Port on the node :param public: [Optional] Port available to the user. If none won’t open any port to the user :param name: [Optional] name to differentiate ports if you have multiple

internal

Model Integer field

public

Base class for all model fields

name

Model Integer field

expose_publicly

public_port

class aztk.models.PluginTarget

Bases: enum.Enum

Where this plugin should run

SparkContainer = 'spark-container'

Host = 'host'

class aztk.models.PluginTargetRole

Bases: enum.Enum

An enumeration.

Master = 'master'

Worker = 'worker'

All = 'all-nodes'

class aztk.models.PortForwardingSpecification(**kwargs)

Bases: aztk.core.models.model.Model

remote_port

Model Integer field

local_port

Model Integer field

class aztk.models.RemoteLogin(ip_address, port)

Bases: object

class aztk.models.SSHLog(output, node_id)

Bases: object

class aztk.models.SchedulingTarget

Bases: enum.Enum

Target where task will get scheduled. For spark this is where the driver will live.

Master = 'master'

Only master is allowed to run task

Any = 'any'

Any node(Not recommended if using low pri) (Default)

class aztk.models.SecretsConfiguration(**kwargs)

Bases: aztk.core.models.model.Model

service_principal

Field is another model

Parameters:

• model (aztk.core.models.Model) – Model object that field should be
• merge_strategy (ModelMergeStrategy) – When merging models how should the nested model be merged. Default: ModelMergeStrategy.merge

shared_key

Field is another model

Parameters:

• model (aztk.core.models.Model) – Model object that field should be
• merge_strategy (ModelMergeStrategy) – When merging models how should the nested model be merged. Default: ModelMergeStrategy.merge

docker

Field is another model

Parameters:

• model (aztk.core.models.Model) – Model object that field should be
• merge_strategy (ModelMergeStrategy) – When merging models how should the nested model be merged. Default: ModelMergeStrategy.merge

ssh_pub_key

Model String field

ssh_priv_key

Model String field

is_aad()

class aztk.models.ServicePrincipalConfiguration(**kwargs)

Bases: aztk.core.models.model.Model

Container class for AAD authentication

tenant_id

Model String field

client_id

Model String field

credential

Model String field

batch_account_resource_id

Model String field

storage_account_resource_id

Model String field

class aztk.models.SharedKeyConfiguration(**kwargs)

Bases: aztk.core.models.model.Model

Container class for shared key authentication

batch_account_name

Model String field

batch_account_key

Model String field

batch_service_url

Model String field

storage_account_name

Model String field

storage_account_key

Model String field

storage_account_suffix

Model String field

class aztk.models.Software

Bases: object

Enum with list of available softwares

spark = 'spark'

class aztk.models.Task(**kwargs)

Bases: aztk.core.models.model.Model

id

Model String field

node_id

Model String field

state

Model String field

state_transition_time

Model String field

command_line

Model String field

exit_code

Model Integer field

start_time

Field that should be an datetime

end_time

Field that should be an datetime

failure_info

Model String field

class aztk.models.TaskState

Bases: enum.Enum

An enumeration.

Running = 'running'

Completed = 'completed'

Failed = 'failed'

Preparing = 'preparing'

class aztk.models.TextPluginFile(target: str, content: Union[str, _io.StringIO])

Bases: aztk.core.models.model.Model

Reference to a file for a plugin.

Args: target (str): Where should the file be uploaded relative to the plugin working dir content (str|io.StringIO): Content of the file. Can either be a string or a StringIO

target

Model String field

content()

class aztk.models.Toolkit(**kwargs)

Bases: aztk.core.models.model.Model

Toolkit for a cluster. This will help pick the docker image needed

Parameters:

• software (str) – Name of the toolkit(spark)
• version (str) – Version of the toolkit
• environment (str) – Which environment to use for this toolkit
• environment_version (str) – If there is multiple version for an environment you can specify which one
• docker_repo (str) – Optional docker repo
• docker_run_options (str) – Optional command-line options for docker run

software

Model String field

version

Model String field

environment

Model String field

environment_version

Model String field

docker_repo

Model String field

docker_run_options

Model String field

get_docker_repo(gpu: bool)

get_docker_run_options()

class aztk.models.Union

Bases: typing.Final

Union type; Union[X, Y] means either X or Y.

To define a union, use e.g. Union[int, str]. Details:

• The arguments must be types and there must be at least one.

• None as an argument is a special case and is replaced by type(None).

• Unions of unions are flattened, e.g.:

  Union[Union[int, str], float] == Union[int, str, float]
  

• Unions of a single argument vanish, e.g.:

  Union[int] == int  # The constructor actually returns int
  

• Redundant arguments are skipped, e.g.:

  Union[int, str, int] == Union[int, str]
  

• When comparing unions, the argument order is ignored, e.g.:

  Union[int, str] == Union[str, int]
  

• When two arguments have a subclass relationship, the least derived argument is kept, e.g.:

  class Employee: pass
  class Manager(Employee): pass
  Union[int, Employee, Manager] == Union[int, Employee]
  Union[Manager, int, Employee] == Union[int, Employee]
  Union[Employee, Manager] == Employee
  

• Corollary: if Any is present it is the sole survivor, e.g.:

  Union[int, Any] == Any
  

• Similar for object:

  Union[int, object] == object
  

• To cut a tie: Union[object, Any] == Union[Any, object] == Any.

• You cannot subclass or instantiate a union.

• You cannot write Union[X][Y] (what would it mean?).

• You can use Optional[X] as a shorthand for Union[X, None].

class aztk.models.UserConfiguration(**kwargs)

Bases: aztk.core.models.model.Model

username

Model String field

ssh_key

Model String field

password

Model String field

class aztk.models.VmImage(publisher, offer, sku)

Bases: object