aztk.models package

class aztk.models.ClusterConfiguration(*args, **kwargs)

Bases: aztk.core.models.model.Model

Cluster configuration model

Parameters:

• cluster_id (str) – Id of the Aztk cluster
• toolkit (aztk.models.Toolkit) – Toolkit to be used for this cluster
• size (int) – Number of dedicated nodes for this cluster
• size_low_priority (int) – Number of low priority nodes for this cluster
• vm_size (int) – Azure Vm size to be used for each node
• subnet_id (str) – Full resource id of the subnet to be used(Required for mixed mode clusters)
• plugins (List[aztk.models.plugins.PluginConfiguration]) – List of plugins to be used
• file_shares (List[aztk.models.FileShare]) – List of File shares to be used
• user_configuration (aztk.models.UserConfiguration) – Configuration of the user to be created on the master node to ssh into.

mixed_mode() → bool

Returns:if the pool is using mixed mode(Both dedicated and low priority nodes)

class aztk.models.CustomScript(**kwargs)

Bases: aztk.core.models.model.Model

class aztk.models.DockerConfiguration(**kwargs)

Bases: aztk.core.models.model.Model

Configuration for connecting to private docker

Parameters:

• endpoint (str) – Which docker endpoint to use. Default to docker hub.
• username (str) – Docker endpoint username
• password (str) – Docker endpoint password

class aztk.models.Enum

Bases: object

Generic enumeration.

Derive from this class to define new enumerations.

class aztk.models.FileShare(**kwargs)

Bases: aztk.core.models.model.Model

class aztk.models.Model(**kwargs)

Bases: object

Base class for all aztk models

To implement model wide validation implement __validate__ method

__getstate__()

For pickle serialization. This return the state of the model

__setstate__(state)

For pickle serialization. This update the current model with the given state

validate()

Validate the entire model

class aztk.models.PluginConfiguration(**kwargs)

Bases: aztk.core.models.model.Model

Plugin manifest that should be returned in the main.py of your plugin

Args

name: Name of the plugin. Used to reference the plugin runOn: Where the plugin should run execute: Path to the file to execute(This must match the target of one of the files) files: List of files to upload args: List of arguments to pass to the executing script env: Dict of environment variables to pass to the script

class aztk.models.PluginFile(target: str = None, local_path: str = None)

Bases: aztk.core.models.model.Model

Reference to a file for a plugin.

class aztk.models.PluginPort(**kwargs)

Bases: aztk.core.models.model.Model

Definition for a port that should be opened on node :param internal: Port on the node :param public: [Optional] Port available to the user. If none won’t open any port to the user :param name: [Optional] name to differentiate ports if you have multiple

class aztk.models.PluginTarget

Bases: enum.Enum

Where this plugin should run

class aztk.models.PluginTargetRole

Bases: enum.Enum

An enumeration.

class aztk.models.PortForwardingSpecification(**kwargs)

Bases: aztk.core.models.model.Model

class aztk.models.SchedulingTarget

Bases: enum.Enum

Target where task will get scheduled. For spark this is where the driver will live.

Master = 'master'

Only master is allowed to run task

Dedicated = 'dedicated'

Any dedicated node is allowed to run task(Default)

Any = 'any'

Any node(Not recommended if using low pri)

class aztk.models.SecretsConfiguration(**kwargs)

Bases: aztk.core.models.model.Model

class aztk.models.ServicePrincipalConfiguration(**kwargs)

Bases: aztk.core.models.model.Model

Container class for AAD authentication

class aztk.models.SharedKeyConfiguration(**kwargs)

Bases: aztk.core.models.model.Model

Container class for shared key authentication

class aztk.models.Software

Bases: object

Enum with list of available softwares

class aztk.models.TextPluginFile(target: str, content: Union[str, _io.StringIO])

Bases: aztk.core.models.model.Model

Reference to a file for a plugin.

Args: target (str): Where should the file be uploaded relative to the plugin working dir content (str|io.StringIO): Content of the file. Can either be a string or a StringIO

class aztk.models.Toolkit(**kwargs)

Bases: aztk.core.models.model.Model

Toolkit for a cluster. This will help pick the docker image needed

Parameters:

• software (str) – Name of the toolkit(spark)
• version (str) – Version of the toolkit
• environment (str) – Which environment to use for this toolkit
• environment_version (str) – If there is multiple version for an environment you can specify which one
• docker_repo (str) – Optional docker repo

class aztk.models.Union

Bases: typing.Final

Union type; Union[X, Y] means either X or Y.

To define a union, use e.g. Union[int, str]. Details:

• The arguments must be types and there must be at least one.

• None as an argument is a special case and is replaced by type(None).

• Unions of unions are flattened, e.g.:

  Union[Union[int, str], float] == Union[int, str, float]
  

• Unions of a single argument vanish, e.g.:

  Union[int] == int  # The constructor actually returns int
  

• Redundant arguments are skipped, e.g.:

  Union[int, str, int] == Union[int, str]
  

• When comparing unions, the argument order is ignored, e.g.:

  Union[int, str] == Union[str, int]
  

• When two arguments have a subclass relationship, the least derived argument is kept, e.g.:

  class Employee: pass
  class Manager(Employee): pass
  Union[int, Employee, Manager] == Union[int, Employee]
  Union[Manager, int, Employee] == Union[int, Employee]
  Union[Employee, Manager] == Employee
  

• Corollary: if Any is present it is the sole survivor, e.g.:

  Union[int, Any] == Any
  

• Similar for object:

  Union[int, object] == object
  

• To cut a tie: Union[object, Any] == Union[Any, object] == Any.

• You cannot subclass or instantiate a union.

• You cannot write Union[X][Y] (what would it mean?).

• You can use Optional[X] as a shorthand for Union[X, None].

class aztk.models.UserConfiguration(**kwargs)

Bases: aztk.core.models.model.Model