aztk.models package¶
-
class
aztk.models.
ApplicationLog
(name: str, cluster_id: str, log: str, total_bytes: int, application_state: azure.batch.models.batch_service_client_enums.TaskState, exit_code: int)[source]¶ Bases:
object
-
class
aztk.models.
Cluster
(pool: azure.batch.models.cloud_pool_py3.CloudPool, nodes: azure.batch.models.compute_node_paged.ComputeNodePaged = None)[source]¶ Bases:
object
-
class
aztk.models.
ClusterConfiguration
(*args, **kwargs)[source]¶ Bases:
aztk.core.models.model.Model
Cluster configuration model
Parameters: - cluster_id (str) – Id of the Aztk cluster
- toolkit (aztk.models.Toolkit) – Toolkit to be used for this cluster
- size (int) – Number of dedicated nodes for this cluster
- size_low_priority (int) – Number of low priority nodes for this cluster
- vm_size (int) – Azure Vm size to be used for each node
- subnet_id (str) – Full resource id of the subnet to be used(Required for mixed mode clusters)
- plugins (List[aztk.models.plugins.PluginConfiguration]) – List of plugins to be used
- file_shares (List[aztk.models.FileShare]) – List of File shares to be used
- user_configuration (aztk.models.UserConfiguration) – Configuration of the user to be created on the master node to ssh into.
-
cluster_id
¶ Model String field
-
toolkit
¶ Field is another model
Parameters: - model (aztk.core.models.Model) – Model object that field should be
- merge_strategy (ModelMergeStrategy) – When merging models how should the nested model be merged. Default: ModelMergeStrategy.merge
-
size
¶ Model Integer field
-
size_low_priority
¶ Model Integer field
-
vm_size
¶ Model String field
-
subnet_id
¶ Model String field
-
plugins
¶ Field that should be a list
Field that should be a list
-
user_configuration
¶ Field is another model
Parameters: - model (aztk.core.models.Model) – Model object that field should be
- merge_strategy (ModelMergeStrategy) – When merging models how should the nested model be merged. Default: ModelMergeStrategy.merge
-
scheduling_target
¶ Field that should be an enum
-
class
aztk.models.
ClusterState
[source]¶ Bases:
enum.Enum
An enumeration.
-
deleting
= 'deleting'¶
-
resizing
= 'resizing'¶
-
steady
= 'steady'¶
-
stopping_resize
= 'stopping'¶
-
-
class
aztk.models.
DockerConfiguration
(**kwargs)[source]¶ Bases:
aztk.core.models.model.Model
Configuration for connecting to private docker
Parameters: -
endpoint
¶ Model String field
-
username
¶ Model String field
-
password
¶ Model String field
-
-
class
aztk.models.
Enum
[source]¶ Bases:
object
Generic enumeration.
Derive from this class to define new enumerations.
Bases:
aztk.core.models.model.Model
Model String field
Model String field
Model String field
Model String field
-
class
aztk.models.
Model
(**kwargs)[source]¶ Bases:
object
Base class for all aztk models
To implement model wide validation implement __validate__ method
-
class
aztk.models.
NodeOutput
(id: str, output: Union[tempfile.SpooledTemporaryFile, str] = None, error: Exception = None)[source]¶ Bases:
object
-
class
aztk.models.
PluginConfiguration
(**kwargs)[source]¶ Bases:
aztk.core.models.model.Model
Plugin manifest that should be returned in the main.py of your plugin
- Args
- name: Name of the plugin. Used to reference the plugin runOn: Where the plugin should run execute: Path to the file to execute(This must match the target of one of the files) files: List of files to upload args: List of arguments to pass to the executing script env: Dict of environment variables to pass to the script
-
name
¶ Model String field
-
files
¶ Field that should be a list
-
execute
¶ Model String field
-
args
¶ Field that should be a list
-
env
¶ Field that should be a list
-
target
¶ Field that should be an enum
-
target_role
¶ Field that should be an enum
-
ports
¶ Field that should be a list
-
class
aztk.models.
PluginFile
(target: str = None, local_path: str = None)[source]¶ Bases:
aztk.core.models.model.Model
Reference to a file for a plugin.
-
target
¶ Model String field
-
local_path
¶ Model String field
-
-
class
aztk.models.
PluginPort
(**kwargs)[source]¶ Bases:
aztk.core.models.model.Model
Definition for a port that should be opened on node :param internal: Port on the node :param public: [Optional] Port available to the user. If none won’t open any port to the user :param name: [Optional] name to differentiate ports if you have multiple
-
internal
¶ Model Integer field
-
public
¶ Base class for all model fields
-
name
¶ Model Integer field
-
expose_publicly
¶
-
public_port
¶
-
-
class
aztk.models.
PluginTarget
[source]¶ Bases:
enum.Enum
Where this plugin should run
-
SparkContainer
= 'spark-container'¶
-
Host
= 'host'¶
-
-
class
aztk.models.
PluginTargetRole
[source]¶ Bases:
enum.Enum
An enumeration.
-
Master
= 'master'¶
-
Worker
= 'worker'¶
-
All
= 'all-nodes'¶
-
-
class
aztk.models.
PortForwardingSpecification
(**kwargs)[source]¶ Bases:
aztk.core.models.model.Model
-
remote_port
¶ Model Integer field
-
local_port
¶ Model Integer field
-
-
class
aztk.models.
SchedulingTarget
[source]¶ Bases:
enum.Enum
Target where task will get scheduled. For spark this is where the driver will live.
-
Master
= 'master'¶ Only master is allowed to run task
-
Any
= 'any'¶ Any node(Not recommended if using low pri) (Default)
-
-
class
aztk.models.
SecretsConfiguration
(**kwargs)[source]¶ Bases:
aztk.core.models.model.Model
-
service_principal
¶ Field is another model
Parameters: - model (aztk.core.models.Model) – Model object that field should be
- merge_strategy (ModelMergeStrategy) – When merging models how should the nested model be merged. Default: ModelMergeStrategy.merge
Field is another model
Parameters: - model (aztk.core.models.Model) – Model object that field should be
- merge_strategy (ModelMergeStrategy) – When merging models how should the nested model be merged. Default: ModelMergeStrategy.merge
-
docker
¶ Field is another model
Parameters: - model (aztk.core.models.Model) – Model object that field should be
- merge_strategy (ModelMergeStrategy) – When merging models how should the nested model be merged. Default: ModelMergeStrategy.merge
-
ssh_pub_key
¶ Model String field
-
ssh_priv_key
¶ Model String field
-
-
class
aztk.models.
ServicePrincipalConfiguration
(**kwargs)[source]¶ Bases:
aztk.core.models.model.Model
Container class for AAD authentication
-
tenant_id
¶ Model String field
-
client_id
¶ Model String field
-
credential
¶ Model String field
-
batch_account_resource_id
¶ Model String field
-
storage_account_resource_id
¶ Model String field
-
Bases:
aztk.core.models.model.Model
Container class for shared key authentication
Model String field
Model String field
Model String field
Model String field
Model String field
Model String field
-
class
aztk.models.
Software
[source]¶ Bases:
object
Enum with list of available softwares
-
spark
= 'spark'¶
-
-
class
aztk.models.
Task
(**kwargs)[source]¶ Bases:
aztk.core.models.model.Model
-
id
¶ Model String field
-
node_id
¶ Model String field
-
state
¶ Model String field
-
state_transition_time
¶ Model String field
-
command_line
¶ Model String field
-
exit_code
¶ Model Integer field
-
start_time
¶ Field that should be an datetime
-
end_time
¶ Field that should be an datetime
-
failure_info
¶ Model String field
-
-
class
aztk.models.
TaskState
[source]¶ Bases:
enum.Enum
An enumeration.
-
Running
= 'running'¶
-
Completed
= 'completed'¶
-
Failed
= 'failed'¶
-
Preparing
= 'preparing'¶
-
-
class
aztk.models.
TextPluginFile
(target: str, content: Union[str, _io.StringIO])[source]¶ Bases:
aztk.core.models.model.Model
Reference to a file for a plugin.
Args: target (str): Where should the file be uploaded relative to the plugin working dir content (str|io.StringIO): Content of the file. Can either be a string or a StringIO
-
target
¶ Model String field
-
-
class
aztk.models.
Toolkit
(**kwargs)[source]¶ Bases:
aztk.core.models.model.Model
Toolkit for a cluster. This will help pick the docker image needed
Parameters: - software (str) – Name of the toolkit(spark)
- version (str) – Version of the toolkit
- environment (str) – Which environment to use for this toolkit
- environment_version (str) – If there is multiple version for an environment you can specify which one
- docker_repo (str) – Optional docker repo
- docker_run_options (str) – Optional command-line options for docker run
-
software
¶ Model String field
-
version
¶ Model String field
-
environment
¶ Model String field
-
environment_version
¶ Model String field
-
docker_repo
¶ Model String field
-
docker_run_options
¶ Model String field
-
class
aztk.models.
Union
[source]¶ Bases:
typing.Final
Union type; Union[X, Y] means either X or Y.
To define a union, use e.g. Union[int, str]. Details:
The arguments must be types and there must be at least one.
None as an argument is a special case and is replaced by type(None).
Unions of unions are flattened, e.g.:
Union[Union[int, str], float] == Union[int, str, float]
Unions of a single argument vanish, e.g.:
Union[int] == int # The constructor actually returns int
Redundant arguments are skipped, e.g.:
Union[int, str, int] == Union[int, str]
When comparing unions, the argument order is ignored, e.g.:
Union[int, str] == Union[str, int]
When two arguments have a subclass relationship, the least derived argument is kept, e.g.:
class Employee: pass class Manager(Employee): pass Union[int, Employee, Manager] == Union[int, Employee] Union[Manager, int, Employee] == Union[int, Employee] Union[Employee, Manager] == Employee
Corollary: if Any is present it is the sole survivor, e.g.:
Union[int, Any] == Any
Similar for object:
Union[int, object] == object
To cut a tie: Union[object, Any] == Union[Any, object] == Any.
You cannot subclass or instantiate a union.
You cannot write Union[X][Y] (what would it mean?).
You can use Optional[X] as a shorthand for Union[X, None].