aztk.spark package¶
aztk.spark.client module¶
-
class
aztk.spark.client.
Client
(secrets_config: aztk.models.secrets_configuration.SecretsConfiguration)[source]¶ Bases:
aztk.client.Client
Aztk Spark Client This is the main entry point for using aztk for spark
Parameters: secrets_config (aztk.spark.models.models.SecretsConfiguration) – Configuration with all the needed credentials -
create_cluster
(cluster_conf: aztk.spark.models.models.ClusterConfiguration, wait: bool = False)[source]¶ Create a new aztk spark cluster
Parameters: - cluster_conf (aztk.spark.models.models.ClusterConfiguration) – Configuration for the the cluster to be created
- wait (bool) – If you should wait for the cluster to be ready before returning
Returns: aztk.spark.models.Cluster
-
submit
(cluster_id: str, application: aztk.spark.models.models.ApplicationConfiguration, remote: bool = False, wait: bool = False)[source]¶
-
create_user
(cluster_id: str, username: str, password: str = None, ssh_key: str = None) → str[source]¶
-
get_application_log
(cluster_id: str, application_name: str, tail=False, current_bytes: int = 0)[source]¶
-
cluster_run
(cluster_id: str, command: str, host=False, internal: bool = False, timeout=None)[source]¶
-
node_run
(cluster_id: str, node_id: str, command: str, host=False, internal: bool = False, timeout=None)[source]¶
-
cluster_copy
(cluster_id: str, source_path: str, destination_path: str, host: bool = False, internal: bool = False, timeout=None)[source]¶
-
cluster_download
(cluster_id: str, source_path: str, destination_path: str, host: bool = False, internal: bool = False, timeout=None)[source]¶
-