DataSync ******** Client ====== class DataSync.Client A low-level client representing AWS DataSync DataSync is an online data movement service that simplifies data migration and helps you quickly, easily, and securely transfer your file or object data to, from, and between Amazon Web Services storage services. This API interface reference includes documentation for using DataSync programmatically. For complete information, see the DataSync User Guide . import boto3 client = boto3.client('datasync') These are the available methods: * can_paginate * cancel_task_execution * close * create_agent * create_location_azure_blob * create_location_efs * create_location_fsx_lustre * create_location_fsx_ontap * create_location_fsx_open_zfs * create_location_fsx_windows * create_location_hdfs * create_location_nfs * create_location_object_storage * create_location_s3 * create_location_smb * create_task * delete_agent * delete_location * delete_task * describe_agent * describe_location_azure_blob * describe_location_efs * describe_location_fsx_lustre * describe_location_fsx_ontap * describe_location_fsx_open_zfs * describe_location_fsx_windows * describe_location_hdfs * describe_location_nfs * describe_location_object_storage * describe_location_s3 * describe_location_smb * describe_task * describe_task_execution * get_paginator * get_waiter * list_agents * list_locations * list_tags_for_resource * list_task_executions * list_tasks * start_task_execution * tag_resource * untag_resource * update_agent * update_location_azure_blob * update_location_efs * update_location_fsx_lustre * update_location_fsx_ontap * update_location_fsx_open_zfs * update_location_fsx_windows * update_location_hdfs * update_location_nfs * update_location_object_storage * update_location_s3 * update_location_smb * update_task * update_task_execution Paginators ========== Paginators are available on a client instance via the "get_paginator" method. For more detailed instructions and examples on the usage of paginators, see the paginators user guide. The available paginators are: * ListAgents * ListLocations * ListTagsForResource * ListTaskExecutions * ListTasks DataSync / Paginator / ListTagsForResource ListTagsForResource ******************* class DataSync.Paginator.ListTagsForResource paginator = client.get_paginator('list_tags_for_resource') paginate(**kwargs) Creates an iterator that will paginate through responses from "DataSync.Client.list_tags_for_resource()". See also: AWS API Documentation **Request Syntax** response_iterator = paginator.paginate( ResourceArn='string', PaginationConfig={ 'MaxItems': 123, 'PageSize': 123, 'StartingToken': 'string' } ) Parameters: * **ResourceArn** (*string*) -- **[REQUIRED]** Specifies the Amazon Resource Name (ARN) of the resource that you want tag information on. * **PaginationConfig** (*dict*) -- A dictionary that provides parameters to control pagination. * **MaxItems** *(integer) --* The total number of items to return. If the total number of items available is more than the value specified in max-items then a "NextToken" will be provided in the output that you can use to resume pagination. * **PageSize** *(integer) --* The size of each page. * **StartingToken** *(string) --* A token to specify where to start paginating. This is the "NextToken" from a previous response. Return type: dict Returns: **Response Syntax** { 'Tags': [ { 'Key': 'string', 'Value': 'string' }, ], } **Response Structure** * *(dict) --* ListTagsForResourceResponse * **Tags** *(list) --* An array of tags applied to the specified resource. * *(dict) --* A key-value pair representing a single tag that's been applied to an Amazon Web Services resource. * **Key** *(string) --* The key for an Amazon Web Services resource tag. * **Value** *(string) --* The value for an Amazon Web Services resource tag. DataSync / Paginator / ListLocations ListLocations ************* class DataSync.Paginator.ListLocations paginator = client.get_paginator('list_locations') paginate(**kwargs) Creates an iterator that will paginate through responses from "DataSync.Client.list_locations()". See also: AWS API Documentation **Request Syntax** response_iterator = paginator.paginate( Filters=[ { 'Name': 'LocationUri'|'LocationType'|'CreationTime', 'Values': [ 'string', ], 'Operator': 'Equals'|'NotEquals'|'In'|'LessThanOrEqual'|'LessThan'|'GreaterThanOrEqual'|'GreaterThan'|'Contains'|'NotContains'|'BeginsWith' }, ], PaginationConfig={ 'MaxItems': 123, 'PageSize': 123, 'StartingToken': 'string' } ) Parameters: * **Filters** (*list*) -- You can use API filters to narrow down the list of resources returned by "ListLocations". For example, to retrieve all tasks on a specific source location, you can use "ListLocations" with filter name "LocationType S3" and "Operator Equals". * *(dict) --* Narrow down the list of resources returned by "ListLocations". For example, to see all your Amazon S3 locations, create a filter using ""Name": "LocationType"", ""Operator": "Equals"", and ""Values": "S3"". For more information, see filtering resources. * **Name** *(string) --* **[REQUIRED]** The name of the filter being used. Each API call supports a list of filters that are available for it (for example, "LocationType" for "ListLocations"). * **Values** *(list) --* **[REQUIRED]** The values that you want to filter for. For example, you might want to display only Amazon S3 locations. * *(string) --* * **Operator** *(string) --* **[REQUIRED]** The operator that is used to compare filter values (for example, "Equals" or "Contains"). * **PaginationConfig** (*dict*) -- A dictionary that provides parameters to control pagination. * **MaxItems** *(integer) --* The total number of items to return. If the total number of items available is more than the value specified in max-items then a "NextToken" will be provided in the output that you can use to resume pagination. * **PageSize** *(integer) --* The size of each page. * **StartingToken** *(string) --* A token to specify where to start paginating. This is the "NextToken" from a previous response. Return type: dict Returns: **Response Syntax** { 'Locations': [ { 'LocationArn': 'string', 'LocationUri': 'string' }, ], } **Response Structure** * *(dict) --* ListLocationsResponse * **Locations** *(list) --* An array that contains a list of locations. * *(dict) --* Represents a single entry in a list of locations. "LocationListEntry" returns an array that contains a list of locations when the ListLocations operation is called. * **LocationArn** *(string) --* The Amazon Resource Name (ARN) of the location. For Network File System (NFS) or Amazon EFS, the location is the export path. For Amazon S3, the location is the prefix path that you want to mount and use as the root of the location. * **LocationUri** *(string) --* Represents a list of URIs of a location. "LocationUri" returns an array that contains a list of locations when the ListLocations operation is called. Format: "TYPE://GLOBAL_ID/SUBDIR". TYPE designates the type of location (for example, "nfs" or "s3"). GLOBAL_ID is the globally unique identifier of the resource that backs the location. An example for EFS is "us-east-2.fs-abcd1234". An example for Amazon S3 is the bucket name, such as "myBucket". An example for NFS is a valid IPv4 or IPv6 address or a hostname that is compliant with DNS. SUBDIR is a valid file system path, delimited by forward slashes as is the >>*< object_storage_certificates.pem" To use this parameter, configure "ServerProtocol" to "HTTPS". * **CmkSecretConfig** (*dict*) -- Specifies configuration information for a DataSync-managed secret, which includes the "SecretKey" that DataSync uses to access a specific object storage location, with a customer- managed KMS key. When you include this paramater as part of a "CreateLocationObjectStorage" request, you provide only the KMS key ARN. DataSync uses this KMS key together with the value you specify for the "SecretKey" parameter to create a DataSync-managed secret to store the location access credentials. Make sure the DataSync has permission to access the KMS key that you specify. Note: You can use either "CmkSecretConfig" (with "SecretKey") or "CustomSecretConfig" (without "SecretKey") to provide credentials for a "CreateLocationObjectStorage" request. Do not provide both parameters for the same request. * **SecretArn** *(string) --* Specifies the ARN for the DataSync-managed Secrets Manager secret that that is used to access a specific storage location. This property is generated by DataSync and is read-only. DataSync encrypts this secret with the KMS key that you specify for "KmsKeyArn". * **KmsKeyArn** *(string) --* Specifies the ARN for the customer-managed KMS key that DataSync uses to encrypt the DataSync-managed secret stored for "SecretArn". DataSync provides this key to Secrets Manager. * **CustomSecretConfig** (*dict*) -- Specifies configuration information for a customer-managed Secrets Manager secret where the secret key for a specific object storage location is stored in plain text. This configuration includes the secret ARN, and the ARN for an IAM role that provides access to the secret. Note: You can use either "CmkSecretConfig" (with "SecretKey") or "CustomSecretConfig" (without "SecretKey") to provide credentials for a "CreateLocationObjectStorage" request. Do not provide both parameters for the same request. * **SecretArn** *(string) --* Specifies the ARN for an Secrets Manager secret. * **SecretAccessRoleArn** *(string) --* Specifies the ARN for the Identity and Access Management role that DataSync uses to access the secret specified for "SecretArn". Return type: dict Returns: **Response Syntax** { 'LocationArn': 'string' } **Response Structure** * *(dict) --* CreateLocationObjectStorageResponse * **LocationArn** *(string) --* Specifies the ARN of the object storage system location that you create. **Exceptions** * "DataSync.Client.exceptions.InvalidRequestException" * "DataSync.Client.exceptions.InternalException" DataSync / Client / describe_task describe_task ************* DataSync.Client.describe_task(**kwargs) Provides information about a *task*, which defines where and how DataSync transfers your data. See also: AWS API Documentation **Request Syntax** response = client.describe_task( TaskArn='string' ) Parameters: **TaskArn** (*string*) -- **[REQUIRED]** Specifies the Amazon Resource Name (ARN) of the transfer task that you want information about. Return type: dict Returns: **Response Syntax** { 'TaskArn': 'string', 'Status': 'AVAILABLE'|'CREATING'|'QUEUED'|'RUNNING'|'UNAVAILABLE', 'Name': 'string', 'CurrentTaskExecutionArn': 'string', 'SourceLocationArn': 'string', 'DestinationLocationArn': 'string', 'CloudWatchLogGroupArn': 'string', 'SourceNetworkInterfaceArns': [ 'string', ], 'DestinationNetworkInterfaceArns': [ 'string', ], 'Options': { 'VerifyMode': 'POINT_IN_TIME_CONSISTENT'|'ONLY_FILES_TRANSFERRED'|'NONE', 'OverwriteMode': 'ALWAYS'|'NEVER', 'Atime': 'NONE'|'BEST_EFFORT', 'Mtime': 'NONE'|'PRESERVE', 'Uid': 'NONE'|'INT_VALUE'|'NAME'|'BOTH', 'Gid': 'NONE'|'INT_VALUE'|'NAME'|'BOTH', 'PreserveDeletedFiles': 'PRESERVE'|'REMOVE', 'PreserveDevices': 'NONE'|'PRESERVE', 'PosixPermissions': 'NONE'|'PRESERVE', 'BytesPerSecond': 123, 'TaskQueueing': 'ENABLED'|'DISABLED', 'LogLevel': 'OFF'|'BASIC'|'TRANSFER', 'TransferMode': 'CHANGED'|'ALL', 'SecurityDescriptorCopyFlags': 'NONE'|'OWNER_DACL'|'OWNER_DACL_SACL', 'ObjectTags': 'PRESERVE'|'NONE' }, 'Excludes': [ { 'FilterType': 'SIMPLE_PATTERN', 'Value': 'string' }, ], 'Schedule': { 'ScheduleExpression': 'string', 'Status': 'ENABLED'|'DISABLED' }, 'ErrorCode': 'string', 'ErrorDetail': 'string', 'CreationTime': datetime(2015, 1, 1), 'Includes': [ { 'FilterType': 'SIMPLE_PATTERN', 'Value': 'string' }, ], 'ManifestConfig': { 'Action': 'TRANSFER', 'Format': 'CSV', 'Source': { 'S3': { 'ManifestObjectPath': 'string', 'BucketAccessRoleArn': 'string', 'S3BucketArn': 'string', 'ManifestObjectVersionId': 'string' } } }, 'TaskReportConfig': { 'Destination': { 'S3': { 'Subdirectory': 'string', 'S3BucketArn': 'string', 'BucketAccessRoleArn': 'string' } }, 'OutputType': 'SUMMARY_ONLY'|'STANDARD', 'ReportLevel': 'ERRORS_ONLY'|'SUCCESSES_AND_ERRORS', 'ObjectVersionIds': 'INCLUDE'|'NONE', 'Overrides': { 'Transferred': { 'ReportLevel': 'ERRORS_ONLY'|'SUCCESSES_AND_ERRORS' }, 'Verified': { 'ReportLevel': 'ERRORS_ONLY'|'SUCCESSES_AND_ERRORS' }, 'Deleted': { 'ReportLevel': 'ERRORS_ONLY'|'SUCCESSES_AND_ERRORS' }, 'Skipped': { 'ReportLevel': 'ERRORS_ONLY'|'SUCCESSES_AND_ERRORS' } } }, 'ScheduleDetails': { 'StatusUpdateTime': datetime(2015, 1, 1), 'DisabledReason': 'string', 'DisabledBy': 'USER'|'SERVICE' }, 'TaskMode': 'BASIC'|'ENHANCED' } **Response Structure** * *(dict) --* DescribeTaskResponse * **TaskArn** *(string) --* The ARN of your task. * **Status** *(string) --* The status of your task. For information about what each status means, see Task statuses. * **Name** *(string) --* The name of your task. * **CurrentTaskExecutionArn** *(string) --* The ARN of the most recent task execution. * **SourceLocationArn** *(string) --* The ARN of your transfer's source location. * **DestinationLocationArn** *(string) --* The ARN of your transfer's destination location. * **CloudWatchLogGroupArn** *(string) --* The Amazon Resource Name (ARN) of an Amazon CloudWatch log group for monitoring your task. For more information, see Monitoring data transfers with CloudWatch Logs. * **SourceNetworkInterfaceArns** *(list) --* The ARNs of the network interfaces that DataSync created for your source location. * *(string) --* * **DestinationNetworkInterfaceArns** *(list) --* The ARNs of the network interfaces that DataSync created for your destination location. * *(string) --* * **Options** *(dict) --* The task's settings. For example, what file metadata gets preserved, how data integrity gets verified at the end of your transfer, bandwidth limits, among other options. * **VerifyMode** *(string) --* Specifies if and how DataSync checks the integrity of your data at the end of your transfer. * "ONLY_FILES_TRANSFERRED" (recommended) - DataSync calculates the checksum of transferred data (including metadata) at the source location. At the end of the transfer, DataSync then compares this checksum to the checksum calculated on that data at the destination. Note: This is the default option for Enhanced mode tasks. We recommend this option when transferring to S3 Glacier Flexible Retrieval or S3 Glacier Deep Archive storage classes. For more information, see Storage class considerations with Amazon S3 locations. * "POINT_IN_TIME_CONSISTENT" - At the end of the transfer, DataSync checks the entire source and destination to verify that both locations are fully synchronized. Note: The is the default option for Basic mode tasks and isn't currently supported with Enhanced mode tasks. If you use a manifest, DataSync only scans and verifies what's listed in the manifest. You can't use this option when transferring to S3 Glacier Flexible Retrieval or S3 Glacier Deep Archive storage classes. For more information, see Storage class considerations with Amazon S3 locations. * "NONE" - DataSync performs data integrity checks only during your transfer. Unlike other options, there's no additional verification at the end of your transfer. * **OverwriteMode** *(string) --* Specifies whether DataSync should modify or preserve data at the destination location. * "ALWAYS" (default) - DataSync modifies data in the destination location when source data (including metadata) has changed. If DataSync overwrites objects, you might incur additional charges for certain Amazon S3 storage classes (for example, for retrieval or early deletion). For more information, see Storage class considerations with Amazon S3 transfers. * "NEVER" - DataSync doesn't overwrite data in the destination location even if the source data has changed. You can use this option to protect against overwriting changes made to files or objects in the destination. * **Atime** *(string) --* Specifies whether to preserve metadata indicating the last time a file was read or written to. Note: The behavior of "Atime" isn't fully standard across platforms, so DataSync can only do this on a best-effort basis. * "BEST_EFFORT" (default) - DataSync attempts to preserve the original "Atime" attribute on all source files (that is, the version before the "PREPARING" steps of the task execution). This option is recommended. * "NONE" - Ignores "Atime". Note: If "Atime" is set to "BEST_EFFORT", "Mtime" must be set to "PRESERVE".If "Atime" is set to "NONE", "Mtime" must also be "NONE". * **Mtime** *(string) --* Specifies whether to preserve metadata indicating the last time that a file was written to before the "PREPARING" step of your task execution. This option is required when you need to run the a task more than once. * "PRESERVE" (default) - Preserves original "Mtime", which is recommended. * "NONE" - Ignores "Mtime". Note: If "Mtime" is set to "PRESERVE", "Atime" must be set to "BEST_EFFORT".If "Mtime" is set to "NONE", "Atime" must also be set to "NONE". * **Uid** *(string) --* Specifies the POSIX user ID (UID) of the file's owner. * "INT_VALUE" (default) - Preserves the integer value of UID and group ID (GID), which is recommended. * "NONE" - Ignores UID and GID. For more information, see Metadata copied by DataSync. * **Gid** *(string) --* Specifies the POSIX group ID (GID) of the file's owners. * "INT_VALUE" (default) - Preserves the integer value of user ID (UID) and GID, which is recommended. * "NONE" - Ignores UID and GID. For more information, see Understanding how DataSync handles file and object metadata. * **PreserveDeletedFiles** *(string) --* Specifies whether files in the destination location that don't exist in the source should be preserved. This option can affect your Amazon S3 storage cost. If your task deletes objects, you might incur minimum storage duration charges for certain storage classes. For detailed information, see Considerations when working with Amazon S3 storage classes in DataSync. * "PRESERVE" (default) - Ignores such destination files, which is recommended. * "REMOVE" - Deletes destination files that aren’t present in the source. Note: If you set this parameter to "REMOVE", you can't set "TransferMode" to "ALL". When you transfer all data, DataSync doesn't scan your destination location and doesn't know what to delete. * **PreserveDevices** *(string) --* Specifies whether DataSync should preserve the metadata of block and character devices in the source location and recreate the files with that device name and metadata on the destination. DataSync copies only the name and metadata of such devices. Note: DataSync can't copy the actual contents of these devices because they're nonterminal and don't return an end-of- file (EOF) marker. * "NONE" (default) - Ignores special devices (recommended). * "PRESERVE" - Preserves character and block device metadata. This option currently isn't supported for Amazon EFS. * **PosixPermissions** *(string) --* Specifies which users or groups can access a file for a specific purpose such as reading, writing, or execution of the file. For more information, see Understanding how DataSync handles file and object metadata. * "PRESERVE" (default) - Preserves POSIX-style permissions, which is recommended. * "NONE" - Ignores POSIX-style permissions. Note: DataSync can preserve extant permissions of a source location. * **BytesPerSecond** *(integer) --* Limits the bandwidth used by a DataSync task. For example, if you want DataSync to use a maximum of 1 MB, set this value to "1048576" ( "=1024*1024"). Note: Not applicable to Enhanced mode tasks. * **TaskQueueing** *(string) --* Specifies whether your transfer tasks should be put into a queue during certain scenarios when running multiple tasks. This is "ENABLED" by default. * **LogLevel** *(string) --* Specifies the type of logs that DataSync publishes to a Amazon CloudWatch Logs log group. To specify the log group, see CloudWatchLogGroupArn. * "BASIC" - Publishes logs with only basic information (such as transfer errors). * "TRANSFER" - Publishes logs for all files or objects that your DataSync task transfers and performs data- integrity checks on. * "OFF" - No logs are published. * **TransferMode** *(string) --* Specifies whether DataSync transfers only the data (including metadata) that differs between locations following an initial copy or transfers all data every time you run the task. If you're planning on recurring transfers, you might only want to transfer what's changed since your previous task execution. * "CHANGED" (default) - After your initial full transfer, DataSync copies only the data and metadata that differs between the source and destination location. * "ALL" - DataSync copies everything in the source to the destination without comparing differences between the locations. * **SecurityDescriptorCopyFlags** *(string) --* Specifies which components of the SMB security descriptor are copied from source to destination objects. This value is only used for transfers between SMB and Amazon FSx for Windows File Server locations or between two FSx for Windows File Server locations. For more information, see Understanding how DataSync handles file and object metadata. * "OWNER_DACL" (default) - For each copied object, DataSync copies the following metadata: * The object owner. * NTFS discretionary access control lists (DACLs), which determine whether to grant access to an object. DataSync won't copy NTFS system access control lists (SACLs) with this option. * "OWNER_DACL_SACL" - For each copied object, DataSync copies the following metadata: * The object owner. * NTFS discretionary access control lists (DACLs), which determine whether to grant access to an object. * SACLs, which are used by administrators to log attempts to access a secured object. Copying SACLs requires granting additional permissions to the Windows user that DataSync uses to access your SMB location. For information about choosing a user with the right permissions, see required permissions for SMB, FSx for Windows File Server, or FSx for ONTAP (depending on the type of location in your transfer). * "NONE" - None of the SMB security descriptor components are copied. Destination objects are owned by the user that was provided for accessing the destination location. DACLs and SACLs are set based on the destination server’s configuration. * **ObjectTags** *(string) --* Specifies whether you want DataSync to "PRESERVE" object tags (default behavior) when transferring between object storage systems. If you want your DataSync task to ignore object tags, specify the "NONE" value. * **Excludes** *(list) --* The exclude filters that define the files, objects, and folders in your source location that you don't want DataSync to transfer. For more information and examples, see Specifying what DataSync transfers by using filters. * *(dict) --* Specifies which files, folders, and objects to include or exclude when transferring files from source to destination. * **FilterType** *(string) --* The type of filter rule to apply. DataSync only supports the SIMPLE_PATTERN rule type. * **Value** *(string) --* A single filter string that consists of the patterns to include or exclude. The patterns are delimited by "|" (that is, a pipe), for example: "/folder1|/folder2" * **Schedule** *(dict) --* The schedule for when you want your task to run. For more information, see Scheduling your task. * **ScheduleExpression** *(string) --* Specifies your task schedule by using a cron or rate expression. Use cron expressions for task schedules that run on a specific time and day. For example, the following cron expression creates a task schedule that runs at 8 AM on the first Wednesday of every month: "cron(0 8 * * 3#1)" Use rate expressions for task schedules that run on a regular interval. For example, the following rate expression creates a task schedule that runs every 12 hours: "rate(12 hours)" For information about cron and rate expression syntax, see the Amazon EventBridge User Guide. * **Status** *(string) --* Specifies whether to enable or disable your task schedule. Your schedule is enabled by default, but there can be situations where you need to disable it. For example, you might need to pause a recurring transfer to fix an issue with your task or perform maintenance on your storage system. DataSync might disable your schedule automatically if your task fails repeatedly with the same error. For more information, see TaskScheduleDetails. * **ErrorCode** *(string) --* If there's an issue with your task, you can use the error code to help you troubleshoot the problem. For more information, see Troubleshooting issues with DataSync transfers. * **ErrorDetail** *(string) --* If there's an issue with your task, you can use the error details to help you troubleshoot the problem. For more information, see Troubleshooting issues with DataSync transfers. * **CreationTime** *(datetime) --* The time that the task was created. * **Includes** *(list) --* The include filters that define the files, objects, and folders in your source location that you want DataSync to transfer. For more information and examples, see Specifying what DataSync transfers by using filters. * *(dict) --* Specifies which files, folders, and objects to include or exclude when transferring files from source to destination. * **FilterType** *(string) --* The type of filter rule to apply. DataSync only supports the SIMPLE_PATTERN rule type. * **Value** *(string) --* A single filter string that consists of the patterns to include or exclude. The patterns are delimited by "|" (that is, a pipe), for example: "/folder1|/folder2" * **ManifestConfig** *(dict) --* The configuration of the manifest that lists the files or objects that you want DataSync to transfer. For more information, see Specifying what DataSync transfers by using a manifest. * **Action** *(string) --* Specifies what DataSync uses the manifest for. * **Format** *(string) --* Specifies the file format of your manifest. For more information, see Creating a manifest. * **Source** *(dict) --* Specifies the manifest that you want DataSync to use and where it's hosted. Note: You must specify this parameter if you're configuring a new manifest on or after February 7, 2024.If you don't, you'll get a 400 status code and "ValidationException" error stating that you're missing the IAM role for DataSync to access the S3 bucket where you're hosting your manifest. For more information, see Providing DataSync access to your manifest. * **S3** *(dict) --* Specifies the S3 bucket where you're hosting your manifest. * **ManifestObjectPath** *(string) --* Specifies the Amazon S3 object key of your manifest. This can include a prefix (for example, "prefix/my- manifest.csv"). * **BucketAccessRoleArn** *(string) --* Specifies the Identity and Access Management (IAM) role that allows DataSync to access your manifest. For more information, see Providing DataSync access to your manifest. * **S3BucketArn** *(string) --* Specifies the Amazon Resource Name (ARN) of the S3 bucket where you're hosting your manifest. * **ManifestObjectVersionId** *(string) --* Specifies the object version ID of the manifest that you want DataSync to use. If you don't set this, DataSync uses the latest version of the object. * **TaskReportConfig** *(dict) --* The configuration of your task report, which provides detailed information about your DataSync transfer. For more information, see Monitoring your DataSync transfers with task reports. * **Destination** *(dict) --* Specifies the Amazon S3 bucket where DataSync uploads your task report. For more information, see Task reports. * **S3** *(dict) --* Specifies the Amazon S3 bucket where DataSync uploads your task report. * **Subdirectory** *(string) --* Specifies a bucket prefix for your report. * **S3BucketArn** *(string) --* Specifies the ARN of the S3 bucket where DataSync uploads your report. * **BucketAccessRoleArn** *(string) --* Specifies the Amazon Resource Name (ARN) of the IAM policy that allows DataSync to upload a task report to your S3 bucket. For more information, see Allowing DataSync to upload a task report to an Amazon S3 bucket. * **OutputType** *(string) --* Specifies the type of task report that you want: * "SUMMARY_ONLY": Provides necessary details about your task, including the number of files, objects, and directories transferred and transfer duration. * "STANDARD": Provides complete details about your task, including a full list of files, objects, and directories that were transferred, skipped, verified, and more. * **ReportLevel** *(string) --* Specifies whether you want your task report to include only what went wrong with your transfer or a list of what succeeded and didn't. * "ERRORS_ONLY": A report shows what DataSync was unable to transfer, skip, verify, and delete. * "SUCCESSES_AND_ERRORS": A report shows what DataSync was able and unable to transfer, skip, verify, and delete. * **ObjectVersionIds** *(string) --* Specifies whether your task report includes the new version of each object transferred into an S3 bucket. This only applies if you enable versioning on your bucket. Keep in mind that setting this to "INCLUDE" can increase the duration of your task execution. * **Overrides** *(dict) --* Customizes the reporting level for aspects of your task report. For example, your report might generally only include errors, but you could specify that you want a list of successes and errors just for the files that DataSync attempted to delete in your destination location. * **Transferred** *(dict) --* Specifies the level of reporting for the files, objects, and directories that DataSync attempted to transfer. * **ReportLevel** *(string) --* Specifies whether your task report includes errors only or successes and errors. For example, your report might mostly include only what didn't go well in your transfer ( "ERRORS_ONLY"). At the same time, you want to verify that your task filter is working correctly. In this situation, you can get a list of what files DataSync successfully skipped and if something transferred that you didn't to transfer ( "SUCCESSES_AND_ERRORS"). * **Verified** *(dict) --* Specifies the level of reporting for the files, objects, and directories that DataSync attempted to verify at the end of your transfer. * **ReportLevel** *(string) --* Specifies whether your task report includes errors only or successes and errors. For example, your report might mostly include only what didn't go well in your transfer ( "ERRORS_ONLY"). At the same time, you want to verify that your task filter is working correctly. In this situation, you can get a list of what files DataSync successfully skipped and if something transferred that you didn't to transfer ( "SUCCESSES_AND_ERRORS"). * **Deleted** *(dict) --* Specifies the level of reporting for the files, objects, and directories that DataSync attempted to delete in your destination location. This only applies if you configure your task to delete data in the destination that isn't in the source. * **ReportLevel** *(string) --* Specifies whether your task report includes errors only or successes and errors. For example, your report might mostly include only what didn't go well in your transfer ( "ERRORS_ONLY"). At the same time, you want to verify that your task filter is working correctly. In this situation, you can get a list of what files DataSync successfully skipped and if something transferred that you didn't to transfer ( "SUCCESSES_AND_ERRORS"). * **Skipped** *(dict) --* Specifies the level of reporting for the files, objects, and directories that DataSync attempted to skip during your transfer. * **ReportLevel** *(string) --* Specifies whether your task report includes errors only or successes and errors. For example, your report might mostly include only what didn't go well in your transfer ( "ERRORS_ONLY"). At the same time, you want to verify that your task filter is working correctly. In this situation, you can get a list of what files DataSync successfully skipped and if something transferred that you didn't to transfer ( "SUCCESSES_AND_ERRORS"). * **ScheduleDetails** *(dict) --* The details about your task schedule. * **StatusUpdateTime** *(datetime) --* Indicates the last time the status of your task schedule changed. For example, if DataSync automatically disables your schedule because of a repeated error, you can see when the schedule was disabled. * **DisabledReason** *(string) --* Provides a reason if the task schedule is disabled. If your schedule is disabled by "USER", you see a "Manually disabled by user." message. If your schedule is disabled by "SERVICE", you see an error message to help you understand why the task keeps failing. For information on resolving DataSync errors, see Troubleshooting issues with DataSync transfers. * **DisabledBy** *(string) --* Indicates how your task schedule was disabled. * "USER" - Your schedule was manually disabled by using the UpdateTask operation or DataSync console. * "SERVICE" - Your schedule was automatically disabled by DataSync because the task failed repeatedly with the same error. * **TaskMode** *(string) --* The task mode that you're using. For more information, see Choosing a task mode for your data transfer. **Exceptions** * "DataSync.Client.exceptions.InvalidRequestException" * "DataSync.Client.exceptions.InternalException" DataSync / Client / list_locations list_locations ************** DataSync.Client.list_locations(**kwargs) Returns a list of source and destination locations. If you have more locations than are returned in a response (that is, the response returns only a truncated list of your agents), the response contains a token that you can specify in your next request to fetch the next page of locations. See also: AWS API Documentation **Request Syntax** response = client.list_locations( MaxResults=123, NextToken='string', Filters=[ { 'Name': 'LocationUri'|'LocationType'|'CreationTime', 'Values': [ 'string', ], 'Operator': 'Equals'|'NotEquals'|'In'|'LessThanOrEqual'|'LessThan'|'GreaterThanOrEqual'|'GreaterThan'|'Contains'|'NotContains'|'BeginsWith' }, ] ) Parameters: * **MaxResults** (*integer*) -- The maximum number of locations to return. * **NextToken** (*string*) -- An opaque string that indicates the position at which to begin the next list of locations. * **Filters** (*list*) -- You can use API filters to narrow down the list of resources returned by "ListLocations". For example, to retrieve all tasks on a specific source location, you can use "ListLocations" with filter name "LocationType S3" and "Operator Equals". * *(dict) --* Narrow down the list of resources returned by "ListLocations". For example, to see all your Amazon S3 locations, create a filter using ""Name": "LocationType"", ""Operator": "Equals"", and ""Values": "S3"". For more information, see filtering resources. * **Name** *(string) --* **[REQUIRED]** The name of the filter being used. Each API call supports a list of filters that are available for it (for example, "LocationType" for "ListLocations"). * **Values** *(list) --* **[REQUIRED]** The values that you want to filter for. For example, you might want to display only Amazon S3 locations. * *(string) --* * **Operator** *(string) --* **[REQUIRED]** The operator that is used to compare filter values (for example, "Equals" or "Contains"). Return type: dict Returns: **Response Syntax** { 'Locations': [ { 'LocationArn': 'string', 'LocationUri': 'string' }, ], 'NextToken': 'string' } **Response Structure** * *(dict) --* ListLocationsResponse * **Locations** *(list) --* An array that contains a list of locations. * *(dict) --* Represents a single entry in a list of locations. "LocationListEntry" returns an array that contains a list of locations when the ListLocations operation is called. * **LocationArn** *(string) --* The Amazon Resource Name (ARN) of the location. For Network File System (NFS) or Amazon EFS, the location is the export path. For Amazon S3, the location is the prefix path that you want to mount and use as the root of the location. * **LocationUri** *(string) --* Represents a list of URIs of a location. "LocationUri" returns an array that contains a list of locations when the ListLocations operation is called. Format: "TYPE://GLOBAL_ID/SUBDIR". TYPE designates the type of location (for example, "nfs" or "s3"). GLOBAL_ID is the globally unique identifier of the resource that backs the location. An example for EFS is "us-east-2.fs-abcd1234". An example for Amazon S3 is the bucket name, such as "myBucket". An example for NFS is a valid IPv4 or IPv6 address or a hostname that is compliant with DNS. SUBDIR is a valid file system path, delimited by forward slashes as is the >>*< object_storage_certificates.pem" To use this parameter, configure "ServerProtocol" to "HTTPS". Updating this parameter doesn't interfere with tasks that you have in progress. * **CmkSecretConfig** (*dict*) -- Specifies configuration information for a DataSync-managed secret, such as an authentication token or set of credentials that DataSync uses to access a specific transfer location, and a customer-managed KMS key. * **SecretArn** *(string) --* Specifies the ARN for the DataSync-managed Secrets Manager secret that that is used to access a specific storage location. This property is generated by DataSync and is read-only. DataSync encrypts this secret with the KMS key that you specify for "KmsKeyArn". * **KmsKeyArn** *(string) --* Specifies the ARN for the customer-managed KMS key that DataSync uses to encrypt the DataSync-managed secret stored for "SecretArn". DataSync provides this key to Secrets Manager. * **CustomSecretConfig** (*dict*) -- Specifies configuration information for a customer-managed secret, such as an authentication token or set of credentials that DataSync uses to access a specific transfer location, and a customer-managed KMS key. * **SecretArn** *(string) --* Specifies the ARN for an Secrets Manager secret. * **SecretAccessRoleArn** *(string) --* Specifies the ARN for the Identity and Access Management role that DataSync uses to access the secret specified for "SecretArn". Return type: dict Returns: **Response Syntax** {} **Response Structure** * *(dict) --* **Exceptions** * "DataSync.Client.exceptions.InvalidRequestException" * "DataSync.Client.exceptions.InternalException" DataSync / Client / update_agent update_agent ************ DataSync.Client.update_agent(**kwargs) Updates the name of an DataSync agent. See also: AWS API Documentation **Request Syntax** response = client.update_agent( AgentArn='string', Name='string' ) Parameters: * **AgentArn** (*string*) -- **[REQUIRED]** The Amazon Resource Name (ARN) of the agent to update. * **Name** (*string*) -- The name that you want to use to configure the agent. Return type: dict Returns: **Response Syntax** {} **Response Structure** * *(dict) --* **Exceptions** * "DataSync.Client.exceptions.InvalidRequestException" * "DataSync.Client.exceptions.InternalException" DataSync / Client / create_location_hdfs create_location_hdfs ******************** DataSync.Client.create_location_hdfs(**kwargs) Creates a transfer *location* for a Hadoop Distributed File System (HDFS). DataSync can use this location as a source or destination for transferring data. Before you begin, make sure that you understand how DataSync accesses HDFS clusters. See also: AWS API Documentation **Request Syntax** response = client.create_location_hdfs( Subdirectory='string', NameNodes=[ { 'Hostname': 'string', 'Port': 123 }, ], BlockSize=123, ReplicationFactor=123, KmsKeyProviderUri='string', QopConfiguration={ 'RpcProtection': 'DISABLED'|'AUTHENTICATION'|'INTEGRITY'|'PRIVACY', 'DataTransferProtection': 'DISABLED'|'AUTHENTICATION'|'INTEGRITY'|'PRIVACY' }, AuthenticationType='SIMPLE'|'KERBEROS', SimpleUser='string', KerberosPrincipal='string', KerberosKeytab=b'bytes', KerberosKrb5Conf=b'bytes', AgentArns=[ 'string', ], Tags=[ { 'Key': 'string', 'Value': 'string' }, ] ) Parameters: * **Subdirectory** (*string*) -- A subdirectory in the HDFS cluster. This subdirectory is used to read data from or write data to the HDFS cluster. If the subdirectory isn't specified, it will default to "/". * **NameNodes** (*list*) -- **[REQUIRED]** The NameNode that manages the HDFS namespace. The NameNode performs operations such as opening, closing, and renaming files and directories. The NameNode contains the information to map blocks of data to the DataNodes. You can use only one NameNode. * *(dict) --* The NameNode of the Hadoop Distributed File System (HDFS). The NameNode manages the file system's namespace. The NameNode performs operations such as opening, closing, and renaming files and directories. The NameNode contains the information to map blocks of data to the DataNodes. * **Hostname** *(string) --* **[REQUIRED]** The hostname of the NameNode in the HDFS cluster. This value is the IP address or Domain Name Service (DNS) name of the NameNode. An agent that's installed on-premises uses this hostname to communicate with the NameNode in the network. * **Port** *(integer) --* **[REQUIRED]** The port that the NameNode uses to listen to client requests. * **BlockSize** (*integer*) -- The size of data blocks to write into the HDFS cluster. The block size must be a multiple of 512 bytes. The default block size is 128 mebibytes (MiB). * **ReplicationFactor** (*integer*) -- The number of DataNodes to replicate the data to when writing to the HDFS cluster. By default, data is replicated to three DataNodes. * **KmsKeyProviderUri** (*string*) -- The URI of the HDFS cluster's Key Management Server (KMS). * **QopConfiguration** (*dict*) -- The Quality of Protection (QOP) configuration specifies the Remote Procedure Call (RPC) and data transfer protection settings configured on the Hadoop Distributed File System (HDFS) cluster. If "QopConfiguration" isn't specified, "RpcProtection" and "DataTransferProtection" default to "PRIVACY". If you set "RpcProtection" or "DataTransferProtection", the other parameter assumes the same value. * **RpcProtection** *(string) --* The RPC protection setting configured on the HDFS cluster. This setting corresponds to your "hadoop.rpc.protection" setting in your "core-site.xml" file on your Hadoop cluster. * **DataTransferProtection** *(string) --* The data transfer protection setting configured on the HDFS cluster. This setting corresponds to your "dfs.data.transfer.protection" setting in the "hdfs- site.xml" file on your Hadoop cluster. * **AuthenticationType** (*string*) -- **[REQUIRED]** The type of authentication used to determine the identity of the user. * **SimpleUser** (*string*) -- The user name used to identify the client on the host operating system. Note: If "SIMPLE" is specified for "AuthenticationType", this parameter is required. * **KerberosPrincipal** (*string*) -- The Kerberos principal with access to the files and folders on the HDFS cluster. Note: If "KERBEROS" is specified for "AuthenticationType", this parameter is required. * **KerberosKeytab** (*bytes*) -- The Kerberos key table (keytab) that contains mappings between the defined Kerberos principal and the encrypted keys. You can load the keytab from a file by providing the file's address. Note: If "KERBEROS" is specified for "AuthenticationType", this parameter is required. * **KerberosKrb5Conf** (*bytes*) -- The "krb5.conf" file that contains the Kerberos configuration information. You can load the "krb5.conf" file by providing the file's address. If you're using the CLI, it performs the base64 encoding for you. Otherwise, provide the base64-encoded text. Note: If "KERBEROS" is specified for "AuthenticationType", this parameter is required. * **AgentArns** (*list*) -- **[REQUIRED]** The Amazon Resource Names (ARNs) of the DataSync agents that can connect to your HDFS cluster. * *(string) --* * **Tags** (*list*) -- The key-value pair that represents the tag that you want to add to the location. The value can be an empty string. We recommend using tags to name your resources. * *(dict) --* A key-value pair representing a single tag that's been applied to an Amazon Web Services resource. * **Key** *(string) --* **[REQUIRED]** The key for an Amazon Web Services resource tag. * **Value** *(string) --* The value for an Amazon Web Services resource tag. Return type: dict Returns: **Response Syntax** { 'LocationArn': 'string' } **Response Structure** * *(dict) --* * **LocationArn** *(string) --* The ARN of the source HDFS cluster location that you create. **Exceptions** * "DataSync.Client.exceptions.InvalidRequestException" * "DataSync.Client.exceptions.InternalException" DataSync / Client / tag_resource tag_resource ************ DataSync.Client.tag_resource(**kwargs) Applies a *tag* to an Amazon Web Services resource. Tags are key- value pairs that can help you manage, filter, and search for your resources. These include DataSync resources, such as locations, tasks, and task executions. See also: AWS API Documentation **Request Syntax** response = client.tag_resource( ResourceArn='string', Tags=[ { 'Key': 'string', 'Value': 'string' }, ] ) Parameters: * **ResourceArn** (*string*) -- **[REQUIRED]** Specifies the Amazon Resource Name (ARN) of the resource to apply the tag to. * **Tags** (*list*) -- **[REQUIRED]** Specifies the tags that you want to apply to the resource. * *(dict) --* A key-value pair representing a single tag that's been applied to an Amazon Web Services resource. * **Key** *(string) --* **[REQUIRED]** The key for an Amazon Web Services resource tag. * **Value** *(string) --* The value for an Amazon Web Services resource tag. Return type: dict Returns: **Response Syntax** {} **Response Structure** * *(dict) --* **Exceptions** * "DataSync.Client.exceptions.InvalidRequestException" * "DataSync.Client.exceptions.InternalException" DataSync / Client / describe_location_nfs describe_location_nfs ********************* DataSync.Client.describe_location_nfs(**kwargs) Provides details about how an DataSync transfer location for a Network File System (NFS) file server is configured. See also: AWS API Documentation **Request Syntax** response = client.describe_location_nfs( LocationArn='string' ) Parameters: **LocationArn** (*string*) -- **[REQUIRED]** Specifies the Amazon Resource Name (ARN) of the NFS location that you want information about. Return type: dict Returns: **Response Syntax** { 'LocationArn': 'string', 'LocationUri': 'string', 'OnPremConfig': { 'AgentArns': [ 'string', ] }, 'MountOptions': { 'Version': 'AUTOMATIC'|'NFS3'|'NFS4_0'|'NFS4_1' }, 'CreationTime': datetime(2015, 1, 1) } **Response Structure** * *(dict) --* DescribeLocationNfsResponse * **LocationArn** *(string) --* The ARN of the NFS location. * **LocationUri** *(string) --* The URI of the NFS location. * **OnPremConfig** *(dict) --* The DataSync agents that can connect to your Network File System (NFS) file server. * **AgentArns** *(list) --* The Amazon Resource Names (ARNs) of the DataSync agents that can connect to your NFS file server. You can specify more than one agent. For more information, see Using multiple DataSync agents. * *(string) --* * **MountOptions** *(dict) --* The mount options that DataSync uses to mount your NFS file server. * **Version** *(string) --* Specifies the NFS version that you want DataSync to use when mounting your NFS share. If the server refuses to use the version specified, the task fails. You can specify the following options: * "AUTOMATIC" (default): DataSync chooses NFS version 4.1. * "NFS3": Stateless protocol version that allows for asynchronous writes on the server. * "NFSv4_0": Stateful, firewall-friendly protocol version that supports delegations and pseudo file systems. * "NFSv4_1": Stateful protocol version that supports sessions, directory delegations, and parallel data processing. NFS version 4.1 also includes all features available in version 4.0. Note: DataSync currently only supports NFS version 3 with Amazon FSx for NetApp ONTAP locations. * **CreationTime** *(datetime) --* The time when the NFS location was created. **Exceptions** * "DataSync.Client.exceptions.InvalidRequestException" * "DataSync.Client.exceptions.InternalException" DataSync / Client / update_location_fsx_lustre update_location_fsx_lustre ************************** DataSync.Client.update_location_fsx_lustre(**kwargs) Modifies the following configuration parameters of the Amazon FSx for Lustre transfer location that you're using with DataSync. For more information, see Configuring DataSync transfers with FSx for Lustre. See also: AWS API Documentation **Request Syntax** response = client.update_location_fsx_lustre( LocationArn='string', Subdirectory='string' ) Parameters: * **LocationArn** (*string*) -- **[REQUIRED]** Specifies the Amazon Resource Name (ARN) of the FSx for Lustre transfer location that you're updating. * **Subdirectory** (*string*) -- Specifies a mount path for your FSx for Lustre file system. The path can include subdirectories. When the location is used as a source, DataSync reads data from the mount path. When the location is used as a destination, DataSync writes data to the mount path. If you don't include this parameter, DataSync uses the file system's root directory ( "/"). Return type: dict Returns: **Response Syntax** {} **Response Structure** * *(dict) --* **Exceptions** * "DataSync.Client.exceptions.InvalidRequestException" * "DataSync.Client.exceptions.InternalException" DataSync / Client / start_task_execution start_task_execution ******************** DataSync.Client.start_task_execution(**kwargs) Starts an DataSync transfer task. For each task, you can only run one task execution at a time. There are several steps to a task execution. For more information, see Task execution statuses. Warning: If you're planning to transfer data to or from an Amazon S3 location, review how DataSync can affect your S3 request charges and the DataSync pricing page before you begin. See also: AWS API Documentation **Request Syntax** response = client.start_task_execution( TaskArn='string', OverrideOptions={ 'VerifyMode': 'POINT_IN_TIME_CONSISTENT'|'ONLY_FILES_TRANSFERRED'|'NONE', 'OverwriteMode': 'ALWAYS'|'NEVER', 'Atime': 'NONE'|'BEST_EFFORT', 'Mtime': 'NONE'|'PRESERVE', 'Uid': 'NONE'|'INT_VALUE'|'NAME'|'BOTH', 'Gid': 'NONE'|'INT_VALUE'|'NAME'|'BOTH', 'PreserveDeletedFiles': 'PRESERVE'|'REMOVE', 'PreserveDevices': 'NONE'|'PRESERVE', 'PosixPermissions': 'NONE'|'PRESERVE', 'BytesPerSecond': 123, 'TaskQueueing': 'ENABLED'|'DISABLED', 'LogLevel': 'OFF'|'BASIC'|'TRANSFER', 'TransferMode': 'CHANGED'|'ALL', 'SecurityDescriptorCopyFlags': 'NONE'|'OWNER_DACL'|'OWNER_DACL_SACL', 'ObjectTags': 'PRESERVE'|'NONE' }, Includes=[ { 'FilterType': 'SIMPLE_PATTERN', 'Value': 'string' }, ], Excludes=[ { 'FilterType': 'SIMPLE_PATTERN', 'Value': 'string' }, ], ManifestConfig={ 'Action': 'TRANSFER', 'Format': 'CSV', 'Source': { 'S3': { 'ManifestObjectPath': 'string', 'BucketAccessRoleArn': 'string', 'S3BucketArn': 'string', 'ManifestObjectVersionId': 'string' } } }, TaskReportConfig={ 'Destination': { 'S3': { 'Subdirectory': 'string', 'S3BucketArn': 'string', 'BucketAccessRoleArn': 'string' } }, 'OutputType': 'SUMMARY_ONLY'|'STANDARD', 'ReportLevel': 'ERRORS_ONLY'|'SUCCESSES_AND_ERRORS', 'ObjectVersionIds': 'INCLUDE'|'NONE', 'Overrides': { 'Transferred': { 'ReportLevel': 'ERRORS_ONLY'|'SUCCESSES_AND_ERRORS' }, 'Verified': { 'ReportLevel': 'ERRORS_ONLY'|'SUCCESSES_AND_ERRORS' }, 'Deleted': { 'ReportLevel': 'ERRORS_ONLY'|'SUCCESSES_AND_ERRORS' }, 'Skipped': { 'ReportLevel': 'ERRORS_ONLY'|'SUCCESSES_AND_ERRORS' } } }, Tags=[ { 'Key': 'string', 'Value': 'string' }, ] ) Parameters: * **TaskArn** (*string*) -- **[REQUIRED]** Specifies the Amazon Resource Name (ARN) of the task that you want to start. * **OverrideOptions** (*dict*) -- Indicates how your transfer task is configured. These options include how DataSync handles files, objects, and their associated metadata during your transfer. You also can specify how to verify data integrity, set bandwidth limits for your task, among other options. Each option has a default value. Unless you need to, you don't have to configure any option before calling StartTaskExecution. You also can override your task options for each task execution. For example, you might want to adjust the "LogLevel" for an individual execution. * **VerifyMode** *(string) --* Specifies if and how DataSync checks the integrity of your data at the end of your transfer. * "ONLY_FILES_TRANSFERRED" (recommended) - DataSync calculates the checksum of transferred data (including metadata) at the source location. At the end of the transfer, DataSync then compares this checksum to the checksum calculated on that data at the destination. Note: This is the default option for Enhanced mode tasks. We recommend this option when transferring to S3 Glacier Flexible Retrieval or S3 Glacier Deep Archive storage classes. For more information, see Storage class considerations with Amazon S3 locations. * "POINT_IN_TIME_CONSISTENT" - At the end of the transfer, DataSync checks the entire source and destination to verify that both locations are fully synchronized. Note: The is the default option for Basic mode tasks and isn't currently supported with Enhanced mode tasks. If you use a manifest, DataSync only scans and verifies what's listed in the manifest. You can't use this option when transferring to S3 Glacier Flexible Retrieval or S3 Glacier Deep Archive storage classes. For more information, see Storage class considerations with Amazon S3 locations. * "NONE" - DataSync performs data integrity checks only during your transfer. Unlike other options, there's no additional verification at the end of your transfer. * **OverwriteMode** *(string) --* Specifies whether DataSync should modify or preserve data at the destination location. * "ALWAYS" (default) - DataSync modifies data in the destination location when source data (including metadata) has changed. If DataSync overwrites objects, you might incur additional charges for certain Amazon S3 storage classes (for example, for retrieval or early deletion). For more information, see Storage class considerations with Amazon S3 transfers. * "NEVER" - DataSync doesn't overwrite data in the destination location even if the source data has changed. You can use this option to protect against overwriting changes made to files or objects in the destination. * **Atime** *(string) --* Specifies whether to preserve metadata indicating the last time a file was read or written to. Note: The behavior of "Atime" isn't fully standard across platforms, so DataSync can only do this on a best-effort basis. * "BEST_EFFORT" (default) - DataSync attempts to preserve the original "Atime" attribute on all source files (that is, the version before the "PREPARING" steps of the task execution). This option is recommended. * "NONE" - Ignores "Atime". Note: If "Atime" is set to "BEST_EFFORT", "Mtime" must be set to "PRESERVE".If "Atime" is set to "NONE", "Mtime" must also be "NONE". * **Mtime** *(string) --* Specifies whether to preserve metadata indicating the last time that a file was written to before the "PREPARING" step of your task execution. This option is required when you need to run the a task more than once. * "PRESERVE" (default) - Preserves original "Mtime", which is recommended. * "NONE" - Ignores "Mtime". Note: If "Mtime" is set to "PRESERVE", "Atime" must be set to "BEST_EFFORT".If "Mtime" is set to "NONE", "Atime" must also be set to "NONE". * **Uid** *(string) --* Specifies the POSIX user ID (UID) of the file's owner. * "INT_VALUE" (default) - Preserves the integer value of UID and group ID (GID), which is recommended. * "NONE" - Ignores UID and GID. For more information, see Metadata copied by DataSync. * **Gid** *(string) --* Specifies the POSIX group ID (GID) of the file's owners. * "INT_VALUE" (default) - Preserves the integer value of user ID (UID) and GID, which is recommended. * "NONE" - Ignores UID and GID. For more information, see Understanding how DataSync handles file and object metadata. * **PreserveDeletedFiles** *(string) --* Specifies whether files in the destination location that don't exist in the source should be preserved. This option can affect your Amazon S3 storage cost. If your task deletes objects, you might incur minimum storage duration charges for certain storage classes. For detailed information, see Considerations when working with Amazon S3 storage classes in DataSync. * "PRESERVE" (default) - Ignores such destination files, which is recommended. * "REMOVE" - Deletes destination files that aren’t present in the source. Note: If you set this parameter to "REMOVE", you can't set "TransferMode" to "ALL". When you transfer all data, DataSync doesn't scan your destination location and doesn't know what to delete. * **PreserveDevices** *(string) --* Specifies whether DataSync should preserve the metadata of block and character devices in the source location and recreate the files with that device name and metadata on the destination. DataSync copies only the name and metadata of such devices. Note: DataSync can't copy the actual contents of these devices because they're nonterminal and don't return an end-of- file (EOF) marker. * "NONE" (default) - Ignores special devices (recommended). * "PRESERVE" - Preserves character and block device metadata. This option currently isn't supported for Amazon EFS. * **PosixPermissions** *(string) --* Specifies which users or groups can access a file for a specific purpose such as reading, writing, or execution of the file. For more information, see Understanding how DataSync handles file and object metadata. * "PRESERVE" (default) - Preserves POSIX-style permissions, which is recommended. * "NONE" - Ignores POSIX-style permissions. Note: DataSync can preserve extant permissions of a source location. * **BytesPerSecond** *(integer) --* Limits the bandwidth used by a DataSync task. For example, if you want DataSync to use a maximum of 1 MB, set this value to "1048576" ( "=1024*1024"). Note: Not applicable to Enhanced mode tasks. * **TaskQueueing** *(string) --* Specifies whether your transfer tasks should be put into a queue during certain scenarios when running multiple tasks. This is "ENABLED" by default. * **LogLevel** *(string) --* Specifies the type of logs that DataSync publishes to a Amazon CloudWatch Logs log group. To specify the log group, see CloudWatchLogGroupArn. * "BASIC" - Publishes logs with only basic information (such as transfer errors). * "TRANSFER" - Publishes logs for all files or objects that your DataSync task transfers and performs data-integrity checks on. * "OFF" - No logs are published. * **TransferMode** *(string) --* Specifies whether DataSync transfers only the data (including metadata) that differs between locations following an initial copy or transfers all data every time you run the task. If you're planning on recurring transfers, you might only want to transfer what's changed since your previous task execution. * "CHANGED" (default) - After your initial full transfer, DataSync copies only the data and metadata that differs between the source and destination location. * "ALL" - DataSync copies everything in the source to the destination without comparing differences between the locations. * **SecurityDescriptorCopyFlags** *(string) --* Specifies which components of the SMB security descriptor are copied from source to destination objects. This value is only used for transfers between SMB and Amazon FSx for Windows File Server locations or between two FSx for Windows File Server locations. For more information, see Understanding how DataSync handles file and object metadata. * "OWNER_DACL" (default) - For each copied object, DataSync copies the following metadata: * The object owner. * NTFS discretionary access control lists (DACLs), which determine whether to grant access to an object. DataSync won't copy NTFS system access control lists (SACLs) with this option. * "OWNER_DACL_SACL" - For each copied object, DataSync copies the following metadata: * The object owner. * NTFS discretionary access control lists (DACLs), which determine whether to grant access to an object. * SACLs, which are used by administrators to log attempts to access a secured object. Copying SACLs requires granting additional permissions to the Windows user that DataSync uses to access your SMB location. For information about choosing a user with the right permissions, see required permissions for SMB, FSx for Windows File Server, or FSx for ONTAP (depending on the type of location in your transfer). * "NONE" - None of the SMB security descriptor components are copied. Destination objects are owned by the user that was provided for accessing the destination location. DACLs and SACLs are set based on the destination server’s configuration. * **ObjectTags** *(string) --* Specifies whether you want DataSync to "PRESERVE" object tags (default behavior) when transferring between object storage systems. If you want your DataSync task to ignore object tags, specify the "NONE" value. * **Includes** (*list*) -- Specifies a list of filter rules that determines which files to include when running a task. The pattern should contain a single filter string that consists of the patterns to include. The patterns are delimited by "|" (that is, a pipe), for example, ""/folder1|/folder2"". * *(dict) --* Specifies which files, folders, and objects to include or exclude when transferring files from source to destination. * **FilterType** *(string) --* The type of filter rule to apply. DataSync only supports the SIMPLE_PATTERN rule type. * **Value** *(string) --* A single filter string that consists of the patterns to include or exclude. The patterns are delimited by "|" (that is, a pipe), for example: "/folder1|/folder2" * **Excludes** (*list*) -- Specifies a list of filter rules that determines which files to exclude from a task. The list contains a single filter string that consists of the patterns to exclude. The patterns are delimited by "|" (that is, a pipe), for example, ""/folder1|/folder2"". * *(dict) --* Specifies which files, folders, and objects to include or exclude when transferring files from source to destination. * **FilterType** *(string) --* The type of filter rule to apply. DataSync only supports the SIMPLE_PATTERN rule type. * **Value** *(string) --* A single filter string that consists of the patterns to include or exclude. The patterns are delimited by "|" (that is, a pipe), for example: "/folder1|/folder2" * **ManifestConfig** (*dict*) -- Configures a manifest, which is a list of files or objects that you want DataSync to transfer. For more information and configuration examples, see Specifying what DataSync transfers by using a manifest. When using this parameter, your caller identity (the role that you're using DataSync with) must have the "iam:PassRole" permission. The AWSDataSyncFullAccess policy includes this permission. To remove a manifest configuration, specify this parameter with an empty value. * **Action** *(string) --* Specifies what DataSync uses the manifest for. * **Format** *(string) --* Specifies the file format of your manifest. For more information, see Creating a manifest. * **Source** *(dict) --* Specifies the manifest that you want DataSync to use and where it's hosted. Note: You must specify this parameter if you're configuring a new manifest on or after February 7, 2024.If you don't, you'll get a 400 status code and "ValidationException" error stating that you're missing the IAM role for DataSync to access the S3 bucket where you're hosting your manifest. For more information, see Providing DataSync access to your manifest. * **S3** *(dict) --* **[REQUIRED]** Specifies the S3 bucket where you're hosting your manifest. * **ManifestObjectPath** *(string) --* **[REQUIRED]** Specifies the Amazon S3 object key of your manifest. This can include a prefix (for example, "prefix/my- manifest.csv"). * **BucketAccessRoleArn** *(string) --* **[REQUIRED]** Specifies the Identity and Access Management (IAM) role that allows DataSync to access your manifest. For more information, see Providing DataSync access to your manifest. * **S3BucketArn** *(string) --* **[REQUIRED]** Specifies the Amazon Resource Name (ARN) of the S3 bucket where you're hosting your manifest. * **ManifestObjectVersionId** *(string) --* Specifies the object version ID of the manifest that you want DataSync to use. If you don't set this, DataSync uses the latest version of the object. * **TaskReportConfig** (*dict*) -- Specifies how you want to configure a task report, which provides detailed information about your DataSync transfer. For more information, see Monitoring your DataSync transfers with task reports. When using this parameter, your caller identity (the role that you're using DataSync with) must have the "iam:PassRole" permission. The AWSDataSyncFullAccess policy includes this permission. To remove a task report configuration, specify this parameter as empty. * **Destination** *(dict) --* Specifies the Amazon S3 bucket where DataSync uploads your task report. For more information, see Task reports. * **S3** *(dict) --* Specifies the Amazon S3 bucket where DataSync uploads your task report. * **Subdirectory** *(string) --* Specifies a bucket prefix for your report. * **S3BucketArn** *(string) --* **[REQUIRED]** Specifies the ARN of the S3 bucket where DataSync uploads your report. * **BucketAccessRoleArn** *(string) --* **[REQUIRED]** Specifies the Amazon Resource Name (ARN) of the IAM policy that allows DataSync to upload a task report to your S3 bucket. For more information, see Allowing DataSync to upload a task report to an Amazon S3 bucket. * **OutputType** *(string) --* Specifies the type of task report that you want: * "SUMMARY_ONLY": Provides necessary details about your task, including the number of files, objects, and directories transferred and transfer duration. * "STANDARD": Provides complete details about your task, including a full list of files, objects, and directories that were transferred, skipped, verified, and more. * **ReportLevel** *(string) --* Specifies whether you want your task report to include only what went wrong with your transfer or a list of what succeeded and didn't. * "ERRORS_ONLY": A report shows what DataSync was unable to transfer, skip, verify, and delete. * "SUCCESSES_AND_ERRORS": A report shows what DataSync was able and unable to transfer, skip, verify, and delete. * **ObjectVersionIds** *(string) --* Specifies whether your task report includes the new version of each object transferred into an S3 bucket. This only applies if you enable versioning on your bucket. Keep in mind that setting this to "INCLUDE" can increase the duration of your task execution. * **Overrides** *(dict) --* Customizes the reporting level for aspects of your task report. For example, your report might generally only include errors, but you could specify that you want a list of successes and errors just for the files that DataSync attempted to delete in your destination location. * **Transferred** *(dict) --* Specifies the level of reporting for the files, objects, and directories that DataSync attempted to transfer. * **ReportLevel** *(string) --* Specifies whether your task report includes errors only or successes and errors. For example, your report might mostly include only what didn't go well in your transfer ( "ERRORS_ONLY"). At the same time, you want to verify that your task filter is working correctly. In this situation, you can get a list of what files DataSync successfully skipped and if something transferred that you didn't to transfer ( "SUCCESSES_AND_ERRORS"). * **Verified** *(dict) --* Specifies the level of reporting for the files, objects, and directories that DataSync attempted to verify at the end of your transfer. * **ReportLevel** *(string) --* Specifies whether your task report includes errors only or successes and errors. For example, your report might mostly include only what didn't go well in your transfer ( "ERRORS_ONLY"). At the same time, you want to verify that your task filter is working correctly. In this situation, you can get a list of what files DataSync successfully skipped and if something transferred that you didn't to transfer ( "SUCCESSES_AND_ERRORS"). * **Deleted** *(dict) --* Specifies the level of reporting for the files, objects, and directories that DataSync attempted to delete in your destination location. This only applies if you configure your task to delete data in the destination that isn't in the source. * **ReportLevel** *(string) --* Specifies whether your task report includes errors only or successes and errors. For example, your report might mostly include only what didn't go well in your transfer ( "ERRORS_ONLY"). At the same time, you want to verify that your task filter is working correctly. In this situation, you can get a list of what files DataSync successfully skipped and if something transferred that you didn't to transfer ( "SUCCESSES_AND_ERRORS"). * **Skipped** *(dict) --* Specifies the level of reporting for the files, objects, and directories that DataSync attempted to skip during your transfer. * **ReportLevel** *(string) --* Specifies whether your task report includes errors only or successes and errors. For example, your report might mostly include only what didn't go well in your transfer ( "ERRORS_ONLY"). At the same time, you want to verify that your task filter is working correctly. In this situation, you can get a list of what files DataSync successfully skipped and if something transferred that you didn't to transfer ( "SUCCESSES_AND_ERRORS"). * **Tags** (*list*) -- Specifies the tags that you want to apply to the Amazon Resource Name (ARN) representing the task execution. *Tags* are key-value pairs that help you manage, filter, and search for your DataSync resources. * *(dict) --* A key-value pair representing a single tag that's been applied to an Amazon Web Services resource. * **Key** *(string) --* **[REQUIRED]** The key for an Amazon Web Services resource tag. * **Value** *(string) --* The value for an Amazon Web Services resource tag. Return type: dict Returns: **Response Syntax** { 'TaskExecutionArn': 'string' } **Response Structure** * *(dict) --* StartTaskExecutionResponse * **TaskExecutionArn** *(string) --* The ARN of the running task execution. **Exceptions** * "DataSync.Client.exceptions.InvalidRequestException" * "DataSync.Client.exceptions.InternalException" DataSync / Client / update_location_fsx_ontap update_location_fsx_ontap ************************* DataSync.Client.update_location_fsx_ontap(**kwargs) Modifies the following configuration parameters of the Amazon FSx for NetApp ONTAP transfer location that you're using with DataSync. For more information, see Configuring DataSync transfers with FSx for ONTAP. See also: AWS API Documentation **Request Syntax** response = client.update_location_fsx_ontap( LocationArn='string', Protocol={ 'NFS': { 'MountOptions': { 'Version': 'AUTOMATIC'|'NFS3'|'NFS4_0'|'NFS4_1' } }, 'SMB': { 'Domain': 'string', 'MountOptions': { 'Version': 'AUTOMATIC'|'SMB2'|'SMB3'|'SMB1'|'SMB2_0' }, 'Password': 'string', 'User': 'string' } }, Subdirectory='string' ) Parameters: * **LocationArn** (*string*) -- **[REQUIRED]** Specifies the Amazon Resource Name (ARN) of the FSx for ONTAP transfer location that you're updating. * **Protocol** (*dict*) -- Specifies the data transfer protocol that DataSync uses to access your Amazon FSx file system. * **NFS** *(dict) --* Specifies the Network File System (NFS) protocol configuration that DataSync uses to access your FSx for OpenZFS file system or FSx for ONTAP file system's storage virtual machine (SVM). * **MountOptions** *(dict) --* Specifies how DataSync can access a location using the NFS protocol. * **Version** *(string) --* Specifies the NFS version that you want DataSync to use when mounting your NFS share. If the server refuses to use the version specified, the task fails. You can specify the following options: * "AUTOMATIC" (default): DataSync chooses NFS version 4.1. * "NFS3": Stateless protocol version that allows for asynchronous writes on the server. * "NFSv4_0": Stateful, firewall-friendly protocol version that supports delegations and pseudo file systems. * "NFSv4_1": Stateful protocol version that supports sessions, directory delegations, and parallel data processing. NFS version 4.1 also includes all features available in version 4.0. Note: DataSync currently only supports NFS version 3 with Amazon FSx for NetApp ONTAP locations. * **SMB** *(dict) --* Specifies the Server Message Block (SMB) protocol configuration that DataSync uses to access your FSx for ONTAP file system's storage virtual machine (SVM). * **Domain** *(string) --* Specifies the name of the Windows domain that your storage virtual machine (SVM) belongs to. If you have multiple Active Directory domains in your environment, configuring this parameter makes sure that DataSync connects to the right SVM. * **MountOptions** *(dict) --* Specifies the version of the Server Message Block (SMB) protocol that DataSync uses to access an SMB file server. * **Version** *(string) --* By default, DataSync automatically chooses an SMB protocol version based on negotiation with your SMB file server. You also can configure DataSync to use a specific SMB version, but we recommend doing this only if DataSync has trouble negotiating with the SMB file server automatically. These are the following options for configuring the SMB version: * "AUTOMATIC" (default): DataSync and the SMB file server negotiate the highest version of SMB that they mutually support between 2.1 and 3.1.1. This is the recommended option. If you instead choose a specific version that your file server doesn't support, you may get an "Operation Not Supported" error. * "SMB3": Restricts the protocol negotiation to only SMB version 3.0.2. * "SMB2": Restricts the protocol negotiation to only SMB version 2.1. * "SMB2_0": Restricts the protocol negotiation to only SMB version 2.0. * "SMB1": Restricts the protocol negotiation to only SMB version 1.0. Note: The "SMB1" option isn't available when creating an Amazon FSx for NetApp ONTAP location. * **Password** *(string) --* Specifies the password of a user who has permission to access your SVM. * **User** *(string) --* Specifies a user that can mount and access the files, folders, and metadata in your SVM. For information about choosing a user with the right level of access for your transfer, see Using the SMB protocol. * **Subdirectory** (*string*) -- Specifies a path to the file share in the storage virtual machine (SVM) where you want to transfer data to or from. You can specify a junction path (also known as a mount point), qtree path (for NFS file shares), or share name (for SMB file shares). For example, your mount path might be "/vol1", "/vol1/tree1", or "/share1". Note: Don't specify a junction path in the SVM's root volume. For more information, see Managing FSx for ONTAP storage virtual machines in the *Amazon FSx for NetApp ONTAP User Guide*. Return type: dict Returns: **Response Syntax** {} **Response Structure** * *(dict) --* **Exceptions** * "DataSync.Client.exceptions.InvalidRequestException" * "DataSync.Client.exceptions.InternalException" DataSync / Client / update_location_smb update_location_smb ******************* DataSync.Client.update_location_smb(**kwargs) Modifies the following configuration parameters of the Server Message Block (SMB) transfer location that you're using with DataSync. For more information, see Configuring DataSync transfers with an SMB file server. See also: AWS API Documentation **Request Syntax** response = client.update_location_smb( LocationArn='string', Subdirectory='string', ServerHostname='string', User='string', Domain='string', Password='string', AgentArns=[ 'string', ], MountOptions={ 'Version': 'AUTOMATIC'|'SMB2'|'SMB3'|'SMB1'|'SMB2_0' }, AuthenticationType='NTLM'|'KERBEROS', DnsIpAddresses=[ 'string', ], KerberosPrincipal='string', KerberosKeytab=b'bytes', KerberosKrb5Conf=b'bytes' ) Parameters: * **LocationArn** (*string*) -- **[REQUIRED]** Specifies the ARN of the SMB location that you want to update. * **Subdirectory** (*string*) -- Specifies the name of the share exported by your SMB file server where DataSync will read or write data. You can include a subdirectory in the share path (for example, "/path/to/subdirectory"). Make sure that other SMB clients in your network can also mount this path. To copy all data in the specified subdirectory, DataSync must be able to mount the SMB share and access all of its data. For more information, see Providing DataSync access to SMB file servers. * **ServerHostname** (*string*) -- Specifies the domain name or IP address (IPv4 or IPv6) of the SMB file server that your DataSync agent connects to. Note: If you're using Kerberos authentication, you must specify a domain name. * **User** (*string*) -- Specifies the user name that can mount your SMB file server and has permission to access the files and folders involved in your transfer. This parameter applies only if "AuthenticationType" is set to "NTLM". For information about choosing a user with the right level of access for your transfer, see Providing DataSync access to SMB file servers. * **Domain** (*string*) -- Specifies the Windows domain name that your SMB file server belongs to. This parameter applies only if "AuthenticationType" is set to "NTLM". If you have multiple domains in your environment, configuring this parameter makes sure that DataSync connects to the right file server. * **Password** (*string*) -- Specifies the password of the user who can mount your SMB file server and has permission to access the files and folders involved in your transfer. This parameter applies only if "AuthenticationType" is set to "NTLM". * **AgentArns** (*list*) -- Specifies the DataSync agent (or agents) that can connect to your SMB file server. You specify an agent by using its Amazon Resource Name (ARN). * *(string) --* * **MountOptions** (*dict*) -- Specifies the version of the Server Message Block (SMB) protocol that DataSync uses to access an SMB file server. * **Version** *(string) --* By default, DataSync automatically chooses an SMB protocol version based on negotiation with your SMB file server. You also can configure DataSync to use a specific SMB version, but we recommend doing this only if DataSync has trouble negotiating with the SMB file server automatically. These are the following options for configuring the SMB version: * "AUTOMATIC" (default): DataSync and the SMB file server negotiate the highest version of SMB that they mutually support between 2.1 and 3.1.1. This is the recommended option. If you instead choose a specific version that your file server doesn't support, you may get an "Operation Not Supported" error. * "SMB3": Restricts the protocol negotiation to only SMB version 3.0.2. * "SMB2": Restricts the protocol negotiation to only SMB version 2.1. * "SMB2_0": Restricts the protocol negotiation to only SMB version 2.0. * "SMB1": Restricts the protocol negotiation to only SMB version 1.0. Note: The "SMB1" option isn't available when creating an Amazon FSx for NetApp ONTAP location. * **AuthenticationType** (*string*) -- Specifies the authentication protocol that DataSync uses to connect to your SMB file server. DataSync supports "NTLM" (default) and "KERBEROS" authentication. For more information, see Providing DataSync access to SMB file servers. * **DnsIpAddresses** (*list*) -- Specifies the IP addresses (IPv4 or IPv6) for the DNS servers that your SMB file server belongs to. This parameter applies only if "AuthenticationType" is set to "KERBEROS". If you have multiple domains in your environment, configuring this parameter makes sure that DataSync connects to the right SMB file server. * *(string) --* * **KerberosPrincipal** (*string*) -- Specifies a Kerberos prinicpal, which is an identity in your Kerberos realm that has permission to access the files, folders, and file metadata in your SMB file server. A Kerberos principal might look like "HOST/kerberosuser@MYDOMAIN.ORG". Principal names are case sensitive. Your DataSync task execution will fail if the principal that you specify for this parameter doesn’t exactly match the principal that you use to create the keytab file. * **KerberosKeytab** (*bytes*) -- Specifies your Kerberos key table (keytab) file, which includes mappings between your Kerberos principal and encryption keys. To avoid task execution errors, make sure that the Kerberos principal that you use to create the keytab file matches exactly what you specify for "KerberosPrincipal". * **KerberosKrb5Conf** (*bytes*) -- Specifies a Kerberos configuration file ( "krb5.conf") that defines your Kerberos realm configuration. The file must be base64 encoded. If you're using the CLI, the encoding is done for you. Return type: dict Returns: **Response Syntax** {} **Response Structure** * *(dict) --* **Exceptions** * "DataSync.Client.exceptions.InvalidRequestException" * "DataSync.Client.exceptions.InternalException" DataSync / Client / update_task_execution update_task_execution ********************* DataSync.Client.update_task_execution(**kwargs) Updates the configuration of a running DataSync task execution. Note: Currently, the only "Option" that you can modify with "UpdateTaskExecution" is >>``<>``<<, which throttles bandwidth for a running or queued task execution. See also: AWS API Documentation **Request Syntax** response = client.update_task_execution( TaskExecutionArn='string', Options={ 'VerifyMode': 'POINT_IN_TIME_CONSISTENT'|'ONLY_FILES_TRANSFERRED'|'NONE', 'OverwriteMode': 'ALWAYS'|'NEVER', 'Atime': 'NONE'|'BEST_EFFORT', 'Mtime': 'NONE'|'PRESERVE', 'Uid': 'NONE'|'INT_VALUE'|'NAME'|'BOTH', 'Gid': 'NONE'|'INT_VALUE'|'NAME'|'BOTH', 'PreserveDeletedFiles': 'PRESERVE'|'REMOVE', 'PreserveDevices': 'NONE'|'PRESERVE', 'PosixPermissions': 'NONE'|'PRESERVE', 'BytesPerSecond': 123, 'TaskQueueing': 'ENABLED'|'DISABLED', 'LogLevel': 'OFF'|'BASIC'|'TRANSFER', 'TransferMode': 'CHANGED'|'ALL', 'SecurityDescriptorCopyFlags': 'NONE'|'OWNER_DACL'|'OWNER_DACL_SACL', 'ObjectTags': 'PRESERVE'|'NONE' } ) Parameters: * **TaskExecutionArn** (*string*) -- **[REQUIRED]** Specifies the Amazon Resource Name (ARN) of the task execution that you're updating. * **Options** (*dict*) -- **[REQUIRED]** Indicates how your transfer task is configured. These options include how DataSync handles files, objects, and their associated metadata during your transfer. You also can specify how to verify data integrity, set bandwidth limits for your task, among other options. Each option has a default value. Unless you need to, you don't have to configure any option before calling StartTaskExecution. You also can override your task options for each task execution. For example, you might want to adjust the "LogLevel" for an individual execution. * **VerifyMode** *(string) --* Specifies if and how DataSync checks the integrity of your data at the end of your transfer. * "ONLY_FILES_TRANSFERRED" (recommended) - DataSync calculates the checksum of transferred data (including metadata) at the source location. At the end of the transfer, DataSync then compares this checksum to the checksum calculated on that data at the destination. Note: This is the default option for Enhanced mode tasks. We recommend this option when transferring to S3 Glacier Flexible Retrieval or S3 Glacier Deep Archive storage classes. For more information, see Storage class considerations with Amazon S3 locations. * "POINT_IN_TIME_CONSISTENT" - At the end of the transfer, DataSync checks the entire source and destination to verify that both locations are fully synchronized. Note: The is the default option for Basic mode tasks and isn't currently supported with Enhanced mode tasks. If you use a manifest, DataSync only scans and verifies what's listed in the manifest. You can't use this option when transferring to S3 Glacier Flexible Retrieval or S3 Glacier Deep Archive storage classes. For more information, see Storage class considerations with Amazon S3 locations. * "NONE" - DataSync performs data integrity checks only during your transfer. Unlike other options, there's no additional verification at the end of your transfer. * **OverwriteMode** *(string) --* Specifies whether DataSync should modify or preserve data at the destination location. * "ALWAYS" (default) - DataSync modifies data in the destination location when source data (including metadata) has changed. If DataSync overwrites objects, you might incur additional charges for certain Amazon S3 storage classes (for example, for retrieval or early deletion). For more information, see Storage class considerations with Amazon S3 transfers. * "NEVER" - DataSync doesn't overwrite data in the destination location even if the source data has changed. You can use this option to protect against overwriting changes made to files or objects in the destination. * **Atime** *(string) --* Specifies whether to preserve metadata indicating the last time a file was read or written to. Note: The behavior of "Atime" isn't fully standard across platforms, so DataSync can only do this on a best-effort basis. * "BEST_EFFORT" (default) - DataSync attempts to preserve the original "Atime" attribute on all source files (that is, the version before the "PREPARING" steps of the task execution). This option is recommended. * "NONE" - Ignores "Atime". Note: If "Atime" is set to "BEST_EFFORT", "Mtime" must be set to "PRESERVE".If "Atime" is set to "NONE", "Mtime" must also be "NONE". * **Mtime** *(string) --* Specifies whether to preserve metadata indicating the last time that a file was written to before the "PREPARING" step of your task execution. This option is required when you need to run the a task more than once. * "PRESERVE" (default) - Preserves original "Mtime", which is recommended. * "NONE" - Ignores "Mtime". Note: If "Mtime" is set to "PRESERVE", "Atime" must be set to "BEST_EFFORT".If "Mtime" is set to "NONE", "Atime" must also be set to "NONE". * **Uid** *(string) --* Specifies the POSIX user ID (UID) of the file's owner. * "INT_VALUE" (default) - Preserves the integer value of UID and group ID (GID), which is recommended. * "NONE" - Ignores UID and GID. For more information, see Metadata copied by DataSync. * **Gid** *(string) --* Specifies the POSIX group ID (GID) of the file's owners. * "INT_VALUE" (default) - Preserves the integer value of user ID (UID) and GID, which is recommended. * "NONE" - Ignores UID and GID. For more information, see Understanding how DataSync handles file and object metadata. * **PreserveDeletedFiles** *(string) --* Specifies whether files in the destination location that don't exist in the source should be preserved. This option can affect your Amazon S3 storage cost. If your task deletes objects, you might incur minimum storage duration charges for certain storage classes. For detailed information, see Considerations when working with Amazon S3 storage classes in DataSync. * "PRESERVE" (default) - Ignores such destination files, which is recommended. * "REMOVE" - Deletes destination files that aren’t present in the source. Note: If you set this parameter to "REMOVE", you can't set "TransferMode" to "ALL". When you transfer all data, DataSync doesn't scan your destination location and doesn't know what to delete. * **PreserveDevices** *(string) --* Specifies whether DataSync should preserve the metadata of block and character devices in the source location and recreate the files with that device name and metadata on the destination. DataSync copies only the name and metadata of such devices. Note: DataSync can't copy the actual contents of these devices because they're nonterminal and don't return an end-of- file (EOF) marker. * "NONE" (default) - Ignores special devices (recommended). * "PRESERVE" - Preserves character and block device metadata. This option currently isn't supported for Amazon EFS. * **PosixPermissions** *(string) --* Specifies which users or groups can access a file for a specific purpose such as reading, writing, or execution of the file. For more information, see Understanding how DataSync handles file and object metadata. * "PRESERVE" (default) - Preserves POSIX-style permissions, which is recommended. * "NONE" - Ignores POSIX-style permissions. Note: DataSync can preserve extant permissions of a source location. * **BytesPerSecond** *(integer) --* Limits the bandwidth used by a DataSync task. For example, if you want DataSync to use a maximum of 1 MB, set this value to "1048576" ( "=1024*1024"). Note: Not applicable to Enhanced mode tasks. * **TaskQueueing** *(string) --* Specifies whether your transfer tasks should be put into a queue during certain scenarios when running multiple tasks. This is "ENABLED" by default. * **LogLevel** *(string) --* Specifies the type of logs that DataSync publishes to a Amazon CloudWatch Logs log group. To specify the log group, see CloudWatchLogGroupArn. * "BASIC" - Publishes logs with only basic information (such as transfer errors). * "TRANSFER" - Publishes logs for all files or objects that your DataSync task transfers and performs data-integrity checks on. * "OFF" - No logs are published. * **TransferMode** *(string) --* Specifies whether DataSync transfers only the data (including metadata) that differs between locations following an initial copy or transfers all data every time you run the task. If you're planning on recurring transfers, you might only want to transfer what's changed since your previous task execution. * "CHANGED" (default) - After your initial full transfer, DataSync copies only the data and metadata that differs between the source and destination location. * "ALL" - DataSync copies everything in the source to the destination without comparing differences between the locations. * **SecurityDescriptorCopyFlags** *(string) --* Specifies which components of the SMB security descriptor are copied from source to destination objects. This value is only used for transfers between SMB and Amazon FSx for Windows File Server locations or between two FSx for Windows File Server locations. For more information, see Understanding how DataSync handles file and object metadata. * "OWNER_DACL" (default) - For each copied object, DataSync copies the following metadata: * The object owner. * NTFS discretionary access control lists (DACLs), which determine whether to grant access to an object. DataSync won't copy NTFS system access control lists (SACLs) with this option. * "OWNER_DACL_SACL" - For each copied object, DataSync copies the following metadata: * The object owner. * NTFS discretionary access control lists (DACLs), which determine whether to grant access to an object. * SACLs, which are used by administrators to log attempts to access a secured object. Copying SACLs requires granting additional permissions to the Windows user that DataSync uses to access your SMB location. For information about choosing a user with the right permissions, see required permissions for SMB, FSx for Windows File Server, or FSx for ONTAP (depending on the type of location in your transfer). * "NONE" - None of the SMB security descriptor components are copied. Destination objects are owned by the user that was provided for accessing the destination location. DACLs and SACLs are set based on the destination server’s configuration. * **ObjectTags** *(string) --* Specifies whether you want DataSync to "PRESERVE" object tags (default behavior) when transferring between object storage systems. If you want your DataSync task to ignore object tags, specify the "NONE" value. Return type: dict Returns: **Response Syntax** {} **Response Structure** * *(dict) --* **Exceptions** * "DataSync.Client.exceptions.InvalidRequestException" * "DataSync.Client.exceptions.InternalException"