Overview
The Amazon S3 Task is an SSIS Control Flow task for sending and receiving files to Amazon S3 service.
Demonstration
Quick Start
In this section we will show you how to set up an Amazon S3 Task to receive a file.
- Before you begin, configure an Amazon S3 connection manager and a connection manager for the local folder. The name(s) of the file(s) downloaded from Amazon S3 will be used as the local name(s) so do not specify a path that includes a filename in the connection manager. In this example, we are using a File connection manager to specify a path to an existing folder.
- Drag the Amazon S3 Task from the SSIS Toolbox to the Control Flow canvas.
- Double-click on the task on the canvas to open the task editor.
- Once the task editor opens, select the Action you wish to perform (Send files, Receive files, Remove remote files, Get remote files list, Rename remote file, Create remote directory, Remove remote directory). In this example, we will choose Receive files.
- For the Local parameter, choose either a Flat File or a File connection manager. In this example we’re using a File connection manager named Existing folder which points at a local path (C:....\Diane's Files\Amazon S3 Files).
- For the AmazonS3Connection parameter, choose the Amazon S3 connection manager you already set up.
- For the Remote parameter, specify the path to the file to be downloaded from Amazon S3, starting with the name of the bucket. In this example, the bucket name is “coldestdayinschaumburg”.
- Click OK to close the task editor.
Congratulations! You have successfully configured the Amazon S3 task to receive a file.
Case Study
Parameters
General
Use the General page of the Amazon S3 Task Editor dialog to configure the parameters needed to send and receive files.
Specify the task name.
Specify the task description.
Specify the task action. This parameter has the options listed in the following table.
Option Description Send files Sends a file. Selecting this action displays the dynamic parameters IsLocalVariable, Local, LocalVariable, ContentType, AccessPolicy, IsLocalFilterVariable, LocalFilter, LocalFilterVariable, AmazonS3Connection, IsRemoteVariable, Remote, RemoteVariable. Receive files Downloads a single or multiple files. Selecting this action displays the dynamic parameters IsLocalVariable, Local, LocalVariable, AmazonS3Connection, IsRemoteVariable, Remote, RemoteVariable, IsRemoteFilterVariable, RemoteFilter, RemoteFilterVariable. Remove remote files Removes a remote file. Selecting this action displays the dynamic parameters AmazonS3Connection, IsRemoteVariable, Remote, RemoteVariable, IsRemoteFilterVariable, RemoteFilter, RemoteFilterVariable. Get remote files list Gets a list of remote files. Selecting this action displays the dynamic parameters ResultVariable, AmazonS3Connection, IsRemoteVariable, Remote, RemoteVariable, IsRemoteFilterVariable, RemoteFilter, RemoteFilterVariable. Rename remote file Renames remote file. Selecting this action displays the dynamic parameters AmazonS3Connection, IsOldNameVariable, OldName, OldNameVariable, IsNewNameVariable, NewName, NewNameVariable. Create remote directory Creates remote directory. Selecting this action displays the dynamic parameters AmazonS3Connection, IsRemoteVariable, Remote, RemoteVariable. Remove remote directory Removes remote directory. Selecting this action displays the dynamic parameters AmazonS3Connection, IsRemoteVariable, Remote, RemoteVariable. Indicate whether or not the local path is stored in a variable. This parameter has the options listed in the following table.
Option Description True The local path is stored in a variable. Selecting this option displays the dynamic parameter LocalVariable. False The local path is specified in a File connection manager. Selecting this option displays the dynamic parameter Local. Select an existing File connection manager, or click <New connection...> to create a new connection manager.
Related topics: File Connection Manager Editor
Contains the local path or Stream object. Select an existing user-defined variable, or click <New variable...> to create a new variable.
Related Topics: Integration Services Variables, Add Variable
Specify the content type of the uploaded file.
Select the access policy of the uploaded file. This parameter has the options listed in the following table.
Option Description Private The file is not available for access from users other than the owner. Public-read The file is available for read by anyone. Public-read-write The file is available for read and write by anyone. Authenticated-read The file is available for read only by authenticated users. Indicate whether or not the local filter is stored in a variable. This parameter has the options listed in the following table.
Options Description True The local filter is stored in a variable. Selecting this option displays the dynamic parameter LocalFilterVariable. False The local filter is explicitly specified. Selecting this option displays the dynamic parameter LocalFilter. Use this parameter to specify filter criteria for local files. The criteria is specified using a FLEE expression. You can use the following elements from the System.IO FileInfo class: CreationTime, Directory, DirectoryName, Extension, FullName, IsReadOnly, LastAccessTime, LastWriteTime, Length, and Name. The following additional .NET Framework elements can also be used in FLEE expressions for filtering: DateTime, Now and Regex.
For example, to get all zip files from the past 5 days use the following expression: LastWriteTime > Now.AddDays(-5) and Name.EndsWith(".zip")
Select an existing user-defined variable, or click <New variable...> to create a new variable.
Related Topics: Integration Services Variables, Add Variable
Select an existing Amazon S3 connection manager, or click <New connection...> to create a new connection manager.
Related Topics: Amazon S3 Connection Manager
Select a bucket from your Amazon S3 account.
DEPRECATED: No longer used. The bucket is now part of Remote parameter.Indicate whether or not the remote name is stored in a variable. This parameter has the options listed in the following table.
Option Description True The remote name is stored in a variable. Selecting this option displays the dynamic parameter RemoteVariable. False The remote name is specified in a File connection manager. Selecting this option displays the dynamic parameter Remote. The syntax for specifying the Remote parameter is
bucket1/rootFolder1/subFolder1/for a folder andbucket1/rootFolder1/subFolder1/file1.txtfor a file.Select an existing user-defined variable, or click <New variable...> to create a new variable.
Related Topics: Integration Services Variables, Add Variable
Select an existing user-defined variable, or click <New variable...> to create a new variable.
Related Topics: Integration Services Variables, Add Variable
Indicate whether or not ResultVariable contains an array of file names or extended file information. This parameter has the options listed in the following table.
Options Description True The ResultVariable contains an array of IFileInfo interface objects. False The ResultVariable contains an array of file names. Not visible in the task user interface.Check the following script showing how to access ExtendedRemoteFileInfo ResultVariable (you must reference the CozyRoc.SSIS library in your script project):
Indicate whether or not the remote path is a wildcard. This property has the options listed in the following table.
Options Description True The remote path is a wildcard. False The remote path is an explicit specification. Not visible in the task user interface.Specify if the task should fail when the specified remote files don't exist. This parameter has the options listed in the following table.
Options Description True The task fails if the specified remote files don't exist. False The task ignores and completes successfully if the specified remote files don't exist. Not visible in the task user interface.Specify if remote files are recursively included. This parameter has the options listed in the following table.
Options Description True The task includes all remote files recursively, starting from the specified remote path. False The task includes only the specified remote path files. This is the default option. Indicate whether or not the remote filter is stored in a variable. This parameter has the options listed in the following table.
Options Description True The remote filter is stored in a variable. Selecting this option displays the dynamic parameter RemoteFilterVariable. False The remote filter is explicitly specified. Selecting this option displays the dynamic parameter RemoteFilter. Use this parameter to specify filter criteria for remote files. The criteria is specified using a FLEE expression. You can use the following elements from the IFileInfo interface: Name, FullName, Size, ModifiedTime, and IsDirectory. The following additional .NET Framework elements can also be used in FLEE expressions for filtering: DateTime, Now and Regex.
An example of an expression that can be used to get all zip files from the past 5 days is as follows: Modifiedtime > Now.AddDays(-5) and Name.EndsWith(".zip")
Select an existing user-defined variable, or click <New variable...> to create a new variable.
Related Topics: Integration Services Variables, Add Variable
Knowledge Base
- Error Message: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.
- Error Message: Access Denied when using Amazon S3 Task
- Where can I find the documentation for the Amazon S3 Task?
- Error Message: The authorization mechanism you have provided is not supported. Please use AWS4-HMAC-SHA256
- Where do I enter the bucket name in the Amazon S3 Task
Related scripts
What's New
- New: The result returned from 'Get remote files list' action can now be iterated using the standard 'Foreach ADO Enumerator'.
- Fixed: Failed with error "Index was out of range. Must be non-negative and less than the size of the collection." (Thank you, Carl).
- New: A new ExtendedRemoteFileInfo parameter to retrieve extended file information: size, last modified, etc.
- New: A new 'Rename remote file' action.
- New: A new 'Create remote directory' action.
- New: A new 'Remove remote directory' action.
- New: A new parameter RemoteFilter to set remote file filter expression when using ‘Receive files’, ‘Remove remote files’, ‘Get remote files list’ actions.
- New: A new parameter LocalFilter to set local file filter expression when using ‘Send files’ action.
- Fixed: Failed to process files containing space in name (Thank you, Steve).
- Fixed: 'Get remote files list' action returned only the first 1000 files in a bucket (Thank you, Bob).
- Fixed: Failed to process correctly wildcard specified source.
- New: Introduced task.
Related documentation
COZYROC SSIS+ Components Suite is free for testing in your development environment.
A licensed version can be deployed on-premises, on Azure-SSIS IR and on COZYROC Cloud.