Administer & Use S3 Browser Pro - Processing

Although you can leverage any file transfer application such as S3 Browser Pro, FileZilla, etc. to move your data, Reveal Processing recommends using S3 Browser Pro to streamline the data transfer process with direct import from S3 storage. This will be the preferred method for importing data into the processing environment. Using S3 Browser Pro requires some initial setup and configuration. Please use the instructions and link below to setup access to your S3 storage and configure an external S3 bucket for your data.

Note

To use S3 Browser Pro, you must purchase a license from Amazon. The free version of the S3 Browser is designed for personal use only.

S3 Browser Pro Setup

Amazon provides a Simple Storage Service ("S3") which Reveal utilizes heavily with a variety of our operations. A third-party client utility, S3 Browser, provides mechanisms to upload to and download content from S3 with a user-friendly interface. Due to its advanced functionality, including parallel uploads and downloads, hashed uploads, and filtering tools, Reveal recommends the use of the S3 Browser.

You can download S3 Browser at: https://s3browser.com/buypro.aspx

To configure an S3 Browser account:

1. Accounts -> New Account.

2. Account name can be whatever you like.

3. Account type is S3 storage.

4. Paste your access key and secret access key into the appropriate fields.

5. Ensure that Use secure transfer is selected.

6. Click Add New Account. You should see the following prompt: 

7. Answer "Yes".

8. Input the Bucket name below when prompted, where the 8-digit “XXXXXXXX” will be replaced by an MSA number provided by Reveal, and click the "Add External bucket" button:
   

   Note

   During initial S3 Browser setup, you may be prompted a second time to add an External Bucket. This second time you should answer No; answering ‘Yes’ to the second prompt will result in an error.

9. If you receive an error after adding the external bucket, close S3 Browser and re-open.

Note

In order to use advanced features such as parallel upload – the ability to set multiple threads for transfer -- you must purchase and register S3 Browser:

1. Purchase a license at https://s3browser.com/buypro.aspx. This license costs $39.99 and can be activated on three workstations.

2. Help -> Activate Pro Version.

3. Paste your activation code into the box.

Reveal strongly recommends using S3 Browser Pro to handle project-level volumes of data.

Configuring S3 Browser Pro

To configure advanced S3 Browser options:

1. From the S3 Browser menu, navigate to Tools → Options → General.

2. Enable multi-part uploads with a part size of: 100MB.
   ​​Disable​ multi-part downloads​​​: this has been found to be slower as it downloads in parts, then stitches the file back together in a single-threaded fashion, which may take at least twice as long as downloading with a single part.  
   

   Note

   Change values to 512 when uploading more than 1 TB.  

3. Switch to Data Integrity tab.

4. Ensure Perform data integrity check when uploading is unchecked.  

5. Switch to Queueing tab.

6. Switch Maximum number of concurrent uploads and Maximum number of other concurrent tasks to ​8​ per the screen recommendation. This may need to be adjusted based upon your bandwidth and load. (Maximum number of concurrent downloads is here set to ​1​​ with multipart downloads disabled.)
    
   

   Note

   Changing to queuing values greater than 2 requires the Pro version.

7. Click Save changes.

Note

Server-side encryption is automatically applied through S3 bucket policies, and no longer has to be configured manually.

Adding an External Bucket

To create a additional buckets:

1. Go to S3 Browser's Buckets menu.

2. Select Add External Bucket.
    

3. Enter the name of the bucket that has been supplied by Reveal. If that information cannot be easily found, the bucket name is always ‘revealdata-s3staging-MSA’ of the environment. Users can find the 8-digit MSA number in the title of the browser when accessing the Load Machine through landing.revealdata.com. If the MSA was 99999999, the external bucket would be ‘revealdata-s3staging-99999999’ as seen in the below image.
    

4. Click Add External bucket.

5. The new bucket and anything already in it will appear in your bucket list.

Q&A

Q: Do I need to upgrade to the Pro version?

A: Yes. The free version is limited to a maximum of two threads for upload or download, which is far too limited for even average discovery datasets. There are additional performance enhancements in the Pro version.

Q: I licensed the Pro version over a year ago. Do I need to renew?

A: Yes. License renewal is required to upgrade to the latest version.

Q: What version of S3 is required by Reveal?

A: Reveal requires S3 version 10.3.x or above.

Q: What are Reveal’s policies for S3 data retention?

A: There are different user-defined retention polices placed on the Persistent and Temporary object storage locations. These are governed by the S3 Object Lock retention modes to enforce retention policies, governance, and compliance regulations. We also use this as a second layer of fall back should something get deleted on accident. Details are as follows:

• Persistent Folder: There is a 7-day retention period placed on an object once you have executed a delete action. If you have the appropriate permissions you can delete, move, copy, these objects. The objects in this folder are tied to Reveal Processing and Reveal Review. If you were to delete an object from here, it will be retained for 7-days until the retention time period has been met.

• Temporary Folder: There is a blanket 30-day retention policy set for objects in this folder. Any object moved into this folder is automatically deleted after the 30-day time period has been reached for the object.

Q: Are S3 access keys issued to individuals or can they be shared by a team?

A: Access keys are shared among the users. They will all have the same access and secret keys. If one person’s S3 Browser is working and someone else’s is not, then the second user is probably missing something and should re-enter the keys.

Please contact [email protected] with any additional questions you may have.