Upload files to AWS S3 programmatically— Java

Before beginning, please create an AWS account, and get your access_key_id and secret_access_key. Also create a bucket named “test-dir-1313‘ and make sure to unselect block all permissions when creating s3 so that it has public access.

Below is the program to upload files from your local computer to AWS S3 bucket. I have added inline comments as well to make it easier for you to understand this!

We have made a directory in our local computer, which consists of all the files that needs to be uploaded. As we are uploading multiple files, if it is synchronous i.e. if it uploads files one by one, then the process would be slow and hence we are making asynchronous calls and not waiting for an upload to be completed. Instead after every call to putObject(), we return a CompletableFuture which keeps track of the request being completed.

CompletableFuture is a Java8 feature which lets us make asynchronous calls very effectively, giving us all the options to keep a track of request without blocking any operation.

package com.mycompany.app;

import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.auth.credentials.StaticCredentialsProvider;
import software.amazon.awssdk.core.async.AsyncRequestBody;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3AsyncClient;
import software.amazon.awssdk.services.s3.model.ObjectCannedACL;
import software.amazon.awssdk.services.s3.model.PutObjectRequest;

import java.io.File;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.ArrayList;
import java.util.List;
import java.util.UUID;
import java.util.concurrent.CompletableFuture;
import java.util.concurrent.ExecutionException;
import java.util.stream.Collectors;

public class S3POCMultipleFiles {

    static S3AsyncClient client;
    static String bucketName = "test-dir-1313"; // bucket name

    public static void main(String[] args) throws ExecutionException, InterruptedException {
        // getting all file names in a directory
        File folder = new File("C:\\NWEA\\test-dir");
        File[] listOfFiles = folder.listFiles();
        List<String> list= new ArrayList<>();
        for (int i = 0; i < listOfFiles.length; i++) {
            if (listOfFiles[i].isFile()) 
        // storing all absolute path file in an ArrayList
        list.add(listOfFiles[i].getAbsolutePath());
                System.out.println("File " + listOfFiles[i].getName());
            } else if (listOfFiles[i].isDirectory()) {
                System.out.println("Directory " + listOfFiles[i].getName());
            }
        }
       // pass your keys which you get from AWS
        AwsBasicCredentials awsCreds = AwsBasicCredentials.create(
                "access_key_id",
                "secret_access_key");

        String path = String.format("%s/%s", "", UUID.randomUUID());


        Region region = Region.US_WEST_2; // selecting a region
        client = S3AsyncClient.builder()
         .credentialsProvider(StaticCredentialsProvider.create(awsCreds))
                .region(region)
                .build();


        //Created a stream of files, and calling write on each file
        List<CompletableFuture> futures = list.stream().map(S3POCMultipleFiles::write).collect(Collectors.toList());
        System.out.println("The put operation begun above");
        System.out.println("Operation didn't stop");
        System.out.println("We are able to see this output");
        System.out.println("which suggests this is async");
        System.out.println("Thanks to Completable future");
        // allOf helps to call all the requests parallely at once
        CompletableFuture.allOf(futures.toArray( new CompletableFuture[futures.size()]))
                .whenComplete((res,err) -> {})
                .get();
        System.out.println("I will only print");
        System.out.println("Once all future is resolved");
        System.out.println("remove get if you want to make this async as well");
        System.out.println("However do not remove async");
        System.out.println("The data returned by future could be important for me");

    }

    public static CompletableFuture<Path> write(final String name) {
        Path file = Paths.get(name);
        PutObjectRequest request = PutObjectRequest.builder()
                .bucket(bucketName)
                .key(file.getFileName().toString())
                .acl(ObjectCannedACL.BUCKET_OWNER_FULL_CONTROL)
                .build();
        return client.putObject(request, AsyncRequestBody.fromFile(file.toFile()))
                .thenApply(response -> {
                    return file;
                }).exceptionally(err -> {
                    return null;
                });
    }
}

If you are using maven as the dependency manager, then make sure to include these in your pom.xml file :

<dependencies>
  <dependency>
    <groupId>software.amazon.awssdk</groupId>
    <artifactId>s3</artifactId>
  </dependency>
</dependencies>
<dependencyManagement>
  <dependencies>
    <dependency>
      <groupId>software.amazon.awssdk</groupId>
      <artifactId>bom</artifactId>
      <version>2.15.23</version>
      <type>pom</type>
      <scope>import</scope>
    </dependency>
  </dependencies>
</dependencyManagement>

That’s it.

You can check the AWS official docs : https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/asynchronous.html