The Simple Storage Service (S3) protocol is a standard interface developed by Amazon Web Services (AWS) for storing and retrieving data in the cloud. It’s a key-value store, meaning that each piece of data (an object) is associated with a unique identifier (a key) that you use to access that data. The S3 protocol is crucial in cloud storage services for several reasons: ScalabilityS3 allows for virtually unlimited storage, making it ideal for applications that need to scale rapidly or handle large amounts of data.
Durability and AvailabilityS3 is designed for 99.999999999% (11 9’s) of durability, meaning the chances of losing data are extremely low. It also guarantees 99.99% availability, ensuring that your data is accessible when you need it.
SecurityS3 provides robust security features, including encryption for data at rest and in transit, access control policies, and logging capabilities.
InteroperabilityMany cloud storage providers, not just AWS, support the S3 protocol, making it a versatile choice for developers.
Cost-EffectiveWith S3, you only pay for the storage you use, making it a cost-effective solution for businesses of all sizes. In summary, the S3 protocol is a reliable, secure, and scalable solution for storing data in the cloud, making it a cornerstone of modern cloud-based applications. Setting Up Go Environment install AWS SDK into our project go get github.com/aws/aws-sdk-go-v2/aws
go get github.com/aws/aws-sdk-go-v2/config
go get github.com/aws/aws-sdk-go-v2/credentials
go get github.com/aws/aws-sdk-go-v2/feature/s3/manager
go get github.com/aws/aws-sdk-go-v2/service/s3 Simple project to upload file AWS SDK Wrapper package s3client

import (
 "context"
 "github.com/aws/aws-sdk-go-v2/aws"
 "github.com/aws/aws-sdk-go-v2/config"
 "github.com/aws/aws-sdk-go-v2/credentials"
 "github.com/aws/aws-sdk-go-v2/feature/s3/manager"
 "github.com/aws/aws-sdk-go-v2/service/s3"
 "io"
 "log"
 "sync"
)

var (
 uploader *manager.Uploader
 once     sync.Once
)

func setup() {
 endpoint := "http://s3-test-endpoint.com"
 accessKey := "XVXQ8Q3QGQ9QXQ8QXQ8Q"
 secretKey := "/XQ8Q3QGQ9QXQ8QXQ8QXQ/Q3QGQ9QXQ8QXQ8QXQ8"

 resolver := aws.EndpointResolverWithOptionsFunc(func(service, region string, options ...interface{}) (aws.Endpoint, error) {
  return aws.Endpoint{
   URL:               endpoint,
   HostnameImmutable: true,
   Source:            aws.EndpointSourceCustom,
  }, nil
 })
 awsConfig, err := config.LoadDefaultConfig(context.TODO(),
  config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider(accessKey, secretKey, "")),
  config.WithEndpointResolverWithOptions(resolver),
  config.WithRegion("auto"),
 )
 if err != nil {
  log.Fatalf("failed to load config, %v", err)
 }

 uploader = manager.NewUploader(s3.NewFromConfig(awsConfig)) // Initialize S3 client
}

func UploadFile(ctx context.Context, bucket, filename string, file io.Reader) (string, error) {
 once.Do(setup)
 result, err := uploader.Upload(ctx, &s3.PutObjectInput{
  Bucket: aws.String(bucket),
  Key:    aws.String(filename),
  Body:   file,
  ACL:    "public-read",
 })
 if err != nil {
  log.Fatalf("failed to upload file, %v", err)
 }

 return result.Location, nil
} Why we need to wrap third-party library? The purpose wrapper is to simplify the library’s interface, adapt it to fit our application’s specific needs, or add additional functionality. In this case we simplifies the process of uploading a file to S3 by providing a single UploadFile function that handles the setup and upload process. This makes it easier for the rest of our application to upload files to S3, as it only needs to call s3client.UploadFile with the appropriate parameters. Anyway, let’s explain the code a little bit We need to setup our project to connect with the cloud storage services, in the setupfunction there is 2 key steps that we need to highlight. First, in resolver parameter we need to pass aws.EndpointSourceCustomto indicate we didn’t connect to AWS endpoint. Secondly,config.WithRegion("auto")when load default config. Then in UploadFile function we use once.Do(setup) to make sure setup function only called once. Main Function We will create simple server using chi, then create POST API to upload file to our S3 Cloud Storage. Also Hello World endpoint to make sure our server is working 😆. package main

import (
 "github.com/go-chi/chi/v5"
 "github.com/yudaph/s3-integration/s3client"
 "net/http"
)

func main() {
 r := chi.NewRouter()

 r.Get("/", func(w http.ResponseWriter, r *http.Request) {
  w.Write([]byte("Hello, World!"))
 })

 r.Post("/", HandleUpload)
 http.ListenAndServe(":8080", r)
}

func HandleUpload(w http.ResponseWriter, r *http.Request) {
 file, header, err := r.FormFile("image")
 if err != nil {
  http.Error(w, err.Error(), http.StatusBadRequest)
  return
 }
 defer file.Close()

 // TODO: Do Validation

 // Upload file to S3
 filename := "suffix_" + header.Filename
 url, err := s3client.UploadFile(r.Context(), "bucket-name", filename, file)
 if err != nil {
  http.Error(w, err.Error(), http.StatusInternalServerError)
  return
 }

 w.Write([]byte(url))
} Let’s explain the HandleUploadfunction: We will use r.FormFile("image") to get file from request, in other framework like mux and fiber we only get the header, so if we need to use header.Open() to get the file from header(CMIIW). Then do some validation, after that we simply call our wrapper function s3client.UploadFile to upload to our cloud storage. Conclusion In this article we learn about S3 protocol, how to Integrate with Non-AWS Cloud Storage Service that supports S3 protocol, and a little bit about wrapper. Hopefully, this article can be helpful. Don’t hesitate to give criticism and suggestions if you find any mistakes in this article. Let’s connect on https://www.linkedin.com/in/yudaph/ and https://github.com/yudaph Note: artikel berbahasa indonesia dapat ditemukan di https://medium.com/@yudaph/panduan-untuk-mengintegrasikan-dengan-layanan-cloud-storage-non-aws-yang-mendukung-protokol-s3-20624638911a The Simple Storage Service (S3) protocol is a standard interface developed by Amazon Web Services (AWS) for storing and retrieving data in the cloud. It’s a key-value store, meaning that each piece of data (an object) is associated with a unique identifier (a key) that you use to access that data. The S3 protocol is crucial in cloud storage services for several reasons: ScalabilityS3 allows for virtually unlimited storage, making it ideal for applications that need to scale rapidly or handle large amounts of data. Durability and AvailabilityS3 is designed for 99.999999999% (11 9’s) of durability, meaning the chances of losing data are extremely low. It also guarantees 99.99% availability, ensuring that your data is accessible when you need it. SecurityS3 provides robust security features, including encryption for data at rest and in transit, access control policies, and logging capabilities. InteroperabilityMany cloud storage providers, not just AWS, support the S3 protocol, making it a versatile choice for developers. Cost-EffectiveWith S3, you only pay for the storage you use, making it a cost-effective solution for businesses of all sizes. Scalability S3 allows for virtually unlimited storage, making it ideal for applications that need to scale rapidly or handle large amounts of data. Durability and Availability S3 is designed for 99.999999999% (11 9’s) of durability, meaning the chances of losing data are extremely low. It also guarantees 99.99% availability, ensuring that your data is accessible when you need it. Security S3 provides robust security features, including encryption for data at rest and in transit, access control policies, and logging capabilities. Interoperability Many cloud storage providers, not just AWS, support the S3 protocol, making it a versatile choice for developers. Cost-Effective With S3, you only pay for the storage you use, making it a cost-effective solution for businesses of all sizes. In summary, the S3 protocol is a reliable, secure, and scalable solution for storing data in the cloud, making it a cornerstone of modern cloud-based applications. Setting Up Go Environment install AWS SDK into our project go get github.com/aws/aws-sdk-go-v2/aws
go get github.com/aws/aws-sdk-go-v2/config
go get github.com/aws/aws-sdk-go-v2/credentials
go get github.com/aws/aws-sdk-go-v2/feature/s3/manager
go get github.com/aws/aws-sdk-go-v2/service/s3 go get github.com/aws/aws-sdk-go-v2/aws
go get github.com/aws/aws-sdk-go-v2/config
go get github.com/aws/aws-sdk-go-v2/credentials
go get github.com/aws/aws-sdk-go-v2/feature/s3/manager
go get github.com/aws/aws-sdk-go-v2/service/s3 Simple project to upload file AWS SDK Wrapper package s3client

import (
 "context"
 "github.com/aws/aws-sdk-go-v2/aws"
 "github.com/aws/aws-sdk-go-v2/config"
 "github.com/aws/aws-sdk-go-v2/credentials"
 "github.com/aws/aws-sdk-go-v2/feature/s3/manager"
 "github.com/aws/aws-sdk-go-v2/service/s3"
 "io"
 "log"
 "sync"
)

var (
 uploader *manager.Uploader
 once     sync.Once
)

func setup() {
 endpoint := "http://s3-test-endpoint.com"
 accessKey := "XVXQ8Q3QGQ9QXQ8QXQ8Q"
 secretKey := "/XQ8Q3QGQ9QXQ8QXQ8QXQ/Q3QGQ9QXQ8QXQ8QXQ8"

 resolver := aws.EndpointResolverWithOptionsFunc(func(service, region string, options ...interface{}) (aws.Endpoint, error) {
  return aws.Endpoint{
   URL:               endpoint,
   HostnameImmutable: true,
   Source:            aws.EndpointSourceCustom,
  }, nil
 })
 awsConfig, err := config.LoadDefaultConfig(context.TODO(),
  config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider(accessKey, secretKey, "")),
  config.WithEndpointResolverWithOptions(resolver),
  config.WithRegion("auto"),
 )
 if err != nil {
  log.Fatalf("failed to load config, %v", err)
 }

 uploader = manager.NewUploader(s3.NewFromConfig(awsConfig)) // Initialize S3 client
}

func UploadFile(ctx context.Context, bucket, filename string, file io.Reader) (string, error) {
 once.Do(setup)
 result, err := uploader.Upload(ctx, &s3.PutObjectInput{
  Bucket: aws.String(bucket),
  Key:    aws.String(filename),
  Body:   file,
  ACL:    "public-read",
 })
 if err != nil {
  log.Fatalf("failed to upload file, %v", err)
 }

 return result.Location, nil
} package s3client

import (
 "context"
 "github.com/aws/aws-sdk-go-v2/aws"
 "github.com/aws/aws-sdk-go-v2/config"
 "github.com/aws/aws-sdk-go-v2/credentials"
 "github.com/aws/aws-sdk-go-v2/feature/s3/manager"
 "github.com/aws/aws-sdk-go-v2/service/s3"
 "io"
 "log"
 "sync"
)

var (
 uploader *manager.Uploader
 once     sync.Once
)

func setup() {
 endpoint := "http://s3-test-endpoint.com"
 accessKey := "XVXQ8Q3QGQ9QXQ8QXQ8Q"
 secretKey := "/XQ8Q3QGQ9QXQ8QXQ8QXQ/Q3QGQ9QXQ8QXQ8QXQ8"

 resolver := aws.EndpointResolverWithOptionsFunc(func(service, region string, options ...interface{}) (aws.Endpoint, error) {
  return aws.Endpoint{
   URL:               endpoint,
   HostnameImmutable: true,
   Source:            aws.EndpointSourceCustom,
  }, nil
 })
 awsConfig, err := config.LoadDefaultConfig(context.TODO(),
  config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider(accessKey, secretKey, "")),
  config.WithEndpointResolverWithOptions(resolver),
  config.WithRegion("auto"),
 )
 if err != nil {
  log.Fatalf("failed to load config, %v", err)
 }

 uploader = manager.NewUploader(s3.NewFromConfig(awsConfig)) // Initialize S3 client
}

func UploadFile(ctx context.Context, bucket, filename string, file io.Reader) (string, error) {
 once.Do(setup)
 result, err := uploader.Upload(ctx, &s3.PutObjectInput{
  Bucket: aws.String(bucket),
  Key:    aws.String(filename),
  Body:   file,
  ACL:    "public-read",
 })
 if err != nil {
  log.Fatalf("failed to upload file, %v", err)
 }

 return result.Location, nil
} Why we need to wrap third-party library? The purpose wrapper is to simplify the library’s interface, adapt it to fit our application’s specific needs, or add additional functionality. In this case we simplifies the process of uploading a file to S3 by providing a single UploadFile function that handles the setup and upload process. This makes it easier for the rest of our application to upload files to S3, as it only needs to call s3client.UploadFile with the appropriate parameters. UploadFile s3client.UploadFile Anyway, let’s explain the code a little bit Anyway, let’s explain the code a little bit We need to setup our project to connect with the cloud storage services, in the setup function there is 2 key steps that we need to highlight. setup First, in resolver parameter we need to pass aws.EndpointSourceCustom to indicate we didn’t connect to AWS endpoint. resolver aws.EndpointSourceCustom Secondly, config.WithRegion("auto") when load default config. config.WithRegion("auto") Then in UploadFile function we use once.Do(setup) to make sure setup function only called once. UploadFile once.Do(setup) Main Function We will create simple server using chi , then create POST API to upload file to our S3 Cloud Storage. Also Hello World endpoint to make sure our server is working 😆. chi POST package main

import (
 "github.com/go-chi/chi/v5"
 "github.com/yudaph/s3-integration/s3client"
 "net/http"
)

func main() {
 r := chi.NewRouter()

 r.Get("/", func(w http.ResponseWriter, r *http.Request) {
  w.Write([]byte("Hello, World!"))
 })

 r.Post("/", HandleUpload)
 http.ListenAndServe(":8080", r)
}

func HandleUpload(w http.ResponseWriter, r *http.Request) {
 file, header, err := r.FormFile("image")
 if err != nil {
  http.Error(w, err.Error(), http.StatusBadRequest)
  return
 }
 defer file.Close()

 // TODO: Do Validation

 // Upload file to S3
 filename := "suffix_" + header.Filename
 url, err := s3client.UploadFile(r.Context(), "bucket-name", filename, file)
 if err != nil {
  http.Error(w, err.Error(), http.StatusInternalServerError)
  return
 }

 w.Write([]byte(url))
} package main

import (
 "github.com/go-chi/chi/v5"
 "github.com/yudaph/s3-integration/s3client"
 "net/http"
)

func main() {
 r := chi.NewRouter()

 r.Get("/", func(w http.ResponseWriter, r *http.Request) {
  w.Write([]byte("Hello, World!"))
 })

 r.Post("/", HandleUpload)
 http.ListenAndServe(":8080", r)
}

func HandleUpload(w http.ResponseWriter, r *http.Request) {
 file, header, err := r.FormFile("image")
 if err != nil {
  http.Error(w, err.Error(), http.StatusBadRequest)
  return
 }
 defer file.Close()

 // TODO: Do Validation

 // Upload file to S3
 filename := "suffix_" + header.Filename
 url, err := s3client.UploadFile(r.Context(), "bucket-name", filename, file)
 if err != nil {
  http.Error(w, err.Error(), http.StatusInternalServerError)
  return
 }

 w.Write([]byte(url))
} Let’s explain the HandleUpload function: HandleUpload We will use r.FormFile("image") to get file from request, in other framework like mux and fiber we only get the header , so if we need to use header.Open() to get the file from header (CMIIW). r.FormFile("image") mux fiber header header.Open() file header Then do some validation, after that we simply call our wrapper function s3client.UploadFile to upload to our cloud storage. s3client.UploadFile Conclusion In this article we learn about S3 protocol, how to Integrate with Non-AWS Cloud Storage Service that supports S3 protocol, and a little bit about wrapper. Hopefully, this article can be helpful. Don’t hesitate to give criticism and suggestions if you find any mistakes in this article. Let’s connect on https://www.linkedin.com/in/yudaph/ and https://github.com/yudaph https://www.linkedin.com/in/yudaph/ https://github.com/yudaph Note: artikel berbahasa indonesia dapat ditemukan di https://medium.com/@yudaph/panduan-untuk-mengintegrasikan-dengan-layanan-cloud-storage-non-aws-yang-mendukung-protokol-s3-20624638911a https://medium.com/@yudaph/panduan-untuk-mengintegrasikan-dengan-layanan-cloud-storage-non-aws-yang-mendukung-protokol-s3-20624638911a

Too Long; Didn't Read

Making security fun one episode at a time

Guide to Integrating With Non-AWS Cloud Storage Services Supporting the S3 Protocol using Golang

Too Long; Didn't Read

Yuda Prasetiya Haqqy

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

Guide to Integrating With Non-AWS Cloud Storage Services Supporting the S3 Protocol using Golang

Too Long; Didn't Read

Yuda Prasetiya Haqqy

About Author

TOPICS

THIS ARTICLE WAS FEATURED IN...

RELATED STORIES