diff --git a/README.md b/README.md index 3ea6c1d..477dce0 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,5 @@ [hex-img]: http://img.shields.io/hexpm/v/waffle.svg - [hexdocs-img]: http://img.shields.io/badge/hexdocs-documentation-brightgreen.svg - [evrone-img]: https://img.shields.io/badge/Sponsored_by-Evrone-brightgreen.svg # Waffle [![Sponsored by Evrone][evrone-img]](https://evrone.com?utm_source=waffle) @@ -11,14 +9,14 @@ ![Waffle is a flexible file upload library for Elixir](https://elixir-waffle.github.io/waffle/assets/logo.svg) -Waffle is a flexible file upload library for Elixir with straightforward integrations for Amazon S3 and ImageMagick. +Waffle is a flexible file upload library for Elixir with straightforward integrations for Amazon S3, Azure Blob Storage, and ImageMagick. [Documentation](https://hexdocs.pm/waffle) ## Installation Add the latest stable release to your `mix.exs` file, along with the -required dependencies for `ExAws` if appropriate: +required dependencies for your chosen storage provider: ```elixir defp deps do @@ -29,7 +27,11 @@ defp deps do {:ex_aws, "~> 2.1.2"}, {:ex_aws_s3, "~> 2.0"}, {:hackney, "~> 1.9"}, - {:sweet_xml, "~> 0.6"} + {:sweet_xml, "~> 0.6"}, + + # If using Azure Blob Storage: + {:req, "~> 0.4"}, + {:timex, "~> 3.7"} ] end ``` @@ -45,10 +47,11 @@ After installing Waffle, another two things should be done: ### Setup a storage provider -Waffle has two built-in storage providers: +Waffle has three built-in storage providers: -* `Waffle.Storage.Local` -* `Waffle.Storage.S3` +- `Waffle.Storage.Local` +- `Waffle.Storage.S3` +- `Waffle.Storage.Azure` [Other available storage providers](#other-storage-providers) are supported by the community. @@ -74,16 +77,28 @@ config :ex_aws, # any configurations provided by https://github.com/ex-aws/ex_aws ``` +An example for setting up `Waffle.Storage.Azure`: + +```elixir +config :waffle, + storage: Waffle.Storage.Azure, + storage_account: "mystorageaccount", # or {:system, "AZURE_STORAGE_ACCOUNT"} + container: "uploads", # or {:system, "AZURE_STORAGE_CONTAINER"} + access_key: "your-access-key", # or {:system, "AZURE_ACCESS_KEY"} + public_access: false, # Set to true for public access + expiry_in_minutes: 60 # SAS token expiry for private access +``` + ### Define a definition module Waffle requires a **definition module** which contains the relevant functions to store and retrieve files: -* Optional transformations of the uploaded file -* Where to put your files (the storage directory) -* How to name your files -* How to secure your files (private? Or publicly accessible?) -* Default placeholders +- Optional transformations of the uploaded file +- Where to put your files (the storage directory) +- How to name your files +- How to secure your files (private? Or publicly accessible?) +- Default placeholders This module can be created manually or generated by `mix waffle.g` automatically. @@ -98,8 +113,9 @@ Check this file for descriptions of configurable options. ## Examples -* [An example for Local storage driver](documentation/examples/local.md) -* [An example for S3 storage driver](documentation/examples/s3.md) +- [An example for Local storage driver](documentation/examples/local.md) +- [An example for S3 storage driver](documentation/examples/s3.md) +- [An example for Azure Blob Storage driver](documentation/examples/azure.md) ## Usage with Ecto @@ -107,25 +123,25 @@ Waffle comes with a companion package for use with Ecto. If you intend to use Waffle with Ecto, it is highly recommended you also add the [`waffle_ecto`](https://github.com/elixir-waffle/waffle_ecto) -dependency. Benefits include: +dependency. Benefits include: - * Changeset integration - * Versioned urls for cache busting (`.../thumb.png?v=63601457477`) +- Changeset integration +- Versioned urls for cache busting (`.../thumb.png?v=63601457477`) ## Other Storage Providers - * **Rackspace** - [arc_rackspace](https://github.com/lokalebasen/arc_rackspace) +- **Rackspace** - [arc_rackspace](https://github.com/lokalebasen/arc_rackspace) + +- **Manta** - [arc_manta](https://github.com/onyxrev/arc_manta) - * **Manta** - [arc_manta](https://github.com/onyxrev/arc_manta) +- **OVH** - [arc_ovh](https://github.com/stephenmoloney/arc_ovh) - * **OVH** - [arc_ovh](https://github.com/stephenmoloney/arc_ovh) +- **Google Cloud Storage** - [waffle_gcs](https://github.com/elixir-waffle/waffle_gcs) - * **Google Cloud Storage** - [waffle_gcs](https://github.com/elixir-waffle/waffle_gcs) +- **Microsoft Azure Storage** - Built-in `Waffle.Storage.Azure` or [arc_azure](https://github.com/phil-a/arc_azure) - * **Microsoft Azure Storage** - [arc_azure](https://github.com/phil-a/arc_azure) +- **Aliyun OSS Storage** - [waffle_aliyun_oss](https://github.com/ug0/waffle_aliyun_oss) - * **Aliyun OSS Storage** - [waffle_aliyun_oss](https://github.com/ug0/waffle_aliyun_oss) - ## Testing The basic test suite can be run with without supplying any S3 information: @@ -140,13 +156,21 @@ access. The following environment variables will be used by the test suite: -* WAFFLE_TEST_BUCKET -* WAFFLE_TEST_BUCKET2 -* WAFFLE_TEST_S3_KEY -* WAFFLE_TEST_S3_SECRET -* WAFFLE_TEST_REGION +**For S3 testing:** + +- WAFFLE_TEST_BUCKET +- WAFFLE_TEST_BUCKET2 +- WAFFLE_TEST_S3_KEY +- WAFFLE_TEST_S3_SECRET +- WAFFLE_TEST_REGION + +**For Azure testing:** + +- AZURE_STORAGE_ACCOUNT +- AZURE_STORAGE_CONTAINER +- AZURE_ACCESS_KEY -After setting these variables, you can run the full test suite with `mix test --include s3:true`. +After setting these variables, you can run the full test suite with `mix test --include s3:true` or `mix test --include azure:true`. ## Attribution @@ -160,14 +184,14 @@ Copyright 2019 Boris Kuznetsov Copyright 2015 Sean Stavropoulos - Licensed under the Apache License, Version 2.0 (the "License"); - you may not use this file except in compliance with the License. - You may obtain a copy of the License at +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 - Unless required by applicable law or agreed to in writing, software - distributed under the License is distributed on an "AS IS" BASIS, - WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - See the License for the specific language governing permissions and - limitations under the License. +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. diff --git a/documentation/examples/azure.md b/documentation/examples/azure.md new file mode 100644 index 0000000..b942ac0 --- /dev/null +++ b/documentation/examples/azure.md @@ -0,0 +1,194 @@ +# Azure Blob Storage + +This guide will help you set up Waffle to work with Azure Blob Storage. + +## Configuration + +Add the following to your `config/config.exs`: + +```elixir +config :waffle, + storage: Waffle.Storage.Azure, + storage_account: {:system, "AZURE_STORAGE_ACCOUNT"}, + container: {:system, "AZURE_STORAGE_CONTAINER"}, + access_key: {:system, "AZURE_ACCESS_KEY"}, + public_access: false, + expiry_in_minutes: 60 +``` + +Or with direct values: + +```elixir +config :waffle, + storage: Waffle.Storage.Azure, + storage_account: "mystorageaccount", + container: "uploads", + access_key: "your-access-key", + public_access: false, + expiry_in_minutes: 60 +``` + +## Environment Variables + +Set the following environment variables: + +```bash +export AZURE_STORAGE_ACCOUNT="mystorageaccount" +export AZURE_STORAGE_CONTAINER="uploads" +export AZURE_ACCESS_KEY="your-access-key" +``` + +## Definition Module + +Create a definition module for your uploads: + +```elixir +defmodule MyApp.Avatar do + use Waffle.Definition + + @extension_whitelist ~w(.jpg .jpeg .gif .png) + + def validate({file, _}) do + file_extension = file.file_name |> Path.extname() |> String.downcase() + + case Enum.member?(@extension_whitelist, file_extension) do + true -> :ok + false -> {:error, "invalid file type"} + end + end + + def storage_dir(version, {file, scope}) do + "uploads/users/avatars/#{scope.id}" + end + + # Optional: Override container per definition + def container({_file, scope}), do: scope.container || container() + + # Optional: Override storage account per definition + def storage_account({_file, scope}), do: scope.storage_account || storage_account() + + # Optional: Custom Azure blob headers + def azure_blob_headers(version, {file, scope}) do + [content_type: MIME.from_path(file.file_name)] + end +end +``` + +## Usage + +### Storing Files + +```elixir +# Store a file +{:ok, file_name} = MyApp.Avatar.store({%Plug.Upload{path: "/tmp/avatar.jpg", filename: "avatar.jpg"}, user}) + +# Store with custom scope +{:ok, file_name} = MyApp.Avatar.store({%Plug.Upload{path: "/tmp/avatar.jpg", filename: "avatar.jpg"}, %{id: 123, container: "custom-container"}}) +``` + +### Generating URLs + +```elixir +# Generate public URL (if public_access is true) +url = MyApp.Avatar.url({file_name, user}) + +# Generate signed URL (if public_access is false) +url = MyApp.Avatar.url({file_name, user}, signed: true) + +# Generate signed URL with custom expiry +url = MyApp.Avatar.url({file_name, user}, signed: true, expires_in: 3600) # 1 hour +``` + +### Deleting Files + +```elixir +# Delete a file +:ok = MyApp.Avatar.delete({file_name, user}) +``` + +## Multiple Containers + +You can use different containers for different uploaders: + +```elixir +defmodule MyApp.Document do + use Waffle.Definition + + def container, do: "documents" +end + +defmodule MyApp.Image do + use Waffle.Definition + + def container, do: "images" +end +``` + +## Public vs Private Access + +### Public Access + +When `public_access` is set to `true`, files are accessible via direct URLs without authentication: + +```elixir +config :waffle, + public_access: true +``` + +### Private Access (Default) + +When `public_access` is `false` (default), files are accessed via signed URLs with SAS tokens: + +```elixir +config :waffle, + public_access: false, + expiry_in_minutes: 60 # SAS token expires in 60 minutes +``` + +## Custom Headers + +You can specify custom headers for Azure blob storage: + +```elixir +def azure_blob_headers(version, {file, scope}) do + [ + content_type: MIME.from_path(file.file_name), + cache_control: "public, max-age=31536000", + content_disposition: "inline; filename=\"#{file.file_name}\"" + ] +end +``` + +## Dependencies + +Make sure to add the required dependencies to your `mix.exs`: + +```elixir +defp deps do + [ + {:waffle, "~> 1.1"}, + {:req, "~> 0.4"}, + {:timex, "~> 3.7"} + ] +end +``` + +## Error Handling + +The Azure storage adapter will return appropriate error tuples: + +```elixir +case MyApp.Avatar.store({file, user}) do + {:ok, file_name} -> + # Success + {:error, reason} -> + # Handle error +end +``` + +Common error scenarios: + +- Invalid storage account or container +- Missing access key +- Network connectivity issues +- File read errors diff --git a/example_azure_config.exs b/example_azure_config.exs new file mode 100644 index 0000000..0457248 --- /dev/null +++ b/example_azure_config.exs @@ -0,0 +1,41 @@ +# Example configuration for Azure Blob Storage with Waffle +# Add this to your config/config.exs or config/prod.exs + +# Basic Azure configuration +config :waffle, + storage: Waffle.Storage.Azure, + storage_account: {:system, "AZURE_STORAGE_ACCOUNT"}, + container: {:system, "AZURE_STORAGE_CONTAINER"}, + access_key: {:system, "AZURE_ACCESS_KEY"}, + public_access: false, + expiry_in_minutes: 60 + +# Example definition module +defmodule Example.Avatar do + use Waffle.Definition + + @extension_whitelist ~w(.jpg .jpeg .gif .png) + + def validate({file, _}) do + file_extension = file.file_name |> Path.extname() |> String.downcase() + + case Enum.member?(@extension_whitelist, file_extension) do + true -> :ok + false -> {:error, "invalid file type"} + end + end + + def storage_dir(version, {file, scope}) do + "uploads/users/avatars/#{scope.id}" + end + + # Optional: Custom Azure blob headers + def azure_blob_headers(version, {file, scope}) do + [content_type: MIME.from_path(file.file_name)] + end +end + +# Example usage: +# {:ok, file_name} = Example.Avatar.store({%Plug.Upload{path: "/tmp/avatar.jpg", filename: "avatar.jpg"}, user}) +# url = Example.Avatar.url({file_name, user}, signed: true) +# :ok = Example.Avatar.delete({file_name, user}) diff --git a/lib/waffle/definition/storage.ex b/lib/waffle/definition/storage.ex index eac884a..52f27cf 100644 --- a/lib/waffle/definition/storage.ex +++ b/lib/waffle/definition/storage.ex @@ -54,6 +54,7 @@ defmodule Waffle.Definition.Storage do * `Waffle.Storage.Local` * `Waffle.Storage.S3` + * `Waffle.Storage.Azure` Override the `__storage` function in your definition module if you want to use a different type of storage for a particular uploader. @@ -116,6 +117,16 @@ defmodule Waffle.Definition.Storage do def bucket, do: Application.fetch_env!(:waffle, :bucket) def bucket({_file, _scope}), do: bucket() def asset_host, do: Application.get_env(:waffle, :asset_host) + + # Azure Blob Storage configuration + def storage_account, do: Application.fetch_env!(:waffle, :storage_account) + def storage_account({_file, _scope}), do: storage_account() + def container, do: Application.fetch_env!(:waffle, :container) + def container({_file, _scope}), do: container() + def access_key, do: Application.fetch_env!(:waffle, :access_key) + def access_key({_file, _scope}), do: access_key() + def public_access, do: Application.get_env(:waffle, :public_access, false) + def public_access(_, _), do: public_access() def filename(_, {file, _}), do: Path.basename(file.file_name, Path.extname(file.file_name)) def storage_dir_prefix, do: Application.get_env(:waffle, :storage_dir_prefix, "") def storage_dir(_, _), do: Application.get_env(:waffle, :storage_dir, "uploads") @@ -133,7 +144,15 @@ defmodule Waffle.Definition.Storage do __storage: 0, bucket: 0, bucket: 1, - asset_host: 0 + asset_host: 0, + storage_account: 0, + storage_account: 1, + container: 0, + container: 1, + access_key: 0, + access_key: 1, + public_access: 0, + public_access: 2 @before_compile Waffle.Definition.Storage end @@ -143,6 +162,7 @@ defmodule Waffle.Definition.Storage do quote do def acl(_, _), do: @acl def s3_object_headers(_, _), do: [] + def azure_blob_headers(_, _), do: [] def async, do: @async def remote_file_headers(_), do: [] end diff --git a/lib/waffle/storage/azure.ex b/lib/waffle/storage/azure.ex new file mode 100644 index 0000000..e7fe01c --- /dev/null +++ b/lib/waffle/storage/azure.ex @@ -0,0 +1,213 @@ +defmodule Waffle.Storage.Azure do + @moduledoc ~S""" + The module to facilitate integration with Azure Blob Storage + + config :waffle, + storage: Waffle.Storage.Azure, + storage_account: {:system, "AZURE_STORAGE_ACCOUNT"}, + container: {:system, "AZURE_STORAGE_CONTAINER"}, + access_key: {:system, "AZURE_ACCESS_KEY"} + + Along with any configuration necessary for Azure Blob Storage. + + To store your attachments in Azure Blob Storage, you'll need to provide a + storage account, container, and access key in your application config: + + config :waffle, + storage_account: "mystorageaccount", + container: "uploads", + access_key: "your-access-key" + + You may also set these values from environment variables: + + config :waffle, + storage_account: {:system, "AZURE_STORAGE_ACCOUNT"}, + container: {:system, "AZURE_STORAGE_CONTAINER"}, + access_key: {:system, "AZURE_ACCESS_KEY"} + + ## Specify multiple containers + + Waffle lets you specify a container on a per definition basis. In case + you want to use multiple containers, you can specify a container in the + definition module like this: + + def container, do: :some_custom_container_name + + You can also use the current scope to define a target container + + def container({_file, scope}), do: scope.container || container() + + ## Public vs Private Access + + Waffle defaults all uploads to private. In cases where it is desired to have + your uploads public, you may set the public access at the module level: + + @public_access true + + Or you may have more granular control over each version: + + def public_access(:thumb, _), do: true + + When public access is disabled, Waffle will generate SAS (Shared Access Signature) + tokens for secure access to the blobs. + + ## Azure Blob Headers + + The definition module may specify custom headers to pass through to + Azure Blob Storage during object creation. The available custom headers include: + + * `:cache_control` + * `:content_disposition` + * `:content_encoding` + * `:content_type` + * `:content_language` + * `:content_md5` + + As an example, to explicitly specify the content-type of an object, + you may define a `azure_blob_headers/2` function in your definition, + which returns a Keyword list, or Map of desired headers. + + def azure_blob_headers(version, {file, scope}) do + [content_type: MIME.from_path(file.file_name)] # for "image.png", would produce: "image/png" + end + + ## Configuration example + + A full example configuration for Azure Blob Storage is as follows: + + config :waffle, + storage: Waffle.Storage.Azure, + storage_account: "mystorageaccount", + container: "uploads", + access_key: "your-access-key", + public_access: false, + expiry_in_minutes: 60 + + """ + + require Logger + + @behaviour Waffle.StorageBehavior + + alias Waffle.Definition.Versioning + + @default_expiry_time 60 * 60 # 1 hour in seconds + + @impl true + def put(definition, version, {file, scope}) do + blob_name = build_blob_name(definition, version, {file, scope}) + container = azure_container(definition, {file, scope}) + + blob_headers = + definition.azure_blob_headers(version, {file, scope}) + |> ensure_keyword_list() + + do_put(file, {container, blob_name, blob_headers}) + end + + @impl true + def url(definition, version, file_and_scope, options \\ []) do + case Keyword.get(options, :signed, false) do + false -> build_url(definition, version, file_and_scope, options) + true -> build_signed_url(definition, version, file_and_scope, options) + end + end + + @impl true + def delete(definition, version, {file, scope}) do + blob_name = build_blob_name(definition, version, {file, scope}) + container = azure_container(definition, {file, scope}) + + case Waffle.Storage.Azure.Uploader.delete_blob(container, blob_name) do + {:ok, _} -> :ok + {:error, _} -> :error + end + end + + # + # Private + # + + defp ensure_keyword_list(list) when is_list(list), do: list + defp ensure_keyword_list(map) when is_map(map), do: Map.to_list(map) + + # If the file is stored as a binary in-memory, send to Azure in a single request + defp do_put(file = %Waffle.File{binary: file_binary}, {container, blob_name, blob_headers}) + when is_binary(file_binary) do + case Waffle.Storage.Azure.Uploader.upload_file(file_binary, container, blob_name, blob_headers) do + {:ok, _} -> {:ok, file.file_name} + {:error, error} -> {:error, error} + end + end + + # If the file is a stream, read it and upload to Azure + defp do_put(file = %Waffle.File{stream: file_stream}, {container, blob_name, blob_headers}) + when is_struct(file_stream) do + file_binary = file_stream |> Enum.into(<<>>) + do_put(%{file | binary: file_binary}, {container, blob_name, blob_headers}) + end + + # Stream the file and upload to Azure + defp do_put(file, {container, blob_name, blob_headers}) do + case File.read(file.path) do + {:ok, file_binary} -> + do_put(%{file | binary: file_binary}, {container, blob_name, blob_headers}) + {:error, reason} -> + Logger.error("[AzureStorage] File read failed: #{reason}") + {:error, "File read failed: #{reason}"} + end + end + + defp build_url(definition, version, file_and_scope, _options) do + blob_name = build_blob_name(definition, version, file_and_scope) + container = azure_container(definition, file_and_scope) + storage_account = azure_storage_account(definition, file_and_scope) + + "https://#{storage_account}.blob.core.windows.net/#{container}/#{blob_name}" + end + + defp build_signed_url(definition, version, file_and_scope, options) do + blob_name = build_blob_name(definition, version, file_and_scope) + container = azure_container(definition, file_and_scope) + storage_account = azure_storage_account(definition, file_and_scope) + access_key = azure_access_key(definition, file_and_scope) + + # Get expiry time from options or use default + expiry_in_seconds = Keyword.get(options, :expires_in, @default_expiry_time) + + case Waffle.Storage.Azure.SAS.generate_sas_url( + storage_account, + container, + blob_name, + access_key, + expiry_in_seconds + ) do + {:ok, url} -> url + {:error, reason} -> + Logger.error("[AzureStorage] Failed to generate signed URL: #{reason}") + build_url(definition, version, file_and_scope, options) + end + end + + defp build_blob_name(definition, version, file_and_scope) do + Path.join([ + definition.storage_dir(version, file_and_scope), + Versioning.resolve_file_name(definition, version, file_and_scope) + ]) + end + + defp azure_container(definition, file_and_scope) do + definition.container(file_and_scope) |> parse_config_value() + end + + defp azure_storage_account(definition, file_and_scope) do + definition.storage_account(file_and_scope) |> parse_config_value() + end + + defp azure_access_key(definition, file_and_scope) do + definition.access_key(file_and_scope) |> parse_config_value() + end + + defp parse_config_value({:system, env_var}) when is_binary(env_var), do: System.get_env(env_var) + defp parse_config_value(value), do: value +end diff --git a/lib/waffle/storage/azure/sas.ex b/lib/waffle/storage/azure/sas.ex new file mode 100644 index 0000000..9a8f75a --- /dev/null +++ b/lib/waffle/storage/azure/sas.ex @@ -0,0 +1,73 @@ +defmodule Waffle.Storage.Azure.SAS do + @moduledoc """ + Handles generation of Shared Access Signature (SAS) tokens for Azure Blob Storage. + """ + + @doc """ + Generates a SAS token for accessing a blob. + """ + def generate_sas_token(storage_account, container, blob_name, access_key, expiry_in_seconds) do + now = DateTime.utc_now() + expiry = DateTime.add(now, expiry_in_seconds, :second) + + permissions = "r" # Read permission + resource_type = "b" # Blob resource type + canonicalized_resource = "/blob/#{storage_account}/#{container}/#{blob_name}" + + # String to sign for Azure Blob Storage SAS version 2020-12-06 + string_to_sign = [ + permissions, # sp (signedPermissions) + iso8601_z(now), # st (signedStart) + iso8601_z(expiry), # se (signedExpiry) + canonicalized_resource, # canonicalized resource + "", # signedIdentifier + "", # signedIP + "https", # signedProtocol + "2020-12-06", # sv (signedVersion) + resource_type, # sr (signedResource) + "", # signedSnapshotTime + "", # signedEncryptionScope + "", # rscc + "", # rscd + "", # rsce + "", # rscl + "" # rsct + ] + |> Enum.join("\n") + + decoded_key = :base64.decode(access_key) + signature = :crypto.mac(:hmac, :sha256, decoded_key, string_to_sign) |> Base.encode64() + + # Build query parameters + query_params = %{ + "sv" => "2020-12-06", # version + "st" => iso8601_z(now), # start time + "se" => iso8601_z(expiry), # expiry + "sr" => resource_type, # resource type + "sp" => permissions, # permissions + "spr" => "https", # signed protocol + "sig" => signature # signature + } + + URI.encode_query(query_params) + end + + @doc """ + Generates a complete SAS URL for a blob. + """ + def generate_sas_url(storage_account, container, blob_name, access_key, expiry_in_seconds) do + try do + sas_token = generate_sas_token(storage_account, container, blob_name, access_key, expiry_in_seconds) + url = "https://#{storage_account}.blob.core.windows.net/#{container}/#{blob_name}?#{sas_token}" + {:ok, url} + rescue + error -> + {:error, "Failed to generate SAS URL: #{inspect(error)}"} + end + end + + # Helper function to format datetime in UTC with Z suffix (Azure-compatible) + defp iso8601_z(datetime) do + Calendar.strftime(datetime, "%Y-%m-%dT%H:%M:%SZ") + end +end diff --git a/lib/waffle/storage/azure/uploader.ex b/lib/waffle/storage/azure/uploader.ex new file mode 100644 index 0000000..2a0aa09 --- /dev/null +++ b/lib/waffle/storage/azure/uploader.ex @@ -0,0 +1,122 @@ +defmodule Waffle.Storage.Azure.Uploader do + @moduledoc """ + Handles file uploads to Azure Blob Storage using Shared Key authentication. + """ + + require Logger + + @doc """ + Uploads a file to Azure Blob Storage. + """ + def upload_file(file_binary, container, blob_name, headers \\ []) do + config = azure_config() + + storage_account = config.storage_account + access_key = config.access_key + + url = "https://#{storage_account}.blob.core.windows.net/#{container}/#{blob_name}" + + datetime = generate_utc_datetime() + content_length = byte_size(file_binary) + content_type = Keyword.get(headers, :content_type, "application/octet-stream") + + auth_headers = [ + {"x-ms-date", datetime}, + {"x-ms-version", "2020-08-04"}, + {"x-ms-blob-type", "BlockBlob"}, + {"Content-Length", "#{content_length}"}, + {"Content-Type", content_type} + ] + + # Add custom headers + custom_headers = + headers + |> Keyword.drop([:content_type]) + |> Enum.map(fn {key, value} -> {to_string(key), to_string(value)} end) + + all_headers = auth_headers ++ custom_headers ++ [ + {"Authorization", authorization("PUT", datetime, content_length, blob_name, storage_account, container, access_key, content_type)} + ] + + case Req.put(url, headers: all_headers, body: file_binary) do + {:ok, %Req.Response{status: 201}} -> {:ok, blob_name} + {:ok, %Req.Response{status: status, body: body}} -> + Logger.error("[AzureUploader] Upload failed with status #{status}: #{inspect(body)}") + {:error, "Upload failed with status #{status}"} + {:error, reason} -> + Logger.error("[AzureUploader] Upload error: #{inspect(reason)}") + {:error, reason} + end + end + + @doc """ + Deletes a blob from Azure Blob Storage. + """ + def delete_blob(container, blob_name) do + config = azure_config() + + storage_account = config.storage_account + access_key = config.access_key + + url = "https://#{storage_account}.blob.core.windows.net/#{container}/#{blob_name}" + + datetime = generate_utc_datetime() + + headers = [ + {"x-ms-date", datetime}, + {"x-ms-version", "2020-08-04"}, + {"Authorization", authorization("DELETE", datetime, 0, blob_name, storage_account, container, access_key, "")} + ] + + case Req.delete(url, headers: headers) do + {:ok, %Req.Response{status: 202}} -> {:ok, :deleted} + {:ok, %Req.Response{status: 404}} -> {:ok, :not_found} + {:ok, %Req.Response{status: status, body: body}} -> + Logger.error("[AzureUploader] Delete failed with status #{status}: #{inspect(body)}") + {:error, "Delete failed with status #{status}"} + {:error, reason} -> + Logger.error("[AzureUploader] Delete error: #{inspect(reason)}") + {:error, reason} + end + end + + defp azure_config do + %{ + storage_account: Application.fetch_env!(:waffle, :storage_account), + access_key: Application.fetch_env!(:waffle, :access_key) + } + end + + defp generate_utc_datetime do + Timex.now("Etc/UTC") + |> Timex.format!("{WDshort}, {0D} {Mshort} {YYYY} {h24}:{m}:{s} GMT") + end + + defp authorization(method, date, content_length, blob_name, storage_account, container, access_key, content_type) do + canonicalized_resource = "/#{storage_account}/#{container}/#{blob_name}" + + string_to_sign = [ + method, + "", # Content-Encoding + "", # Content-Language + "#{content_length}", # Content-Length + "", # Content-MD5 + content_type, # Content-Type + "", # Date + "", # If-Modified-Since + "", # If-Match + "", # If-None-Match + "", # If-Unmodified-Since + "", # Range + "x-ms-blob-type:BlockBlob", + "x-ms-date:#{date}", + "x-ms-version:2020-08-04", + canonicalized_resource + ] + |> Enum.join("\n") + + decoded_key = :base64.decode(access_key) + signature = :crypto.mac(:hmac, :sha256, decoded_key, string_to_sign) |> Base.encode64() + "SharedKey #{storage_account}:#{signature}" + end +end diff --git a/mix.exs b/mix.exs index f68b8bd..df47be1 100644 --- a/mix.exs +++ b/mix.exs @@ -41,6 +41,7 @@ defmodule Waffle.Mixfile do "README.md", "documentation/examples/local.md", "documentation/examples/s3.md", + "documentation/examples/azure.md", "documentation/livebooks/custom_transformation.livemd" ] ] @@ -65,6 +66,10 @@ defmodule Waffle.Mixfile do {:ex_aws_s3, "~> 2.1", optional: true}, {:sweet_xml, "~> 0.6", optional: true}, + # If using Azure Blob Storage + {:req, "~> 0.4", optional: true}, + {:timex, "~> 3.7", optional: true}, + # Test {:mock, "~> 0.3", only: :test}, diff --git a/mix.lock b/mix.lock index bc6be50..81826c5 100644 --- a/mix.lock +++ b/mix.lock @@ -1,14 +1,19 @@ %{ "bunt": {:hex, :bunt, "0.2.1", "e2d4792f7bc0ced7583ab54922808919518d0e57ee162901a16a1b6664ef3b14", [:mix], [], "hexpm", "a330bfb4245239787b15005e66ae6845c9cd524a288f0d141c148b02603777a5"}, "certifi": {:hex, :certifi, "2.9.0", "6f2a475689dd47f19fb74334859d460a2dc4e3252a3324bd2111b8f0429e7e21", [:rebar3], [], "hexpm", "266da46bdb06d6c6d35fde799bcb28d36d985d424ad7c08b5bb48f5b5cdd4641"}, + "combine": {:hex, :combine, "0.10.0", "eff8224eeb56498a2af13011d142c5e7997a80c8f5b97c499f84c841032e429f", [:mix], [], "hexpm", "1b1dbc1790073076580d0d1d64e42eae2366583e7aecd455d1215b0d16f2451b"}, "credo": {:hex, :credo, "1.6.7", "323f5734350fd23a456f2688b9430e7d517afb313fbd38671b8a4449798a7854", [:mix], [{:bunt, "~> 0.2.1", [hex: :bunt, repo: "hexpm", optional: false]}, {:file_system, "~> 0.2.8", [hex: :file_system, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: false]}], "hexpm", "41e110bfb007f7eda7f897c10bf019ceab9a0b269ce79f015d54b0dcf4fc7dd3"}, "earmark": {:hex, :earmark, "1.4.4", "4821b8d05cda507189d51f2caeef370cf1e18ca5d7dfb7d31e9cafe6688106a4", [:mix], [], "hexpm"}, "earmark_parser": {:hex, :earmark_parser, "1.4.39", "424642f8335b05bb9eb611aa1564c148a8ee35c9c8a8bba6e129d51a3e3c6769", [:mix], [], "hexpm", "06553a88d1f1846da9ef066b87b57c6f605552cfbe40d20bd8d59cc6bde41944"}, "ex_aws": {:hex, :ex_aws, "2.4.1", "d1dc8965d1dc1c939dd4570e37f9f1d21e047e4ecd4f9373dc89cd4e45dce5ef", [:mix], [{:configparser_ex, "~> 4.0", [hex: :configparser_ex, repo: "hexpm", optional: true]}, {:hackney, "~> 1.16", [hex: :hackney, repo: "hexpm", optional: true]}, {:jason, "~> 1.1", [hex: :jason, repo: "hexpm", optional: true]}, {:jsx, "~> 2.8 or ~> 3.0", [hex: :jsx, repo: "hexpm", optional: true]}, {:mime, "~> 1.2 or ~> 2.0", [hex: :mime, repo: "hexpm", optional: false]}, {:sweet_xml, "~> 0.7", [hex: :sweet_xml, repo: "hexpm", optional: true]}, {:telemetry, "~> 0.4.3 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "803387db51b4e91be4bf0110ba999003ec6103de7028b808ee9b01f28dbb9eee"}, "ex_aws_s3": {:hex, :ex_aws_s3, "2.4.0", "ce8decb6b523381812798396bc0e3aaa62282e1b40520125d1f4eff4abdff0f4", [:mix], [{:ex_aws, "~> 2.0", [hex: :ex_aws, repo: "hexpm", optional: false]}, {:sweet_xml, ">= 0.0.0", [hex: :sweet_xml, repo: "hexpm", optional: true]}], "hexpm", "85dda6e27754d94582869d39cba3241d9ea60b6aa4167f9c88e309dc687e56bb"}, "ex_doc": {:hex, :ex_doc, "0.34.0", "ab95e0775db3df71d30cf8d78728dd9261c355c81382bcd4cefdc74610bef13e", [:mix], [{:earmark_parser, "~> 1.4.39", [hex: :earmark_parser, repo: "hexpm", optional: false]}, {:makeup_c, ">= 0.1.0", [hex: :makeup_c, repo: "hexpm", optional: true]}, {:makeup_elixir, "~> 0.14 or ~> 1.0", [hex: :makeup_elixir, repo: "hexpm", optional: false]}, {:makeup_erlang, "~> 0.1 or ~> 1.0", [hex: :makeup_erlang, repo: "hexpm", optional: false]}, {:makeup_html, ">= 0.1.0", [hex: :makeup_html, repo: "hexpm", optional: true]}], "hexpm", "60734fb4c1353f270c3286df4a0d51e65a2c1d9fba66af3940847cc65a8066d7"}, + "expo": {:hex, :expo, "1.1.0", "f7b9ed7fb5745ebe1eeedf3d6f29226c5dd52897ac67c0f8af62a07e661e5c75", [:mix], [], "hexpm", "fbadf93f4700fb44c331362177bdca9eeb8097e8b0ef525c9cc501cb9917c960"}, "file_system": {:hex, :file_system, "0.2.10", "fb082005a9cd1711c05b5248710f8826b02d7d1784e7c3451f9c1231d4fc162d", [:mix], [], "hexpm", "41195edbfb562a593726eda3b3e8b103a309b733ad25f3d642ba49696bf715dc"}, + "finch": {:hex, :finch, "0.20.0", "5330aefb6b010f424dcbbc4615d914e9e3deae40095e73ab0c1bb0968933cadf", [:mix], [{:mime, "~> 1.0 or ~> 2.0", [hex: :mime, repo: "hexpm", optional: false]}, {:mint, "~> 1.6.2 or ~> 1.7", [hex: :mint, repo: "hexpm", optional: false]}, {:nimble_options, "~> 0.4 or ~> 1.0", [hex: :nimble_options, repo: "hexpm", optional: false]}, {:nimble_pool, "~> 1.1", [hex: :nimble_pool, repo: "hexpm", optional: false]}, {:telemetry, "~> 0.4 or ~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "2658131a74d051aabfcba936093c903b8e89da9a1b63e430bee62045fa9b2ee2"}, + "gettext": {:hex, :gettext, "0.26.2", "5978aa7b21fada6deabf1f6341ddba50bc69c999e812211903b169799208f2a8", [:mix], [{:expo, "~> 0.5.1 or ~> 1.0", [hex: :expo, repo: "hexpm", optional: false]}], "hexpm", "aa978504bcf76511efdc22d580ba08e2279caab1066b76bb9aa81c4a1e0a32a5"}, "hackney": {:hex, :hackney, "1.18.1", "f48bf88f521f2a229fc7bae88cf4f85adc9cd9bcf23b5dc8eb6a1788c662c4f6", [:rebar3], [{:certifi, "~> 2.9.0", [hex: :certifi, repo: "hexpm", optional: false]}, {:idna, "~> 6.1.0", [hex: :idna, repo: "hexpm", optional: false]}, {:metrics, "~> 1.0.0", [hex: :metrics, repo: "hexpm", optional: false]}, {:mimerl, "~> 1.1", [hex: :mimerl, repo: "hexpm", optional: false]}, {:parse_trans, "3.3.1", [hex: :parse_trans, repo: "hexpm", optional: false]}, {:ssl_verify_fun, "~> 1.1.0", [hex: :ssl_verify_fun, repo: "hexpm", optional: false]}, {:unicode_util_compat, "~> 0.7.0", [hex: :unicode_util_compat, repo: "hexpm", optional: false]}], "hexpm", "a4ecdaff44297e9b5894ae499e9a070ea1888c84afdd1fd9b7b2bc384950128e"}, + "hpax": {:hex, :hpax, "1.0.3", "ed67ef51ad4df91e75cc6a1494f851850c0bd98ebc0be6e81b026e765ee535aa", [:mix], [], "hexpm", "8eab6e1cfa8d5918c2ce4ba43588e894af35dbd8e91e6e55c817bca5847df34a"}, "idna": {:hex, :idna, "6.1.1", "8a63070e9f7d0c62eb9d9fcb360a7de382448200fbbd1b106cc96d3d8099df8d", [:rebar3], [{:unicode_util_compat, "~> 0.7.0", [hex: :unicode_util_compat, repo: "hexpm", optional: false]}], "hexpm", "92376eb7894412ed19ac475e4a86f7b413c1b9fbb5bd16dccd57934157944cea"}, "jason": {:hex, :jason, "1.4.0", "e855647bc964a44e2f67df589ccf49105ae039d4179db7f6271dfd3843dc27e6", [:mix], [{:decimal, "~> 1.0 or ~> 2.0", [hex: :decimal, repo: "hexpm", optional: true]}], "hexpm", "79a3791085b2a0f743ca04cec0f7be26443738779d09302e01318f97bdb82121"}, "makeup": {:hex, :makeup, "1.1.2", "9ba8837913bdf757787e71c1581c21f9d2455f4dd04cfca785c70bbfff1a76a3", [:mix], [{:nimble_parsec, "~> 1.2.2 or ~> 1.3", [hex: :nimble_parsec, repo: "hexpm", optional: false]}], "hexpm", "cce1566b81fbcbd21eca8ffe808f33b221f9eee2cbc7a1706fc3da9ff18e6cac"}, @@ -18,11 +23,17 @@ "metrics": {:hex, :metrics, "1.0.1", "25f094dea2cda98213cecc3aeff09e940299d950904393b2a29d191c346a8486", [:rebar3], [], "hexpm", "69b09adddc4f74a40716ae54d140f93beb0fb8978d8636eaded0c31b6f099f16"}, "mime": {:hex, :mime, "2.0.3", "3676436d3d1f7b81b5a2d2bd8405f412c677558c81b1c92be58c00562bb59095", [:mix], [], "hexpm", "27a30bf0db44d25eecba73755acf4068cbfe26a4372f9eb3e4ea3a45956bff6b"}, "mimerl": {:hex, :mimerl, "1.2.0", "67e2d3f571088d5cfd3e550c383094b47159f3eee8ffa08e64106cdf5e981be3", [:rebar3], [], "hexpm", "f278585650aa581986264638ebf698f8bb19df297f66ad91b18910dfc6e19323"}, + "mint": {:hex, :mint, "1.7.1", "113fdb2b2f3b59e47c7955971854641c61f378549d73e829e1768de90fc1abf1", [:mix], [{:castore, "~> 0.1.0 or ~> 1.0", [hex: :castore, repo: "hexpm", optional: true]}, {:hpax, "~> 0.1.1 or ~> 0.2.0 or ~> 1.0", [hex: :hpax, repo: "hexpm", optional: false]}], "hexpm", "fceba0a4d0f24301ddee3024ae116df1c3f4bb7a563a731f45fdfeb9d39a231b"}, "mock": {:hex, :mock, "0.3.7", "75b3bbf1466d7e486ea2052a73c6e062c6256fb429d6797999ab02fa32f29e03", [:mix], [{:meck, "~> 0.9.2", [hex: :meck, repo: "hexpm", optional: false]}], "hexpm", "4da49a4609e41fd99b7836945c26f373623ea968cfb6282742bcb94440cf7e5c"}, + "nimble_options": {:hex, :nimble_options, "1.1.1", "e3a492d54d85fc3fd7c5baf411d9d2852922f66e69476317787a7b2bb000a61b", [:mix], [], "hexpm", "821b2470ca9442c4b6984882fe9bb0389371b8ddec4d45a9504f00a66f650b44"}, "nimble_parsec": {:hex, :nimble_parsec, "1.4.0", "51f9b613ea62cfa97b25ccc2c1b4216e81df970acd8e16e8d1bdc58fef21370d", [:mix], [], "hexpm", "9c565862810fb383e9838c1dd2d7d2c437b3d13b267414ba6af33e50d2d1cf28"}, + "nimble_pool": {:hex, :nimble_pool, "1.1.0", "bf9c29fbdcba3564a8b800d1eeb5a3c58f36e1e11d7b7fb2e084a643f645f06b", [:mix], [], "hexpm", "af2e4e6b34197db81f7aad230c1118eac993acc0dae6bc83bac0126d4ae0813a"}, "parse_trans": {:hex, :parse_trans, "3.3.1", "16328ab840cc09919bd10dab29e431da3af9e9e7e7e6f0089dd5a2d2820011d8", [:rebar3], [], "hexpm", "07cd9577885f56362d414e8c4c4e6bdf10d43a8767abb92d24cbe8b24c54888b"}, + "req": {:hex, :req, "0.5.1", "90584216d064389a4ff2d4279fe2c11ff6c812ab00fa01a9fb9d15457f65ba70", [:mix], [{:brotli, "~> 0.3.1", [hex: :brotli, repo: "hexpm", optional: true]}, {:ezstd, "~> 1.0", [hex: :ezstd, repo: "hexpm", optional: true]}, {:finch, "~> 0.17", [hex: :finch, repo: "hexpm", optional: false]}, {:jason, "~> 1.0", [hex: :jason, repo: "hexpm", optional: false]}, {:mime, "~> 1.6 or ~> 2.0", [hex: :mime, repo: "hexpm", optional: false]}, {:nimble_csv, "~> 1.0", [hex: :nimble_csv, repo: "hexpm", optional: true]}, {:plug, "~> 1.0", [hex: :plug, repo: "hexpm", optional: true]}], "hexpm", "7ea96a1a95388eb0fefa92d89466cdfedba24032794e5c1147d78ec90db7edca"}, "ssl_verify_fun": {:hex, :ssl_verify_fun, "1.1.7", "354c321cf377240c7b8716899e182ce4890c5938111a1296add3ec74cf1715df", [:make, :mix, :rebar3], [], "hexpm", "fe4c190e8f37401d30167c8c405eda19469f34577987c76dde613e838bbc67f8"}, "sweet_xml": {:hex, :sweet_xml, "0.7.3", "debb256781c75ff6a8c5cbf7981146312b66f044a2898f453709a53e5031b45b", [:mix], [], "hexpm", "e110c867a1b3fe74bfc7dd9893aa851f0eed5518d0d7cad76d7baafd30e4f5ba"}, "telemetry": {:hex, :telemetry, "1.2.1", "68fdfe8d8f05a8428483a97d7aab2f268aaff24b49e0f599faa091f1d4e7f61c", [:rebar3], [], "hexpm", "dad9ce9d8effc621708f99eac538ef1cbe05d6a874dd741de2e689c47feafed5"}, + "timex": {:hex, :timex, "3.7.13", "0688ce11950f5b65e154e42b47bf67b15d3bc0e0c3def62199991b8a8079a1e2", [:mix], [{:combine, "~> 0.10", [hex: :combine, repo: "hexpm", optional: false]}, {:gettext, "~> 0.26", [hex: :gettext, repo: "hexpm", optional: false]}, {:tzdata, "~> 1.1", [hex: :tzdata, repo: "hexpm", optional: false]}], "hexpm", "09588e0522669328e973b8b4fd8741246321b3f0d32735b589f78b136e6d4c54"}, + "tzdata": {:hex, :tzdata, "1.1.3", "b1cef7bb6de1de90d4ddc25d33892b32830f907e7fc2fccd1e7e22778ab7dfbc", [:mix], [{:hackney, "~> 1.17", [hex: :hackney, repo: "hexpm", optional: false]}], "hexpm", "d4ca85575a064d29d4e94253ee95912edfb165938743dbf002acdf0dcecb0c28"}, "unicode_util_compat": {:hex, :unicode_util_compat, "0.7.0", "bc84380c9ab48177092f43ac89e4dfa2c6d62b40b8bd132b1059ecc7232f9a78", [:rebar3], [], "hexpm", "25eee6d67df61960cf6a794239566599b09e17e668d3700247bc498638152521"}, } diff --git a/test/storage/azure/sas_test.exs b/test/storage/azure/sas_test.exs new file mode 100644 index 0000000..b42296d --- /dev/null +++ b/test/storage/azure/sas_test.exs @@ -0,0 +1,46 @@ +defmodule Waffle.Storage.Azure.SASTest do + use ExUnit.Case, async: true + + alias Waffle.Storage.Azure.SAS + + describe "generate_sas_token/5" do + test "generates valid SAS token" do + storage_account = "testaccount" + container = "testcontainer" + blob_name = "test/blob.jpg" + access_key = "dGVzdGtleQ==" # "testkey" in base64 + expiry_in_seconds = 3600 + + sas_token = SAS.generate_sas_token(storage_account, container, blob_name, access_key, expiry_in_seconds) + + # Check that the SAS token contains required parameters + assert String.contains?(sas_token, "sv=2020-12-06") + assert String.contains?(sas_token, "sp=r") + assert String.contains?(sas_token, "sr=b") + assert String.contains?(sas_token, "spr=https") + assert String.contains?(sas_token, "sig=") + end + end + + describe "generate_sas_url/5" do + test "generates complete SAS URL" do + storage_account = "testaccount" + container = "testcontainer" + blob_name = "test/blob.jpg" + access_key = "dGVzdGtleQ==" # "testkey" in base64 + expiry_in_seconds = 3600 + + {:ok, url} = SAS.generate_sas_url(storage_account, container, blob_name, access_key, expiry_in_seconds) + + assert String.starts_with?(url, "https://testaccount.blob.core.windows.net/testcontainer/test/blob.jpg") + assert String.contains?(url, "?") + end + + test "handles errors gracefully" do + # Test with invalid access key + result = SAS.generate_sas_url("account", "container", "blob", "invalid_key", 3600) + + assert {:error, _} = result + end + end +end diff --git a/test/storage/azure_test.exs b/test/storage/azure_test.exs new file mode 100644 index 0000000..029d155 --- /dev/null +++ b/test/storage/azure_test.exs @@ -0,0 +1,46 @@ +defmodule Waffle.Storage.AzureTest do + use ExUnit.Case, async: true + + alias Waffle.Storage.Azure + + describe "SAS token generation" do + test "generates valid SAS token" do + storage_account = "testaccount" + container = "testcontainer" + blob_name = "test/blob.jpg" + access_key = "dGVzdGtleQ==" # "testkey" in base64 + expiry_in_seconds = 3600 + + sas_token = Azure.SAS.generate_sas_token(storage_account, container, blob_name, access_key, expiry_in_seconds) + + # Check that the SAS token contains required parameters + assert String.contains?(sas_token, "sv=2020-12-06") + assert String.contains?(sas_token, "sp=r") + assert String.contains?(sas_token, "sr=b") + assert String.contains?(sas_token, "spr=https") + assert String.contains?(sas_token, "sig=") + end + end + + describe "SAS URL generation" do + test "generates complete SAS URL" do + storage_account = "testaccount" + container = "testcontainer" + blob_name = "test/blob.jpg" + access_key = "dGVzdGtleQ==" # "testkey" in base64 + expiry_in_seconds = 3600 + + {:ok, url} = Azure.SAS.generate_sas_url(storage_account, container, blob_name, access_key, expiry_in_seconds) + + assert String.starts_with?(url, "https://testaccount.blob.core.windows.net/testcontainer/test/blob.jpg") + assert String.contains?(url, "?") + end + + test "handles errors gracefully" do + # Test with invalid access key + result = Azure.SAS.generate_sas_url("account", "container", "blob", "invalid_key", 3600) + + assert {:error, _} = result + end + end +end