I have a project where the client can do a POST request to receive back a tarball. On each POST, a new tarball will be generated so that after sending the tarball it can safely be removed. The tarball format is chosen to group a bunch of files in a single download. It’s a relatively small file, however, I want to avoid running out of disk space, because even small files accumulate over time. Given these constraints, I’ve tried a few different solutions I want to share with you. I think it can prove helpful in choosing the right solution for your specific use case.

Keep the file in memory

An easy way out of this pickle is to only write the contents of the file in memory. With Phoenix, you can use send_download/3 to send the filename plus binary contents. It would look something like this:

# %Plug.Conn{} = conn
send_download(conn,
  {:binary, "These are my file contents"},
  filename: "the_filename.txt")

There are a few caveats with this approach. Large binaries will be stored off-heap so it won’t be a problem for your process, but it still won’t scale to very large sizes or many concurrent requests. Another problem can be when you don’t control content generation. In my case, I depend on :erl_tar which only allows writing the archive to a file. I’d have to read the file after creation into memory, delete the file safely, and then send it to the client. Eg:

binary = File.read!("my_archive.tar.gz")
:ok = File.rm!("my_archive.tar.gz")
send_download(conn,
  {:binary, binary},
  filename: "my_archive_20230302.tar.gz")

Delete the file after sending, or so I thought

This assumes Phoenix with Plug.Cowboy as the adapter. The idea is simple, you send the file, you delete the file, and you return %Plug.Conn{} in your controller.

conn =
  conn
  |> put_resp_header("content-disposition", ~s(attachment; filename=my_archive_20230302.tar.gz))
  |> send_file(200, "my_archive.tar.gz")

:ok = File.rm!("my_archive.tar.gz")
conn

When I was testing this locally I found that sometimes I didn’t receive the file back. What is going on here?

Plug’s send_file/3 passes the command down the chain to Cowboy’s :cowboy_req.reply/4 and there we find the culprit. The filename is passed from the ‘replying’-process (my controller) to the ‘requesting’-process (the request worker) with send/2. The function in :cowboy_req that does this is even called cast/2. This means that my controller is racing with the Cowboy process and when it’s faster the file is gone before it’s read and sent back.

You can work around this by monitoring the requesting process and removing the file only when that process exits. This means you have to interface with Plug’s internals. You will also find that the ‘requesting’-process is reused and only recycled after a while. The code would look like this:

{_adapter, adapter_data} = conn.adapter
spawn(fn ->
  ref = Process.monitor(adapter_data.pid)

  receive do
    {:DOWN, ^ref, :process, _, _} -> :ok = File.remove!("my_archive.tar.gz")
  end
end)

conn
|> put_resp_header("content-disposition", ~s(attachment; filename=my_archive_20230302.tar.gz))
|> send_file(200, "my_archive.tar.gz")

Delete the file after sending, Bandit-edition

Bandit is a relatively new HTTP server that is compatible with Plug. I’ve checked the code path that deals with sending files and found out it does send the file from the process I am controlling. This means that my earlier code Just Works™️ when I configure Bandit as my Phoenix adapter. No need for monitors or reading files into memory. In addition, the code now seems to do what I think it would be doing when reading it. For clarity the code looks like this:

conn =
  conn
  |> put_resp_header("content-disposition", ~s(attachment; filename=my_archive_20230302.tar.gz))
  |> send_file(200, "my_archive.tar.gz")

:ok = File.rm!("my_archive.tar.gz")
conn

As I said, Bandit is relatively new and doesn’t (yet) have the proven track record of Cowboy, but it has a promising story and from what I gather works fine.

Alternatives?

Did I miss anything or do you have a different approach that deals with this situation? Do let me know! I’d be glad to hear about it.

This is the written form of a presentation I gave at Code BEAM V 2020. “BEAM” is the name of the Erlang VM.

There are quite a few technical presentations at the Code BEAM conferences. Wonderful stories about advancing the BEAM, adding different languages to the BEAM, creating new libraries in the ecosystem. The list goes on and on. I’m sharing a different story with you. A war story. About going from failure to success.

Failures

It’s OK to have failures. It’s OK to not know how to go about it. There is no ready-to-use solution for your one-of-a-kind problem. Handling failure is about fault-tolerance as a team and as a company. We make mistakes and then we fix them. This is not the effort of a single person. It’s the effort of everyone involved. I’m sharing it from my perspective. There is no universal truth.

I will not cover Erlang/Elixir anti-patterns or how to design applications for the BEAM. It is mostly about things you should try to avoid and how we went about solving our own issues.

Background

We needed to replace the existing communication platform (based on Asterisk and a lot of PHP scripts). A communication platform is a perfect fit for the BEAM. At least, that is what we thought. Picking the proper tools and software is a good start, but there is still a long road ahead.

At that time we weren’t (yet) a group of experienced Erlang or Elixir developers. We designed and created lots of components based on how we thought the BEAM would behave and what we thought our users needed. We were doing a lot of tests and benchmarks to confirm our beliefs. But getting results from a “lab” environment can be quite different from the real world.

The First Two Years

We started building the platform in Elixir. The company already had some experience with Erlang and running the BEAM in production. I started at zero. I picked up the famous book of Joe Armstrong, Programming Erlang. It immediately clicked for me. I recommend this book to anyone that wants to learn Erlang or get a basic understanding of the Erlang Runtime.

At the time we favored Elixir over Erlang for its macro programming capabilities and its syntax. There is a lot more to it, but that would make a different story ;-).

For the first two years, we’ve developed without any users using it. You can imagine that developing a platform for this long by a team for which Elixir is relatively new was a big gamble. I remember a couple of interesting conversations and promises about that with a key stakeholder during one of our Christmas gatherings.

We created a proof of concept. Then we rewrote it. Then we rewrote it once more. Each iteration showed us something and improved our understanding of how to design it. After the third iteration, we were satisfied with the trade-offs we made. Although only the last iteration eventually was exposed to users.

Beta Launch

In the third year, we announced and launched the new platform. We had a big party and some really shiny features. It allowed our users to test-drive the platform. We got our first real user feedback. We also got asked for a ton of features we hadn’t ported yet from the old platform.

The following year we continued developing the platform. Basic functionality broke at times, but we fixed each bug quickly. The shiny new features seemed to always work. Our users weren’t entirely happy about that balance as they wanted to run their businesses using this platform. Understandably, that requires basic functionality to be super stable.

Go-Live

After another year of development, we ripped off the beta label. Three years of development before we got actual businesses operating on our platform. This resulted in quite different feedback than in the year before the go-live. It didn’t take long before we got difficult and complex issues to solve.

We soon realized that our technical design wasn’t lined up with the behavior that our users were expecting. It was even a bit unpredictable from their point of view. The behavior was intertwined with our technical design. This meant that altering the behavior required quite a big refactoring. We had some workarounds in place to reduce the friction for our users. It did require a month or two to make the technical design follow the expected behavior. After we migrated the data we finally got predictable behavior from our users’ perspective.

While working these first months we also reflected on our development process. A lot of work has been done on “islands”. The level of shared understanding was therefore a bit low. We started doing more pairing and better peer review. Not only for approvals but for knowledge sharing. This helped everyone to better understand the patchwork we created.

An Eventful Year

The first months had settled in and we felt confident about our progress. We never predicted the year ahead of us would be so eventful.

For resiliency reasons we were running multiple nodes and because information between nodes needed to be shared in realtime we were using Erlang clustering. After the initial period, we started getting single node outages. The VM was still running but our application didn’t function anymore. The other cluster nodes were not affected at the same time. Investigation showed that the application was no longer able to execute database queries. We updated our libraries. Reduced amount of queries. Upgraded Erlang. We’ve never found out the exact root-cause, but the issue did disappear eventually.

During the same time, we had issues with a specific application in our umbrella application. Technically it was a bug that started to appear under certain timing conditions. If we wouldn’t have developed on islands, we might have found it sooner. Unraveling the timing issue was a delicate task and revealed a couple of related bugs. By joining our efforts it became easier to manage and work through all these bugs.

We made some obvious mistakes as well. Things that are easily preventable but given our focus just were not in the picture. Our database ran out of space unexpectedly, our HTTPS certificates expired and to top it all off we accidentally dropped all tables in our database. I hear you thinking: those are bad operations practices. You are right, but humans make mistakes and we learn from them. We switched our RDBMS and migrated to different certificate handling. Not because it was better, but because it allowed our infrastructure team to better support the architecture. I must add that Ecto helped a lot in this transition as we didn’t have to change much.

Switching the database was good from an operational perspective. There was a big difference in how both databases responded to our workload. We had to finetune a lot. This ranged from changing how we queried our data, adding indexes, increasing performance by adding hardware and caching data in the application.

We were out of the woods for a little while. Our hard work was paying off and our customers were noticing it. The platform was still steadily growing. Three months passed without big issues.

We had a couple of platform-wide outages. This meant our clustered platform had more impact than we wanted and wasn’t as resilient as we wanted it to be. It was kind of a snowball effect that happened. We made the snowball much smaller to decrease the impact and we took some precautions to prevent the snowball altogether. These two things combined allowed us to prevent and otherwise control the behavior. A big relief.

COVID-19

Things were looking promising. I guess we all felt that way. Especially in hindsight. The year could have been so much better.

A part of our communication platform was a videoconferencing solution. It wasn’t widely used and we considered dropping it earlier. Suddenly it peaked in traffic and it was the new hotness of the platform. Unfortunately for us, we hadn’t been paying much attention to it. Bottlenecks became clear quite soon.

We had a single process (gen_server) for signaling from our application to our media gateway. It worked properly until then but it was not ready for scale. We refactored this into multiple processes to remove the bottleneck. If COVID-19 wouldn’t have happened this would probably not have happened.

What Have We Learned

Failures are easy. They come in many shapes and for many reasons. Each one of them requires a different solution. Working together allows you to find the best one. You have to do the work to solve them. There are no short-cuts to success!

I feel very grateful for having been a part of an amazing team while facing these challenges. I’m thankful for the support I received from the team and the company.

Elixir protocols are a powerful mechanism for achieving polymorphism. But with great power comes (you’ve guessed it) great responsibility. I want to share my perspective on this as I’ve seen many useful and very complex protocol implementations.

First of all, if you’re not familiar with protocols yet I recommend to check out the official getting started guide for protocols. It’s a very good resource to learn about the topic and understand how it works. For the rest of this post, I’ll assume you understand what a protocol is and does. I don’t want to repeat the same material with the chance my version is a poor copy of the original. This also has the potential to leave you more clueless than when you started reading this article.

The example given by the official guide is quite useful. It defines the protocol Size and it is defined like this:

defprotocol Size do
  @doc "Calculates the size (and not the length!) of a data structure"
  def size(data)
end

In fact, the intention of this protocol is not to calculate the size, but to return the pre-computed size of an element (). Perhaps, the protocol would be better named PreComputedSize to clarify intent, but it becomes a bit long and you could easily clarify intent in the module documentation. I think it is very important to describe what your protocol is supposed to do. I believe it makes it easier to add new implementations.

My Webshop

For the moment, let’s assume I own a webshop and I sell a couple of office supplies; paper & paperclips. When I need to ship these items I need to know an approximation of the volume so I know how many I can pack in a box. I’ll define a protocol called Product with a function volume/1 and implement the protocol for %Paper and %Paperclips{}. Here is it in code:

defprotocol Product do
  @doc "Calculates the volume of a data structure"
  def volume(term)
end

defimpl Product, for: Paperclips do
  def volume(paperclips) do
    paperclips.volume
  end
end

defimpl Product, for: Paper do
  def volume(paper) do
    paper.x * paper.y * paper.z
  end
end

As you can see, the data structure for paperclips has a volume key and the protocol can just return the value. For paper, it is a bit more complex as for the dimensions multiplication is required, but it’s not too hard. Now when I get a list of items I can easily calculate the volume and pack everything in one or multiple boxes:

items
|> Enum.map(&Product.volume/1)
|> pack_in_a_box()

And voila! I can sell paper & paperclips via my webshop and pack them in appropriate boxes.

Adding a Product (or Two)

But to my surprise, the webshop has been a great success and I want to add a product. This should be fairly easy as I defined a protocol and Product can simply be implemented for %Pencil{}. Here we go:

defimpl Product, for: Pencil do
  def volume(pencil) do
    pi() * :math.pow(pencil.r / 2, 2) * pencil.h
  end
end

It’s probably a good idea not to calculate pi/0 on the spot, but what if I would? From an implementation perspective, it seems to be correct. This is indeed how you calculate the volume for a pencil. However, suddenly my box packing code becomes slow at seemingly random moments. You can see that the protocol makes it easy to extend it. Though the complexity of the implementation may have unintended side-effects. We can still make it a bit worse though since I also plan on selling boxed pencils.

defimpl Product, for: Pencilbox do
  def volume(pencilbox) do
    pencilbox.pencil_count * Product.volume(pencilbox.pencil)
  end
end

Whoa! MIND. BLOWN. We’re not only achieving polymorphism but recursive polymorphism. While I will admit straight away there might be good use-cases for this, I haven’t seen one yet. In most real-world scenarios where you write common business logic, you don’t need this level of complexity. It will be hard to reason or understand my simple box packing code because of the underlying complexity that is hidden from the view.

Here Is My Advice When You’re Considering Creating a Protocol

• Describe your protocol as precise as you can
• Implementations for the same protocol should have similar complexity
• You probably don’t want recursive polymorphism

One last piece of advice, which is only useful if your protocol needs some helpers, is to wrap access to your protocol in a module and provide the helper functions from there. It is similar to the Enumerable protocol and the Enum module.

Did you enjoy this blogpost? Do you have something to add? Perhaps you have a different opinion you’d like to share? I’d love to hear and get some feedback! You can find my social/email at the top of the page!

Thank you Alexey G. for proof-reading this post!

I’ve been meaning to set up my blog for about three years. This time around I decided to push through and instead of using the setup to delay my writing more, use it as a force. Once it’s up and published I no longer have to configure it, make it look a specific way, figure out some new tooling, it’s already there. Ready to publish. Now the site is up before the content is created, step one completed.

Over the last 3-4 years, I have created and implemented my personal website at least 3 times. One of them was with Hugo, a static site generator that allowed me to host it as a static site. I spend days on my own custom theme to make it look the exact way I wanted. I’m not a frontend software engineer so this produced numerous challenges. One reason to do this was that I didn’t like any of the existing themes, so I decided to create my own. Bootstrap, a quite popular CSS framework wasn’t really my thing. All bootstrap sites look the same to me. So I tried Bulma, which I liked, but fiddled around with for far too long. I did like the result but it was a long and sometimes painful journey.

Fast-forward a year, or maybe one-and-a-half, and I just ditched whatever I had created earlier. I wasn’t happy with it. Not the theme I created per se, but also the static site framework. I forgot a little how it worked and having every part created myself, it required understanding what’s what to be able to make a small change. In fact, I just should’ve started writing for the blog, I ended up re-learning the framework, finding a (newer) theme I liked better. But at the end of the day still didn’t get anything done. Yes, the title of this blog post is real. After ditching it I started over with a clean slate.

What Changed -or- Something Cool Happened

The creator of Elixir open-sourced a simple library called Nimble Publisher that takes markdown files and compiles them to modules. “Why is this cool?” you might ask. While it is probably not as fast as a truly static site, it comes very close as Elixir (or Erlang) modules are loaded in memory. This means whenever you request a page, all the computer has to do is dump the right page from memory. The content still needs to be rendered on the server, which is not needed for a static site, but this should really be plenty fast. Another big plus, for both a static site generator and this solution, is that you can write your posts and pages in Markdown. You don’t need a database or fancy editor to make a good-looking page. A benefit of having Elixir backing my blog is that I’m familiar with it and I like to play with it. Instead of learning about some site generator, I’m just applying the things I’m already familiar with. Thankfully there was also a very nice blog post by David Bernheisel explaining how he set it up which I used as inspiration.

All that was left was making it look good. Going through my old attempts didn’t make me happy, I forgot mostly how the frameworks worked and I was actually longing for something that felt more like a library. You just sprinkle a little of it and suddenly your website looks good enough. I read a couple of good things about Tailwind CSS. It’s a utility-first CSS framework. Going through the initial learning curve wasn’t too bad and I noticed I’m not constantly switching between HTML and CSS to prototype a webpage. I was just adding and removing classes in the HTML first and only later abstracted tiny parts of that away in CSS. A blogpost that helped me do it was Why Tailwind by @swyx.

I’m quite happy with the result and this blog post is proof that getting it out helped me write a post. I still need to fix some small things and tidy it up, but please let me know what you think and send me your feedback!