Connecting to a FTP server with Elixir

One of my consulting specialties is “data plumbing”, which includes building data extractors, connectors and data pipelines.

While I use Ruby and Kiba ETL a lot for these tasks, I also use other stacks (e.g. Elixir, Rust or Go), most notably on my most recent ongoing gig, the French national access point to transport data, a portal gathering a lot of transport-related open-data.

For that gig I recently had to “spike” how to connect to FTP (not SFTP) server with Elixir. I’m sharing my findings here.

How to connect & download a file?

Given a FTP url, how to retrieve a file, for instance?

The first step is to parse the URL into individual components:

%{
  host: host,
  scheme: "ftp",
  port: port,
  userinfo: userinfo,
  path: path
} = URI.parse(url)

From there, we can start the Erlang ftp application:

:ftp.start()

Starting the service requires the host, but you have to convert the Elixir string to Erlang with String.to_char_list (or you’re in for quite confusing error messages):

{:ok, pid} = :ftp.start_service(host: host |> String.to_char_list())

You can read more about Charlists in Elixir documentation:

A charlist is a list of integers where all the integers are valid code points. In practice, you will not come across them often, except perhaps when interfacing with Erlang, in particular when using older libraries that do not accept binaries as arguments.

We are in the exact use-case where we need to interface with Erlang, since the ftp app is an Erlang app.

The user info (typically user:pass) must be massaged and provided, if available:

if userinfo do
  [user, pass] = userinfo |> String.split(":")
  :ftp.user(pid, 
    user |> String.to_char_list(),
    pass |> String.to_char_list()
  )
end

As @lostkobrakai pointed out to me, one can use the ~c sigil to make this shorter:

:ftp.user(pid, ~c(#{user}), ~c(#{pass}))

From there it is easy to download a file to the disk:

# extract filename from url
target_filename = Path.basename(path)

:ok = :ftp.recv(pid, ~c(#{path}), "downloads" <> target_filename)

And we just have to shut-down everything once done:

:ftp.stop_service(pid)
:ftp.stop()