Flapstack

Async File Uploads with Web API: Drop the 3rd Party!

Ryan Anderson -

Sleeping Under Rocks

One one the projects I am working on has a file upload work flow in the existing application (ASP.NET MVC). Three years ago when the application was built there were time constraints, so the choice was purchasing a 3rd party component to meet the requirements. It worked, granted rather wonky, being forced into cross browser support, multiple files, and time lines it fit the bill.

Fast forward 3 years and we have all kinds of new loveliness; HTML5, WEB API + "async love", beautiful JavaScript frameworks ala KnockoutJs and DISCO!!! 3rd Party File Components are Bogus too!

Coming out of the Ice Age

So there are already several articles which discuss the bad ass-ness of async file uploads with WEB API, but what is seriously batty in all of it, is actually watching the bits get written to disc while the file is still uploading via the browser (versus to memory first then to disk). Well, simply wild if you remember watching the A-Team prime time... That being said we will not dive all the way down into that but being a tease, here are some codez;

public class UploadController : ApiController
    {
        [HttpPost]
        [Route("~/api/upload")]
        public async Task<IEnumerable<string>> Post()
        {
            if (!Request.Content.IsMimeMultipartContent())
            {
                throw new HttpResponseException(Request.CreateResponse(HttpStatusCode.NotAcceptable, "Invalid Request!"));
            }

            string fullPath = HttpContext.Current.Server.MapPath("~/uploads");
            var streamProvider = new CustomMultipartFormDataStreamProvider(fullPath);

            try
            {
                await Request.Content.ReadAsMultipartAsync(streamProvider);
            }
            catch (Exception)
            {
                throw new HttpResponseException(HttpStatusCode.InternalServerError);
            }

            var fileInfo = streamProvider.FileData.Select(i =>
            {
                var info = new FileInfo(i.LocalFileName);
                return "File uploaded as " + info.FullName + " (" + info.Length + ")";
            });
            return fileInfo;
        }
    }

Things are much more straight forward here than things used be! What we are looking at here is a simple WEB API controller action which accepts a POST method call. The real power under the covers here is twofold;

  • The first is the sneakiness of the async keyword on the Post method and the await keyword when calling the Request.Content.ReadAsMultipartAsync. What this does is It avoids a blocking control flow by letting the method continue past the ReadAsMultipartAsync call before it finishes (a basic explanation is here and a much more thorough explanation here).
  • The secong is the CustomMultipartFormDataStreamProvider class which overrides System.Net.Http.MultipartFormDataStreamProvider class. Basically, it takes the body part of the Request and writes it to a FileStream if the Content-Disposition headers contains a parameter for the file name, otherwise it writes to a MemoryStream.

It is also worth mentioning here that the method is enforcing Multipart Content. On the HTML side the form is set to encoding type multipart/form-data, which allows file data to be passed in the request via the request body.

Magic!!! Well, not really but it still is bitchin!

Last point here the CustomMultipartFormDataStreamProvider class is enabling us to have control over renaming the uploaded file name and retaining the extension by overriding the GetLocaFileName method of the MuplipartFormDataStreamProvider class. This is purely an example for handling a common scenario, and is not required. Here is the override;

public class CustomMultipartFormDataStreamProvider : MultipartFormDataStreamProvider
{
    public CustomMultipartFormDataStreamProvider(string path) : base(path) { }

    public override string GetLocalFileName(System.Net.Http.Headers.HttpContentHeaders headers)
    {
        string fileName;
        if (!string.IsNullOrWhiteSpace(headers.ContentDisposition.FileName))
        {
            var ext = Path.GetExtension(headers.ContentDisposition.FileName.Replace("\"", string.Empty));
            fileName = Guid.NewGuid() + ext;
        }
        else
        {
            fileName = Guid.NewGuid() + ".data";
        }
        return fileName;
    }
}

Me and my GUID's...

Moving Right Along (lé client)

So let's take a step out from here and look at the actual client side HTML which is responsible for pushing the files to the server. The best way, IMO, to provide for a clean user experience in file uploads is using some JavaScript to push the data the the server.

Below is a solution for providing a real nice experience for our file uploads.

<h1>File Upload Demo</h1>
<div class="container">
    <div class="row">
        <div class="col-md-6 col-sm-12 col-xs-12">

            <form role="form" action="/api/upload" method="post" enctype="multipart/form-data">
                <div class="row">
                    <div class="form-group">
                        <p>
                            Testing the default behavior of Multi File Async Uploads with JavaScript and Web API
                        </p>
                        <input type="file" name="fileToUpload" id="fileToUpload" multiple="multiple" title="Browse" class="btn btn-primary btn-sm" data-bind="event: { change: fileSelected($element.files) }" />
                    </div>
                </div>
                <div class="row">
                    <button type="button" id="uploadBtn" class="btn btn-info btn-lg" data-bind="click: upload, visible: files().length > 0">
                        <span class="glyphicon glyphicon-cloud-upload"></span>&nbsp;Upload
                    </button>
                </div>
            </form>
            <br /><br />
        </div>
        <div class="col-md-6 col-sm-12 col-xs-12" data-bind="foreach: { data: files, beforeRemove: fadeOutFileItem, afterAdd: addFileItem }">
            <div data-bind="css: alertCss">
                <button type="button" class="close" data-bind="click: $parent.removeFileClick, visible: !isUploading()">&times;</button>
                <div>
                    <strong><span data-bind="text: name"></span></strong>
                    (<span data-bind="text: size"></span>
                    [<span data-bind="text: type"></span>])
                </div>
                <div class="progress" data-bind="visible: isUploading">
                    <div class="progress-bar progress-bar-success" role="progressbar" aria-valuenow="0" aria-valuemin="0" aria-valuemax="100" data-bind="text: progress() + '%', attr: { 'aria-valuenow': progress() }, style: { width: progress() + '%' }"></div>
                </div>

                <span class="label label-danger" data-bind="text: error, visible: isError"></span>
                <span class="label label-success" data-bind="visible: isComplete">Upload complete!</span>
                <button type="button" class="btn btn-default btn-sm" data-bind="click: abort, visible: isUploading">Cancel</button>
            </div>
        </div>
    </div>
</div>

Wow. A lot of moving parts here which need explanation.

First, we are using BootStrap to make "pretty". Reminds me of PHP a little in regards to its low barrier of entry, and equally as dangerous! Nonetheless, quick and easy way to get a snazzy UI thrown together.

Next, as metioned earlier we can see in the form tag we set the encode type to allow for files; enctype="multipart/form-data"

You will also notice that we have a file input field so we can select a file(s) to be uploaded. What is important here is we have allowed for multiple files to be selected via this imput field with the use of the multiple="multiple" HTML attribute and value. Because allowing for just one file to be uploaded is boring!

Lastly, you will notice this weird HTML attribute, data-bind which is used in several places in the code. If this doesn't look familiar, it is an attribute which is used in KnockoutJS. Knockout is a pretty killer JavaScript framework built by the illustrious Steve Sanderson. This framework allows us to build out some really cool client experiences using an MVVM pattern with JavaScript. This is done by observable objects, dependency tracking, and declarative data bindings. I highly recommend checking out this framework and leveraging it where it make sense on your solutions.

But wait, there's more!

The last component in all of the is the wiring. This is done via JavaScript on the client side with the help of Knockout!

The first piece of this is the ViewModel. Its role in this code allows us to bind data to the View (the HTML reviewed above).


var viewModel = function () {

    var self = this;
    this.files = ko.observableArray([]);

    this.addFileItem = function (elem) {
        if (elem.nodeType === 1) {
            $(elem).hide().fadeIn();
        }
    };

    this.fadeOutFileItem = function (elem) {
        if (elem.nodeType === 1) {
            $(elem).fadeOut(function () { $(elem).remove(); });
        }
    };

    this.removeFileClick = function (elem) {
        self.files.remove(elem);
    };

    this.fileSelected = function (files) {

        var self = this;
        for (var i = 0; i < files.length; i++) {
            if (files[i]) {
                var fileSize = 0;
                if (files[i].size > 1024 * 1024)
                    fileSize = (Math.round(files[i].size * 100 / (1024 * 1024)) / 100).toString() + 'MB';
                else
                    fileSize = (Math.round(files[i].size * 100 / 1024) / 100).toString() + 'KB';

                self.files.push(new fileModel({ name: files[i].name, size: fileSize, type: files[i].type, file: files[i] }));
            }
        }
    };

    this.upload = function () {
        var self = this;

        for (var i = 0; i < self.files().length; i++) {
            self.files()[i].startUpload();
        }
    };
};

The resppnsibilities revolve around the files to be uploaded. We have a few features in the code, add/ remove files and starting the file uploads. There is a sprinkle of jQuery in here which is debatable on whether it should be here or not, but for the sake of simplicity in a little complicated example I think we're cool. Ultimately it would be nice to abstract that code somehow to insulate our code from that dependency.

One of the most important aspects here is the files collection;

this.files = ko.observableArray([]);

The ko object here is Knockout and what we are declaring is an observable array. What we mean by observable is that if there are changes to this array, anythng which is bound to this array in the HTML will update automagically via the DOM. Very cool!

The second part is the core workable model object in our example, a file. In this object we define and implement all of the behaviors of a file upload to provide a better user experience.

function fileModel(data) {

    var self = this;

    var alertSuccess = "alert alert-success",
        alertDanger = "alert alert-danger",
        alertInfo = "alert alert-info";

    this.xhr = {};

    this.name = ko.observable(data.name);
    this.size = ko.observable(data.size);
    this.type = ko.observable(data.type);
    this.file = ko.observable(data.file);
    this.isUploading = ko.observable(false);

    this.progress = ko.observable(0);


    this.error = ko.observable("");
    this.isError = ko.computed(function () {
        return self.error() != "";
    }, this);


    this.isComplete = ko.computed(function () {
        return self.progress() == 100;
    });


    this.alertCss = ko.computed(function () {
        return self.isComplete() ? alertSuccess : self.isError() ? alertDanger : alertInfo;
    });


    this.uploadProgress = function (evt) {
        if (evt.lengthComputable) {
            self.progress(Math.round(evt.loaded * 100 / evt.total));
        }
        else {
            self.progress("unable to compute");
        }
    }


    this.uploadFailed = function (evt) {
        self.error("There was an error attempting to upload the file.");
        self.done();
    };

    this.uploadComplete = function (evt) {
        self.done();
    };

    this.abort = function () {
        self.xhr.abort();
    };

    this.uploadCanceled = function (evt) {
        self.error("The upload has been canceled.");
        self.progress(0);
        self.done();
    };

    this.done = function () {
        self.isUploading(false);
        self.xhr = {};
    };

    this.startUpload = function () {

        self.isUploading(true);

        var fd = new FormData();
        fd.append("id", "123");
        fd.append("fileToUpload", self.file());

        var request = new XMLHttpRequest();
        request.upload.addEventListener("progress", self.uploadProgress, false);
        request.addEventListener("load", self.uploadComplete, false);
        request.addEventListener("error", self.uploadFailed, false);
        request.addEventListener("abort", self.uploadCanceled, false);

        request.open("POST", "/api/upload");
        request.send(fd);

        self.xhr = request;
    };
}

Let's step through this object piece by piece as there, once again, is a lot going on here.

We start out with some properties which will allow use to persist important data for a file while we manage it through the upload process.

this.xhr = {};

this.name = ko.observable(data.name);
this.size = ko.observable(data.size);
this.type = ko.observable(data.type);
this.file = ko.observable(data.file);
this.isUploading = ko.observable(false);

this.progress = ko.observable(0);

A file object and file attributes; name, size, and type. Also, a couple of properties to communicate back to the user the status of the file upload; isUploading, and process. All of which are Knockout observable objects. The meat of the file consists of methods which are used to handle all of the events during a typical file upload like canceling an upload and upload completed events.

We make our money in the startUpload method. This method is called by the ViewModel when the user clicks then upload button (which by the way is hidden until we select a file via Knockout bindings against the files array). Here we are using the FormData JavaScript object. Mozilla summarizes this object and it's use well;

The FormData object lets you compile a set of key/value pairs to send using XMLHttpRequest. Its primarily intended for use in sending form data, but can be used independently from forms in order to transmit keyed data.

We stash a file object in a key value pair with the FormData object and name it fileToUpload (this file name can be anything you wish).

The second money maker for us here is the XmlHttpRequest object. This object was originally designed and implemented by Microsoft years ago but has since been adopted by all the big browsers out there. This object enables us to get and post data to a resource without having to refresh the entire page. It is at the core of almost all AJAX enabled sites out there.

What is really cool about this object is it allows us to wire up to a few events that enable us to provide progress back to the UI via our Knockout observable properties;

  • progress
  • load
  • error
  • abort
var request = new XMLHttpRequest();
        request.upload.addEventListener("progress", self.uploadProgress, false);
        request.addEventListener("load", self.uploadComplete, false);
        request.addEventListener("error", self.uploadFailed, false);
        request.addEventListener("abort", self.uploadCanceled, false);

        request.open("POST", "/api/upload");
        request.send(fd);

        self.xhr = request;

We map each of these event back to functions within our File object and handle them accordingly. As stated, this enables us to provide a much smoother user experience, mainly to provide upload progress via a progress bar which we tick up each time this event is fired during the upload.

Wrapping up

File uploads are a very common part of providing functional user work flows on the web. Historically, there have been issues ranging from cross browser compatibility, lagging standards, and various other issues which may have prevented us from "rolling" our own file upload feature.

However, over the last few years, the more rapid adoption of HTML5 and other browser standards, and better JavaScript frameworks have lowered the barrier to implementing your own killer solution. The scope of technologies used here from WEB API to Knockout may seem a little overwhelming, however, the gained control and flexibility in rolling your own solution can outweigh the 3rd Party component option.

Source Code

https://github.com/ryanande/AsyncFileUploadDemo

Up Next! => Silliness

In our next post, we will flip this on it's head and do something really silly and Stream the file uploads to a MongoDB, MongOhNoYouDidn't!