Stream content to Google Cloud Storage

Google Cloud Storage posted on 8th July 2019


In this project, I need to get videos from Vimeo to Google Cloud Storage in order to run the Video Intelligence API on them.  There are a number of ways to do that, but I'd like to simply stream them directly using a cloud function.

This article looks at the Node implementation for streaming directly to cloud storage, and we'll look at how to covert that to a Cloud run function in a later one. Note that although I'm streaming videos, this technique could work with any kind of file.

Credentials

First we need a service account, a bucketname and a folder that will be the target for the video files loaded from Vimeo. I've covered getting service accounts elsewhere so let's assume you already have a downloaded service account .json file. I keep all my configs on one place, organized like this, with different configurations for different scenarios. These are referenced in the code so here's a redacted version
ns.storage = {
kp: {
serviceAccountFile: "./creds/fxxxxxxd.json",
bucketName: "fxxxxxxs",
folderName: "fxxxxxxxxg"
},
pv: {
serviceAccountFile: "./creds/fyyyyyyyd.json",
bucketName: "fyyyyyyyys",
folderName: "fyyyyyyyyyyyg"
}
};

ns.getGcpCreds = () => ({
credentials: require(ns.storage[global.fidRunMode].serviceAccountFile),
bucketName: ns.storage[global.fidRunMode].bucketName,
folderName: ns.storage[global.fidRunMode].folderName
});

Vimeo download files

I have a note of these in my database, but for simplicity I'll just name the video file i want to load in my app. You'll want the downloadable (as opposed to playable) distribution file. If like me, you plan to run the video intelligence API, the higher the quality, the more accurate the labelling will be.

Dependencies

There are not many - you'll need to install them with npm or yarn.
const mime = require('mime-types');
const Storage = require('@google-cloud/storage').Storage;
const request = require('request');

Local dependencies

In my case, I'll need to set the  mode I'm running in (would normally do this through environment variables, but for this demo I'm setting a global) and pick up my configuration secrets.
const request = require('request');
global.fidRunMode = "kp";

Setting up streaming from vimeo

The idea here is that we do an http request to the vimeo download link and immediately pipe the response to another stream which is connected to a cloud storage file.
/**
* stream video file direct from vimeo storage to google cloud storage
* @param {string} url the file to stream
* @param {stream} stream the stream to pipe it to
* @param {cb} function the callback when done
*/
const downloadFile = (url, stream , cb) => {

// request the video files
const sreq = request.get(url);

// check it worked
sreq.on('response', (response) => {
if (response.statusCode !== 200) {
return cb('Response error ' + response.statusCode);
}
// and pipe the output to the blob stream for cloud storage
sreq.pipe(stream);
});

// close when done
sreq.on('finish', () => stream.close(cb));

// need to report any errors
sreq.on('error', (err) => {
return cb(err.message);
});

};

Setting up streaming to Cloud Storage

By assigning a blob to a cloud storage file and associating a writestream with that blob, the data is magically piped from the input http stream.
/**
* create the cloud storage stream
* the credentials/bucket name and filename are in the secrets file
* @param {string} fn the filename on cloud storage
*/
const createStorageStream = async (fn) => {
const type = mime.lookup(fn);
console.log('creating ', fn, type);
// get storage params
const creds = secrets.getGcpCreds();
const { credentials, bucketName, folderName } = creds;
const { projectId } = credentials;
  const storage = new Storage({
    projectId,
credentials
  });

const bucket = storage.bucket(bucketName);

// we'll actually be streaming to this blob
  const blob = bucket.file(folderName + '/' + fn);
  const stream = blob.createWriteStream({
    resumable: true,
    contentType: type
  });
// this stream will be piped to
return stream;
};

Kicking it off

Just give it a file name on cloud storage, and the input url and that's all there is to it.
// do the thing
(async () => {
// a downloadable vimeo link
const fn = 'https://player.vimeo.com/external/28xxxxxxxxxxx&download=1';

// create the target stream
const storageStream = await createStorageStream('40.mp4');

// get the file
downloadFile(fn, storageStream, (r) => {
console.log('created ', fn);
});
})();

Learning Apps Script, (and transitioning from VBA) are covered comprehensively in my my book, Going Gas - from VBA to Apps script, All formats are available from O'ReillyAmazon and all good bookshops. You can also read a preview on O'Reilly

If you prefer Video style learning I also have two courses available. also published by O'Reilly.
Google Apps Script for Developers and Google Apps Script for Beginners.

Comments