tldr: source code and deployment instructions
Motivation
If you’ve worked for a software company, you’re probably familiar with build servers. There are lots of reasons to use them. When code is built and merged into production, it is usually done so in a standardized environment on dedicated servers called build servers (think tools like Jenkins, CircleCI, etc).
Lately, AWS Lambda and Google Cloud Functions have been gaining popularity because they allow developers to think more about code and less about servers while paying strictly for the memory and time that they consume. Google recently announced very generous free offerings for most of their cloud products, including a hefty allowance of Cloud Functions usage per month.
Since Cloud Functions execute code in Node.js environments, it should be possible to build applications that build in Node.js environments (think all Webpack frontends, like React and Angular apps to name a few). So I decided to build a Cloud Function that could build frontends built in Node.js.
How It Works
The Cloud Function is triggered via HTTP call (think: Github webhooks). At a high level, the steps are pretty simple:
- Download source code — simply clone a Github repo.
- Run the build — execute a build command that you would execute on the command line, like npm run build.
- Package the build results — since the build might output multiple files, it is easiest to package all those results into a tarball and compress it so it can be pushed somewhere else…
- Push the results — Cloud Functions execute in temporary environments, so the build results must be pushed somewhere persistent to be utilized later. Having them packaged and compressed makes this easier and saves on network egress. I chose to push the packaged build to GCS, but it can really be pushed anywhere.
The Numbers
I tested building a basic app created by Create React App. I could only get this to run with the 2GB of RAM, 2.4 GHz configuration. Every run took less than 2 minutes. Thus, according to the Google Cloud Functions always free limits, CPU resources would run out long before RAM.
200000 free GHz-seconds / (2.4 GHz * 120 seconds per run) = ~694
This means it is very possible to run ~694 front-end builds for free on Google Cloud Functions every month. Given the pricing model of Cloud Functions, I’m not sure how many builds would make it worth running a dedicated build server.
Challenges with Cloud Functions
This was my first time building anything on Cloud Functions, and anything serverless in general. Given that Cloud Functions is in beta, I’m sure it will continue to evolve in great ways. Here were some of my biggest challenges building this project:
No Filesystem
Unfortunately, Cloud Functions does not really give you access to a filesystem. They do give you access to /tmp, but it seems to count against your allocated RAM which maxes out at 2GB. This is an issue when downloading a compressed tarball from Github, unzipping it, untarring it, writing to disk, and installing all Node.js build dependencies for that project. I’m not sure how much RAM this actually takes, but naively it could upper bound at ~4x the codebase size + build deps.
Emulator vs. Production
There are some differences between the Cloud Functions emulator and Cloud Functions production environment that made development difficult. While the emulator is developed by Google, they do give a disclaimer about it not being an official Google product so this is fair.
One example is executing commands using Node.js’ child processes. Since the emulator does not seem to run an isolated environment, depending on executables being in $PATH can work on the emulator but not in production.
I ended up mimicking git clone using Github Contents API because I could not get the top few Node.js git packages to work on production even though they worked on the emulator. Common issues included needing basic crypto libraries installed and needing git installed.
Deployments are Slow
Deployments always took a few minutes, even to deploy simple cloud functions. This made debugging emulator vs. production issues very time-consuming, because I had to redeploy for every little change (and I had a lot of them).
No Node.js Executables
Being able to build Node.js applications requires access to the node and npm executables. Given that Cloud Functions are executed in a Node.js environment, I figured executing node code would be trivial — I was wrong (kind of). While I could execSync node commands in the emulator, production could not find node or npm executables. My first course of action was to see if I had any executables — which ls — yep, that worked. echo $PATH — seems there were a few directories in $PATH — good. find / -name node -executable — for some reason this didn’t return anything (not sure why), but I had other ideas. I knew that C allows you to see the calling command via argv, and turns out node does as well. Excellent, I found the node executable (/nodejs/bin/node)! I listed the contents of the containing directory, and sure enough npm was in there too. I added that directory to $PATH and was back on track.
Future Ideas
- Cache node_modules for the project being built. Every build currently runs npm install, which takes ~60% of the build time. Cutting down on this time would greatly decrease the cost of each build.
- Switch from node-tar to tar-fs. tar-fs can apparently be 10x faster than node-tar, but unfortunately requires chmod. chmod is not in $PATH by default on Cloud Functions, so I assume it is not available. It would be nice to get tar-fs working in the future.
Hacker Noon is how hackers start their afternoons. We’re a part of the @AMIfamily. We are now accepting submissions and happy to discuss advertising & sponsorship opportunities.
To learn more, read our about page, like/message us on Facebook, or simply, tweet/DM @HackerNoon.
If you enjoyed this story, we recommend reading our latest tech stories and trending tech stories. Until next time, don’t take the realities of the world for granted!