Run untrusted Javascript

Cost-saving, high performance and easy maintenance.
This are just a few of the benefits of multi-tenant SaaS apps.
But today we shall talk about one of the problems.
How to change the app behavior for a single client.

Or in other words can we safely run untrusted code on the server?

Sure we can. In this blog we shall look into safely running js code in NodeJS.

Terminology

By multi-tenant - we mean an architecture where you have a single app that is used by all clients.
Like an apartment building with multiple tenants - hence the name.

Of course, in practise, you might have multiple servers for fault tolerance or performance.
But each of the servers is running the same app.
Simply put, we don't have tenant specific apps or servers.

Usually, we provide each tenant with a separate database, but this is not strictly required.
You could implement multiple tenants in a single db, just by putting an tenantid column on each db table.
I'm sure you can imagine the headaches, if you ever need to add a tenant specific column.
Or restore data for a single tenant from backup.
So while it is possible to do, and it has a few benefits (size, schema updates), it is not, in our experince, worth the hassle.

Motivation

We wish to execute client specific code in response to some events.
Basically reacting to data-changes, running a simple calculation and sending the results somewhere (like back into the database, or email).

For example: Let’s say we have a customer table and an order table.
And each time an order is created we want to increase the total sum of orders on the customer and send an email if a threshold is reached.

Constraints

• Maximum security. The code cannot see or touch other tenants.
• Safety, the code has to run within limits.
• Focus on small code that is run often.
• Fast deployment. Code will change with business requirements and thus has to be deployable at any time and quickly
• Cost friendly. We use SaaS to share resources, thus having a separate app will not do.

We believe that long running scripts should be run in a dedicated environment.
The wast majority of the code we have to support is small and runs often.

Options

We have a few options on how to implement a solution.

1. Run the code on the client.
2. Add the code to the server for all tenants. Add some configuration mechanism to tweak the parameters.
3. Add the code to the server but only run it for s single tenant. Eg each time an order is updated we check whether the tenant is the right one.

Client side

Possible for some scenarios.
For security reasons we cannot run just any calculation in the client’s browser.
And some functionality is just not available, for example sending emails.

Server side - all tenants

Create a feature for all tenants. Which means we have to have a way for tenants to configure the feature.
We need tests and documentation.
Probably no worth for small code scripts, that are relevant to a single tenant.

Server side - for a single tenant

By now, it is obvious this is the right answer.
Let's review what do we need.

• How do we deploy such code? Hardcoded ifs per tenant?
• How do we make sure it is not using too many resources?
• How do we make sure it is safe?

Basically we want a solution that is safe and easy to use.

• That means no server restarts
• buggy code can’t crash the server (on purpose or by mistake)
• No data leaks, code cannot access other tenant data.
• Resource limits, so that the code cannot overload the server or make the service slower for others.

Solutions:

We shall put the code in the tenant database along with some description of when to run it. For example: run on sales-order-create.

Then we need a sandbox environment where we can control what the code can do.
In particular what data it can access, how much processor and memory it can consume.

Non-startes

That will eliminate the obvious solutions like using eval or new Function(). Please don’t use that.

Nodejs does allow you to start a new environment and there are even solutions that try to build a sandbox out of it. Unfortunately, it was never meant for this use and of course critical bugs were found. jailed

Server-less

Multiserver solution or server-less solutions. We could start an azure function or amazon lambda and execute the code inside.
Very clean, but massive overhead (and cost) if invoked too often. This makes sense for long running and complex scripts, especially if we can shift the cost to the tenant.

Deno, Bun,

AS you probably know already these are new JS runtimes, similar to NodeJS.
Deno was built with security in mind, thus you can lock it down pretty well.
Still it means having a separate server - with the additional communication overhead. And while we can lock down what the JS in Deno can do, we still have to add cpu and memory limits somehow.

Custom runtimes

Custom language runtime - lua. If you have the option of choosing the language then using the lua runtime is potentially a good choice.

Javasript runtimes

For our purposes we prefer JS - it is widely used and approachable. Since we don’t want to give the JS code to nodejs, we need another interpreter.

Figma guys use the ducktape C library (can be compiled to webassembly).
You can read about their journey here it is worth it.

Another library by a great author (ffmpeg fame) is quickjs

For maximum control, security and easy of deployment we will use a JS interpreter written in JS.

Solution outline

1. We will handle script calls async.
2. Each time a script has to be called we will add a record to the async-actions db table.
3. We will run a separate nodejs thread
4. In this thread we will read the async-actions records and execute them one by one.
5. We will create a bridge for db access for the thread.

We shall call this method whenever we write into the database.

// method called after a record was changed in the database (created, update or deleted) 
export const onRecordChanged = async (db: DatabaseConnection, user: {id: string}, operation: string, objectName: string, id: string, prevRecord: any, record: any) => {

	const triggers = Triggers[db.orgName];
	if (triggers) {
		const trig = triggers.find(x => x.objectname === objectName && x.operations && x.operations.indexOf(operation) >= 0);
		if (trig && trig.methods === "plugin") {
			await createAsyncActionRecord(db, user.id, id, operation, objectName, record, prevRecord, trig);
		}
	}
}

This is the asyncActionsServer.ts file. It will launch the handler thread and handle the communication.

import { Worker, isMainThread } from "worker_threads";

export interface DataTrigger {
	id: string;
	name: string;
	objectname: string;
	operations: string; // create;update;delete or create only.
	recipient: string; // ; separated @owner, @admin, email, systemuser guid
	methods: string;
	ratelimit: string; // unique field.
}

export const createAsyncActionRecord = async (db: DatabaseConnection, userid: string, objectid: string, operation: string, objectname: string, record: any, prevRecord: any, dataTrigger: DataTrigger) => {
	
	const params = {}
	// ... turn arguments into params
	return db.insertRow("inu_asyncaction", params);
}

const scheduleAsyncActionsMainThread = (orgName: string) => {
	if (isMainThread)
		sendWorkerMessage({ id: "runjobs", orgName: orgName });
	else {
		console.log("ERROR: scheduleAsyncActionsMainThread called on background thread");
		throw Error("ERROR: scheduleAsyncActionsMainThread called on background thread");
	}
}

const mainHandleDataRequest = async (msg: IThreadMessage) => {
	if (msg.requestid === undefined || !msg.data) {
		console.log("ERROR messageFromThread:" + JSON.stringify(msg));
		return;
	}

	let db;
	try {
		console.log("BEGIN messageFromThread");
		db = await dbMgr.connect(msg.orgName, "asyncActionsServer");

		const requestData = JSON.parse(msg.data);
		let resultData = "";

		switch (msg.id) {
			case "query": {
				const query = requestData;
				const results = await db.query(query);
				resultData = JSON.stringify(results);
				break;
			}
			case "execute": {
				const records = nonQuery;
				const results = db.execute(nonQuery)
				resultData = JSON.stringify(results);
				break;
			}
			//...
		}
		sendWorkerMessage({ id: msg.id, orgName: msg.orgName, requestid: msg.requestid, data: resultData });
	}
	catch (err) {
		console.log("EXCEPTION messageFromThread: " + err);
		try {
			sendWorkerMessage({ id: msg.id, orgName: msg.orgName, requestid: msg.requestid, error: "" + err });
		}
		catch (err2) {
			console.log("CATASROPHIC EXCEPTION: " + err2);
		}
	}
	finally {
		db?.release();
	}
	console.log("END messageFromThread");
}

let worker: Worker | undefined = undefined;

const sendWorkerMessage = (msg: IThreadMessage) => {
	if (isMainThread) {
		if (!worker) {
			worker = new Worker(__filename.replace("asyncActionsServer", "asyncActions"));
			worker.on("message", mainHandleDataRequest)
		}
		worker?.postMessage(msg);
	}
	else
		throw Error("Cannot send message from background worker!")
}

And now let's look at the worker thread.
First the communication part.

import { parentPort, isMainThread } from "worker_threads";

if (isMainThread) {
	// create on-demand
	//worker = new Worker(__filename);

	console.log("ERROR: AsyncActions must not be called on foreground thread");

} else {
	parentPort?.on('message', (msg: IThreadMessage) => {
		if (msg.id === "runjobs") {
			scheduleAsyncActions(msg.orgName);
		} else {
			threadReceiveDataResponse(msg);
		}
	});
}

const PROMISE_MAP: { [key: number]: { res: (data: any) => void, rej: (err: string) => void, parseJson: boolean } } = {};
let PROMISE_COUNTER = 0;

const threadSendDataRequest = async (orgName: string, operation: "retrieveMultiple" | "executeMultiple" | "updateActionRecord", data: any, parseJson: boolean = true) => {
	if (parseJson)
		data = JSON.stringify(data);

	return new Promise<any>((res, rej) => {
		const requestid = ++PROMISE_COUNTER;
		PROMISE_MAP[requestid] = { res, rej, parseJson };
		const msg: IThreadMessage = { id: operation, orgName: orgName, requestid: requestid, data: data }
		parentPort?.postMessage(msg);
	})
}

const threadReceiveDataResponse = (msg: IThreadMessage) => {
	let promise = PROMISE_MAP[msg.requestid!];
	if (!promise) {
		console.log("threadReceiveDataResponse: got response to unknown request:" + JSON.stringify(msg));
		return;
	}
	delete PROMISE_MAP[msg.requestid!];

	if (msg.error)
		promise.rej(msg.error);
	else
		promise.res((msg.data && promise.parseJson) ? JSON.parse(msg.data) : msg.data);
}

const ORG_LOCK: { [name: string]: number } = {};

const scheduleAsyncActions = (orgName: string) => {

	if (ORG_LOCK[orgName]) {
		ORG_LOCK[orgName]++;
		return;
	}
	ORG_LOCK[orgName] = 1;
	/* no await */ doScheduledAsyncActions(orgName);
}

const threadRetrieveMultiple = async (orgName: string, fetch: Fetch | string, parseJson: boolean = true) => {
 	return threadSendDataRequest(orgName, "query", fetch, parseJson);
}

const threadExecuteMultiple = async (orgName: string, requests: IExecuteRequest[] | string, parseJson: boolean = true) => {
	return threadSendDataRequest(orgName, "execute", requests, parseJson);
}

const updateActionRecord = async (orgName: string, action: IAsyncAction) => {
	return threadSendDataRequest(orgName, "updateActionRecord", action);
}

const doScheduledAsyncActions = async (orgName: string) => {
	let reschedule = false;

	try {
		console.log("BEGIN doScheduledAsyncActions");
		
		const actions = await threadRetrieveMultiple(orgName, ACTIONS_QUERY) as IAsyncAction[];
		if (actions && actions[0]) {
			const action = actions[0];
			
			// FIXME: race-condition if we ever have multiple servers for a single org.
			// We should do a conditional write with the versionnumber (or just use updateRecord...) 
			// and if it fails, somebody else is handling the record, so try the next one.
			action.startdate = new Date().toISOString();
			action.status = "done";
			await updateActionRecord(orgName, action);

			try {
				if (!action.ch__conditions)
					throw Error("Async action has no body:" + action.scriptid);

				let result = "ok";

				await executeAction(orgName, action);

				action.enddate = new Date().toISOString();
				action.result = result;
				await updateActionRecord(orgName, action);
			}
			catch (e) {
				action.enddate = new Date().toISOString();
				action.error = "Exception: " + e;
				await updateActionRecord(orgName, action);
			}

			reschedule = true;
		}
	}
	catch (e) {
		console.log("ASYNC OP FAILED:" + e);
	}
	finally {
		console.log("END doScheduledAsyncActions");
		reschedule = reschedule || ORG_LOCK[orgName] > 1;
		ORG_LOCK[orgName] = 0;

		if (reschedule) {
			setTimeout(() => scheduleAsyncActions(orgName), 1);
		}
	}
}

The code below executes the untrusted JS.

To construct the interpreter we pass the code to be executed as text and globals.
It is only through the globals that the code can communicate with the outside world.
The code has no access to NodeJS apis or the window globals or any packages.
Complete isolation, which is exactly what we want.

Keep in mind that objects cannot be shared between our outside code and the code in the interpreter.
Only primitive values, numbers and strings can be passed.
Which is fine, we just JSON our way through it.

Finally, to execute the code we just call interpreter.step() in a loop.

We have a fixed 5 minute execution limit, the script is stopped if the limit is reached.
We do a tiny bit of timekeeping - not counting the time the code waits for data.

createAsyncFunction is a very neat feature of the interpreter.
It exposes an async function as a synchronous function into the interpreter.
That means, the code within the interpreter sees it as a normal method, while the implementation is async, and just calls as callback when done.
To make sure our execution loop (calling step()) is not running if not needed, we use an asyncPromise variable.
We set the variable when we call async functions.
In the step loop we check whether the interpreter is paused_. This flag is set by the interpreter when an createAsyncFunction is called.
Then we just wait on the asyncPromise.

The rest of the code simply prepares a nicer environment for the script, hiding the JSON serialization etc.

Final word of warning. The interpreter only support JS5, thus some newer features like async are not available.
Babel or Typescript can transpile your JS6 code easily, so this wasn't a showstopper for us.

const executeAction = async (orgName: string, action: IAsyncAction) => {
	let result: any | undefined = undefined;

	let asyncPromise: Promise<void> | undefined = undefined;

	const __retrieveMultiple = (fetchString: string, callback: (result: any) => void) => {
		asyncPromise = new Promise<void>(async (res, rej) => {
			try {
				const json = await threadRetrieveMultiple(orgName, fetchString, false);
				callback(json);
				res();
			}
			catch (e) {
				rej(e);
			}
		});
	}

	const __executeMultiple = (requestString: string, callback: (result: any) => void) => {
		asyncPromise = new Promise<void>(async (res, rej) => {
			try {
				const json = await threadExecuteMultiple(orgName, requestString, false);
				callback(json);
				res();
			}
			catch (e) {
				rej(e);
			}
		});
	}

	const __sendEmail = (emailData: string) => {
		const email = JSON.parse(emailData);
		sendEmail(email.to, email.cc, email.bcc, email.subject, email.body);
	}

	const code = `

var context = {};
context.retrieveMultiple = function (fetch) {
	var fs = JSON.stringify(fetch);
	var fr = __retrieveMultiple(fs);
	var result = JSON.parse(fr);
	return result;
}
context.executeMultiple = function (requests) {
	var fs = JSON.stringify(requests);
	var fr = __executeMultiple(fs);
	var result = JSON.parse(fr);
	return result;
};
context.sendEmail = function(to, cc, bcc, subject, body) {
	var p = JSON.stringify({to:to,cc:cc,bcc:bcc,subject:subject,body:body});
	__sendEmail(p);
	return 1;
};
context.action = JSON.parse(action);
context.record = JSON.parse(context.action.record);
context.record.id = context.action.objectid;
context.prevRecord = (context.action.prevrecord && JSON.parse(context.action.prevrecord)) || {};

` + "context.userId = " + JSON.stringify(action.userid) + `;
context.orgName = "`+ orgName +`";`;
	const js_code = code + (action.ch__conditions || "");
	try {
		const ctx = new Interpreter(js_code, (i: Interpreter, globalObject: any) => {
			i.setProperty(globalObject, '__retrieveMultiple', (i as any).createAsyncFunction(__retrieveMultiple));
			i.setProperty(globalObject, '__executeMultiple', (i as any).createAsyncFunction(__executeMultiple));
			i.setProperty(globalObject, '__sendEmail', i.createNativeFunction(__sendEmail));
			i.setProperty(globalObject, 'action', JSON.stringify(action));
			i.setProperty(globalObject, 'setResult', i.createNativeFunction((r: boolean) => (result = r)));
			i.setProperty(globalObject, 'log', i.createNativeFunction((text: string) => console.log(text)));
		});
		let steps = 0;
	
		let startTime = process.hrtime();
		let totalStartTime = startTime;
		while (true) {
			if (ctx.paused_ && asyncPromise) {
				await asyncPromise;
				startTime = process.hrtime();// restart timer after async. Other code had a chance to run.
			}
				
			if (!ctx.step())
				break;
			steps++;

			if ((steps & 0xfff) === 0) {
				const endTime = process.hrtime(startTime);
				if (endTime[0] > 0 || endTime[1] > 1e8) { // if more than 1s or more than 0.1s -> yield
					// let something else run
					await new Promise<void>((res, rej) => {
						setTimeout(() => res(), 1);
					});
					startTime = process.hrtime();
					console.log("SyncAction: yield");
				}
				const totalTime = process.hrtime(totalStartTime);
				if (totalTime[0] > 300) {
					throw Error("SyncAction time budget of 5 minutes exceeded.");
				}

				console.log("SyncAction: running for: " + (endTime[0] + (endTime[1] / 1e9)));
			}
		}
	
		{
			const endTime = process.hrtime(startTime);
			console.log("SyncAction: completed in: " + (endTime[0] + (endTime[1] / 1e9)));
		}
		
		if (ctx.value !== undefined)
			result = ctx.value;
	}
	catch (e: any) {
		if (e.loc && e.loc.line && e.loc.column) {
			const lines = js_code.split('\n');
			const line = lines[e.loc.line - 1];
			const errText = "SYNTAX ERROR\n" + line + "\n" + ' '.repeat(e.loc.column - 1) + "^--here\n";
			throw Error(errText);
		}
		throw e;
	}
	
	return result;
} 

Final words

Allowing tenants or partners to add custom code to your solution is a massive feature.
Something, that can make your app stand out.

What's even better, you can just use it in-house for per-client customization.
Additionally, with a clean API for client specific code, your core code will be cleaner and simpler.

What do you think, is it worth your time? Let us know!

Happy hacking!