Packaging and unit testing API scripts

With all the the stuff that's going on around API scripts at the moment, I thought it might be an opportune moment to raise this: has anyone given thought to how API scripts can nicely be packaged up and unit tested using standard tools (NPM, node, gulp etc)? One of the things that makes this uncomfortable at the moment is that API script files do not specify their own dependencies - the closest thing we have is a package.json, which unhelpfully nameclashes with the node package spec file, and  doesn't interact with any standard tools (e.g. node itself) Really, I'd like to be able to run my Roll20 scripts inside a mocha suite under gulp; and I'd like to be able to require() my dependencies explicitly from with a script (which would be limited to underscore, roll20's own library (for which we could make a test proxy) and any other API scripts upon which we depended). At the moment the best way of achieving this seems to be to simulate the behaviour of the API server and cludge together a big JS file out of a bunch of fragments. I guess this is OK, but it's fragile and deeply non-standard. I guess it's probably a little late given that the one-click stuff is already rolling out, but is there any chance that the way this is handled could be aligned more closely with industry standards? It seems unfortunate, to say the least, the there's a totally custom solution that doesn't interact with any of the available tools when there are well-understood, widely-supported frameworks that do most of this stuff already. What was so wrong with require, module.exports and npm dependencies that necessitated a completely homegrown alternative? Cheers, Lucian

Lucian H. said: I guess it's probably a little late given that the one-click stuff is already rolling out, but is there any chance that the way this is handled could be aligned more closely with industry standards? It seems unfortunate, to say the least, the there's a totally custom solution that doesn't interact with any of the available tools when there are well-understood, widely-supported frameworks that do most of this stuff already. What was so wrong with require, module.exports and npm dependencies that necessitated a completely homegrown alternative? Actually, I think this is exactly the time to talk about this sort of thing. It's several years  late to be asking for something like Node's require(), since the system wasn't setup for modularized user scripts like that (all API scripts in a campaign get concatenated together), but now that the one-click is in development with open feedback, it's certainly appropriate to talk about a formalized and potentially automated packaging process. (And talking about this issue before the devs began work on the one-click install would have been too premature to be particularly useful.) I think the most useful (and most likely) option at the moment would be a third-party tool to setup the directory structure & json data appropriately, update version numbers, and possibly git push and/or git request-pull.

Brian said: I think the most useful (and most likely) option at the moment would be a third-party tool to setup the directory structure & json data appropriately, update version numbers, and possibly git push and/or git request-pull. Yeah, I guess that's probably the most I can realistically hope for. But that will be limited to whatever functionality that (custom written) tool provides. The great thing about doing things with standard toolchains is that you can drop in lots of other functionality for free - so you don't have to manually wire up your code-quality checker, your unit test framework, etc, etc. I guess I'm a bit disappointed that this current phase of development (which will involve script authors doing a bunch of work anyway) wasn't taken as an opportunity to port everything over to using well-established tools rather than carrying on creating custom solutions that are likely to be less functional, buggier and have poorer integration options than the mainstream open-source alternatives. This is a solved problem into which smart people have poured thousands of man hours of development; with the best will in the world, Roll20 are unlikely to come up with anything that matches up :-/ Ah well.

What do you think the easiest first step would be? I'm wondering if there's a relatively easy way to hack a version of require() here that would look for through the available scripts for that campaign and make available anything with a module.exports line—at the least that would make it easier to pull in node functionality that we might want (for example, using&nbsp; <a href="https://github.com/substack/minimist" rel="nofollow">https://github.com/substack/minimist</a> for parsing commands). That would still lack extended dependency support, but it would be a start, and I'd like to think it wouldn't require tons of work on the roll20 side to make possible.

Tadhg said: What do you think the easiest first step would be? I'm wondering if there's a relatively easy way to hack a version of require() here that would look for through the available scripts for that campaign and make available anything with a module.exports line—at the least that would make it easier to pull in node functionality that we might want (for example, using&nbsp; <a href="https://github.com/substack/minimist" rel="nofollow">https://github.com/substack/minimist</a> for parsing commands). That would still lack extended dependency support, but it would be a start, and I'd like to think it wouldn't require tons of work on the roll20 side to make possible. Since writing my original post I have partially solved this problem - sorry should have written an update. The answer for me is webpack. Take a look at&nbsp; <a href="https://github.com/symposion/roll20-shaped-scripts" rel="nofollow">https://github.com/symposion/roll20-shaped-scripts</a>... for inspiration. I have written a couple of simple node modules that are on npm now - roll20-wrapper and roll20-logger - and I require these as normal from my project, which is modularised into files like a normal node module would be. Everything is vanilla node, with require, gulp build, mocha unit tests and Travis CI etc. The only difference is that the final build step uses webpack to build it into a single script as if it was going to be loaded into a browser rather than run under node. Obviously these still doesn't achieve inter-script dependencies within a campaign, but I believe that the Scriptomancer may be working on something that will make this less important in the long run.

I personally have very little experience with JS and can therefore add not much to the discussion, but from a programmer view I wholehearthly support this!