API speed question

So I have scripts I've written for a turn based (originally miniatures based) wargame we're playing on Roll20. While things work, I am trying to optimize for speed.  Background: Tokens on the map represent individual squads of infantry or individual armour units. 'Character Sheets' are basically the unit information with things like armour, weapons information etc. Terrain on the map is 'outlined' by lines that are used (like lines in the DLL) to define the terrain. I use ray-tracing math (intersection of lines between shooter and target and terrain lines) or 'point in polygon' math to determine what terrain a target is in (for cover purposes). This generally means I am doing multiple findObjs for the tokens attacking/being attacked (multiple, as the game is unit based - so a formation of say 7 tanks fires on another formation of 4 tanks for instance) and then getObjs to get information on the various units abilities. I do store the terrain line information in the state variable, as it doesn't change during the game, as well as using state to store certain other information such as the list of tokenIDs in a given formation. I have optimized my LOS and terrain functions - rather than iterating through all terrain I use bounding boxes to only iterate through terrain in the area the LOS line 'crosses' or that a token could be in then running the polygon math to check if it really is. Typically there are 150-200 tokens on a map to start, representing perhaps 50 unit types (so 50 characters).  Problem: However, I still find some slowness in the API, almost as if it hesitates when first starting up then running the code. Before I streamlined some of the code, I could get an 'infinite loop' error in large pitched battles, due to timeout rather than an actual infinite loop.  So my questions for those with more knowledge than I: - is there a way to find out how long a given routine or section of code takes to run? - I have thought of doing all the initial findobjs/getobjs when the game is 'setup' (before any turns are done), and storing all relevant info in state - but am not sure if that really speeds things up - ie. the lookup from an array in state vs. a findObjs or getObjs - as I am not sure how long the relevant lookups take - this last would also require me using an on change event each time a token is moved to update the X,Y coordinates in the state array - however I figure that doing this wouldnt be noticeable given only one token is moved at a time - alternately I could do a mix - store character(unit) information in state and then just use findObjs when I need to do LOS/distance calculations on tokens Thoughts and comments welcome! Don

I think I found an answer for my first question from Aaron on another forum post - finding the time a function or section of code takes. He had this code which I will now be inserting various places to see where time is taken in various functions. let before,after; before=_.now(); doSomething(); after=_.now(); log(`Elapsed time for 'doSomething()' was ${ (after-before)/1000 } seconds`);

Note that the code posted above is fairly simple and won't work with asynchronous functions (which you'll recognize by their callback function arguments, generally speaking).&nbsp; If I wrote that code today, I'd probably use: const start = Date.now(); // code under test const elapsed = Date.now()-start; log(`Elapsed time in seconds: ${elapsed/1000}`); I find myself preferring to use native calls in ES6 and on. &nbsp;General thoughts on the above in no particular order: When you say state , do you mean the persisted global variable state ?&nbsp; The state object has a limited size and is persisted to the database at some interval.&nbsp; You would want to limit the information you store in it (people have stored whole spellbooks in it in the past and it caused quite a bit of problems when it got overloaded).&nbsp; Keeping grouping of tokens in the state (and a mapping from token ID to group and back) sounds great.&nbsp; I would not put data like all of your lines in the state.&nbsp; I'd put that in an object you build at startup.&nbsp; Ditto for often accessed character data, like ability descriptions (really, I'd probably build a lazy load function that caches all that as you use it). "Possible Infinite Loop Detected" happens when script code executes for too long without yielding to the sandbox code.&nbsp; Javascript effectively has cooperative multitasking.&nbsp; I can't speak to the specific implementation for the Roll20 API Sandbox, but basically there is a watchdog process that watches for a flag to get set within some time limit.&nbsp; If it doesn't get set, it kills the sandbox as it is likely a run away process.&nbsp; The time limit is something like 60 seconds.&nbsp; The way you get around that is by making sure to return control to the sandbox regularly so it can tick that flag.&nbsp; Most often you'll hit that issue when you do something like making lots of calls to findObjs() and doing lots of math to figure things out.&nbsp; You can reorganize your operations so that you do them in small chunks asynchronously, and you'll not have that problem (example to follow below). For your calculations, there might be some thinks you could do to speed things up (use square of distance for "in range" checks, rather than doing the square root, etc), but it's hard to know without seeing what you're doing. I use a technique I call "burndown queues" to make sure I don't get the Possible Infinite Loop Detected problem.&nbsp; Here's a script that demonstrates.&nbsp; It builds a classic distance matrix between all graphics in a game. It's not really written in a useful fashion as it assumes all graphics are on the same page, and every graphic is something you'd want to know the distance to, but it scale exponentially, which makes it a good example: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 on ( 'ready' ,() =&gt; { const dist = ( t1 , t2 ) =&gt; Math . sqrt ( Math . pow ( t2 . get ( 'left' ) - t1 . get ( 'left' ), 2 ) + Math . pow ( t2 . get ( 'top' ) - t1 . get ( 'top' ))); on ( 'chat:message' , msg =&gt; { if ( 'api' === msg . type &amp;&amp; /^!do-simple(\b\s|$)/i . test ( msg . content ) &amp;&amp; playerIsGM ( msg . playerid )){ const start = Date . now (); log ( 'starting...' ); let tokens = findObjs ({ type : 'graphic' }); let count = 0 ; let total = tokens . length ; let distanceMap = {}; tokens . forEach (( t1 , idx , arr ) =&gt; { ++ count ; log ( ` $ { count } / $ { total }... ` ); for ( let i = idx + 1 ; i &lt; arr . length ; ++ i ){ let d = dist ( t1 , arr [ i ]); distanceMap [ t1 . id ] = { ... distanceMap [ t1 . id ], [ arr [ i ]. id ] : d }; distanceMap [ arr [ i ]. id ] = { ... distanceMap [ arr [ i ]. id ], [ t1 . id ] : d }; } }); const elapsed = Date . now () - start ; log ( ` Build token lookup for $ { Object . keys ( distanceMap ). length } tokens in $ { elapsed / 1000 } seconds ` ); } }); on ( 'chat:message' , msg =&gt; { if ( 'api' === msg . type &amp;&amp; /^!do-safer(\b\s|$)/i . test ( msg . content ) &amp;&amp; playerIsGM ( msg . playerid )){ const start = Date . now (); log ( 'starting...' ); let tokens = findObjs ({ type : 'graphic' }); let count = 0 ; let total = tokens . length ; let distanceMap = {}; const burndown = () =&gt; { let t = tokens . shift (); if ( t ) { ++ count ; log ( ` $ { count } / $ { total }... ` ); tokens . forEach (( t2 ) =&gt; { let d = dist ( t , t2 ); distanceMap [ t . id ] = { ... distanceMap [ t . id ], [ t2 . id ] : d }; distanceMap [ t2 . id ] = { ... distanceMap [ t2 . id ], [ t . id ] : d }; }); setTimeout ( burndown , 0 ); } else { const elapsed = Date . now () - start ; log ( ` Build token lookup for $ { Object . keys ( distanceMap ). length } tokens in $ { elapsed / 1000 } seconds ` ); } }; burndown (); } }); }); The first command (!do-simple on lines 5–28) does the calculation as a straight forward findObjs(), followed by effectively, a two nested for loops. It runs in 1.876 seconds for 273 tokens in my test campaign.&nbsp; The second command (!do-safer on lines 30–57) does the calculation as a deferred operation for each token.&nbsp; it runs in 2.376 seconds for 273 tokens in my test campaign. The second command is definitely slower, particularly for this arbitrarily expensive operation, but the difference is that while !do-simple will bomb with PILD in Dungeon of the Mad Mage (5969 tokens), !do-safer will keep running until it finishes. The idea behind this technique is that you collect a chunk of data to operate on (usually with findObjs()), then you write a function that will process one item of data, and then defer a call to itself.&nbsp; &nbsp;You do that with setTimeout() with a time of 0, which effectively returns control to the sandbox long enough for it to set the flag, then it processes the next item of data.&nbsp; Another difference between these two is that with !do-simple, all other API commands will hang until it finishes.&nbsp; With !do-safer, individual commands, api events, etc. will all get handled. For a larger example of this, look at my Search script, which builds a corpus of all attributes and text available in the API.&nbsp; It takes a while to load (Dungeon of the Mad Mage Corpus is 100,361 items, builds in 130.93s), but everything is responsive while it does it and PILD does not rear its ugly head. You mentioned using bounding boxes, which is awesome and something I would have suggested.&nbsp; It's hard to give specific suggestions without seeing what you're doing, but you might be interested in my libTypes script in the 1-click (or here:&nbsp; <a href="https://github.com/shdwjk/Roll20API/blob/master/libTypes/libTypes.js" rel="nofollow">https://github.com/shdwjk/Roll20API/blob/master/libTypes/libTypes.js</a> &nbsp;) which provides two classes: Rect and Quadtree.&nbsp; Rect is an optimized bounding box, and Quadtree gives you a fast way to look up things via spacial partitioning. I'm working on an enhancement to add in handling of line segments (and providing utility functions for intersections, splitting, etc), but those aren't ready yet.

Here's an example of using the Quadtree: Example code, run !map-page on a page with drawings on the map layer and it will build the Quadtree and use it for finding potential intersections: (Note the burndown example in LoadPage() =D ) /* global libTypes */ on('ready',()=&gt;{ const {Rect,Quadtree} = libTypes; // eslint-disable-line no-unused-vars let maps = {}; const getPageForPlayer = (playerid) =&gt; { let player = getObj('player',playerid); if(playerIsGM(playerid)){ return player.get('lastpage') || Campaign().get('playerpageid'); } let psp = Campaign().get('playerspecificpages'); if(psp[playerid]){ return psp[playerid]; } return Campaign().get('playerpageid'); }; const simpleObj = (o)=&gt;JSON.parse(JSON.stringify(o)); const AABB = (o2) =&gt; { let o = simpleObj(o2); return new Rect(o.left,o.top,o.width,o.height); }; const makeBox = (parts) =&gt; createObj('path',{ pageid: parts.pageid, path: JSON.stringify([ ["M",parts.stroke,parts.stroke], ["L",parts.width-parts.stroke,parts.stroke], ["L",parts.width-parts.stroke,parts.height-parts.stroke], ["L",parts.stroke,parts.height-parts.stroke], ["L",parts.stroke,parts.stroke] ]), fill: "transparent", stroke: parts.color, rotation: 0, layer: 'gmlayer', stroke_width: parts.stroke*2, width: parts.width||100, height: parts.height||5, top: parts.y, left: parts.x, scaleX: 1, scaleY: 1, controlledby: parts.tag }); const LoadPage = (page,who) =&gt; { maps[page.id] = new Quadtree( Rect.fromPage(page),{ maxObjects:4, maxDepth:10 }); let queue = findObjs({ layer: 'map', type: 'path', pageid: page.id }); const burndown = () =&gt; { let o = queue.shift(); if(o){ maps[page.id].insert(Rect.fromPath(o), o); setTimeout(burndown,0); } else { sendChat('',`/w "${who}" &lt;div&gt;Loaded page &lt;code&gt;${page.get('name')}&lt;/code&gt;&lt;/div&gt;`); findObjs({ type:'path', pageid: page.id, controlledby:'Quadtree' }).forEach(o=&gt;o.remove()); let rep = maps[page.id].toObject(); const drawNode = (n) =&gt; { makeBox({...n.bounds, color:'#00000066', stroke:2, pageid: page.id, tag: 'Quadtree'}); n.nodes.forEach(drawNode); }; drawNode(rep); } }; burndown(); }; on('chat:message',msg=&gt;{ if('api'===msg.type &amp;&amp; /^!map-page(\b\s|$)/i.test(msg.content) &amp;&amp; playerIsGM(msg.playerid)){ let who = (getObj('player',msg.playerid)||{get:()=&gt;'API'}).get('_displayname'); let pid = getPageForPlayer(msg.playerid); let page = getObj('page',pid); LoadPage(page,who); } }); on('change:graphic',(obj,prev)=&gt;{ let q = maps[obj.get('pageid')]; if(q){ findObjs({ type:'path', pageid: prev._pageid, controlledby:'TokenBB' }) .forEach(bb =&gt; { let former = q.retrieve(Rect.fromPath(bb)); former.forEach(o=&gt;o.context.set({stroke:'#00ff00'})); bb.remove(); }); let newBB = Rect.fromGraphic(obj); makeBox({...newBB.toObject(), color:'#66009966', stroke:1, pageid: obj.get('pageid'), tag: 'TokenBB'}); let objs = q.retrieve(newBB); objs.forEach(o=&gt;o.context.set({stroke:'#ff0000'})); } }); });

That’s a lot to digest, I see a learning weekend coming. &nbsp;And I thank you, seeing stuff like this starts giving me ideas. I can see implementing the asynchronous burn down queues in particular. &nbsp;Lots to read and test out.&nbsp; I’m using hexes and hex math (fun fun) with things I’ve learned from Red Blob games. &nbsp;As well as collision detection stuff from a website by Jeffrey Thompson. Things work, it’s now a matter of trying to optimize for speed. So for instance when I first wrote things I would check a token against every terrain polygon to see if it was ‘in’ it, now with bounding boxes it’s down to just the 1 or few in the area. Thats sped things up and I’m going to go through the code now to work on similar bottlenecks . My use of the state variable so far is to store the tokenIDs that are grouped together (representing the various formations) and the array of terrain lines as they don’t change position. I don’t know if the latter speeds up much but I previously would build the terrain array each time I had an API call (say a macro to fire ranged weapons, or a bombardment etc) to use within the function. &nbsp; Using the state not to hold things between sessions but between turns basically. &nbsp;So I build the terrain array at the beginning of the game and just retrieve it as needed. I think I’m averaging only about 30k of memory (used a script of yours I found to measure that lol) And perhaps there’s a better way of doing that? &nbsp;Can a const declared during say an on ready persist until a later API call during the same session for instance?

I love Red Blob Games. =D I would store that path data in a variable you create at on('ready',...), in its closure.&nbsp; Here's a sample script that shows how that works: //////////////////////////////////////////////////////////////////////////////// // functions create a closure, a private space for information that persists // after they are called. You can use the closure as a place to store things // for your script. on ( 'ready' ,() =&gt; { //////////////////////////////////////////////////////////////////////////////// // This will persist for the lifetime of the script running, and will provide // a place to look up cached information about the pages and their path // objects let PagePathMap = {}; //////////////////////////////////////////////////////////////////////////////// // This function finds all path objects and loads them into the PagePathMap, // with pageid as the primary key, and pathid as the secondary one. // // Note: burndown queue pattern const preLoad = () =&gt; { const start = Date . now (); let paths = findObjs ({ type : 'path' }); let c = paths . length ; let s = ( 1 === c ? '' : 's' ); const burndown = () =&gt; { let p = paths . shift (); if ( p ){ let pageid = p . get ( 'pageid' ); // Loading PagePathMap. Note that PagePathMap is not a global, it's // restricted to the closure created by the function passed to // on('ready',...). Effectively, it's a private variable for the // script. PagePathMap [ pageid ] = {... PagePathMap [ pageid ], [ p . id ] : p }; setTimeout ( burndown , 0 ); } else { const elapsed = Date . now () - start ; log ( ` $ { c } path$ { s } loaded in $ { elapsed / 1000 } seconds . ` ); } }; // start the burndown. burndown (); } //////////////////////////////////////////////////////////////////////////////// // Kick off the preload. This is happening on('ready',...), so all objects // are fully created in the API Sandbox. preLoad (); //////////////////////////////////////////////////////////////////////////////// // lifecycle management events. These will add newly created paths to the // map and remove deleted paths from it. on ( 'create:path' ,( p ) =&gt; { PagePathMap [ p . get ( 'pageid' )][ p . id ] = p ; }); on ( 'remove:path' ,( p ) =&gt; { delete PagePathMap [ p . get ( 'pageid' )][ p . id ]; }); //////////////////////////////////////////////////////////////////////////////// //This is a client of the PagePathMap variable. It uses it to get all the //paths on the current page, and build a little summary (number of paths, //number of line segments, etc) const getPathSummary = ( pageid ) =&gt; { let pathMap = PagePathMap [ pageid ]; let keys = Object . keys ( pathMap ); let c = keys . length ; let s = ( 1 === c ? '' : 's' ); let cs = keys . reduce (( m , k ) =&gt; m + (( JSON . parse ( pathMap [ k ]. get ( 'path' )) || []). length - 1 ), 0 ); let s2 = ( 1 === cs ? '' : 's' ); return ` Current page contains $ { c } path$ { s } with $ { cs } line segment$ { s2 }. ` ; } //////////////////////////////////////////////////////////////////////////////// // Utility function to find what page the player is currently on. This is // necessary to know where the gm is, or if a player is split from the // players ribbon, etc. const getPageForPlayer = ( playerid ) =&gt; { let player = getObj ( 'player' , playerid ); if ( playerIsGM ( playerid )){ return player . get ( 'lastpage' ) || Campaign (). get ( 'playerpageid' ); } let psp = Campaign (). get ( 'playerspecificpages' ); if ( psp [ playerid ]){ return psp [ playerid ]; } return Campaign (). get ( 'playerpageid' ); }; //////////////////////////////////////////////////////////////////////////////// // handle !path-info command on ( 'chat:message' , msg =&gt; { if ( 'api' === msg . type &amp;&amp; /^!path-info(\b\s|$)/i . test ( msg . content ) &amp;&amp; playerIsGM ( msg . playerid )){ let who = ( getObj ( 'player' , msg . playerid ) || { get : () =&gt; 'API' }). get ( '_displayname' ); let pageid = getPageForPlayer ( msg . playerid ); sendChat ( '' , ` / w "${who}" $ { getPathSummary ( pageid )} ` ); } }); // end of Ready function Closure }); Just run !path-info to see the output. With the terrain, if what you need to know is what kind of terrain the unit is in, I would either [1] precalculate that for every unit and change it when the token moves (see the above code for events that update the cached data based on changes to the game), or I would [2] precalculate the type of terrain for each grid square (if tokens can be assume to be in a specific hex/square), or [3] precalculate it for an overlayed rectangular grid with sufficient resolution to be correct for unit locations (I only did forest because I'm lazy. =D). One problem with storing that in state is that even though it doesn't change, if it ever did change, you'd have a mismatch (unless you're recreating it every time you load anyway, in which case putting it in a closure variable is all you need to do).

I think I may need to learn things like on ready and how the sandbox data persists and how scripts 'run'. Some lightbulbs in my brain may be slowly turning on... but their dim... So to explain that better, I have written a number of scripts, for specific actions. I have one for ranged weapon fire, one for barrage weapon fire, one for rallying broken units, one for assaults etc Maybe 10 in total. Given my lack of skills, I basically wrote each as an independent script, using this kind of format at the start of each (below is the start of a script for essentially rolling initiative at the start of each turn). This allowed me to write and test each in turn and 'understand' things as I wrote them (not a programmer). Common functions are just 'exposed' - not in any kind of protected space and so accessible by all. All of these scripts follow this basic format. Token macros/buttons call these when appropriate - so for example a player just clicks the Ranged Fire button on a token, which calls the !Ranged script, using the tokens @{selected|token_id} and a @{target|token_id}&nbsp; //Strategy Roll var StrategyScript = StrategyScript || {} on("chat:message", function(msg) { if (msg.type !== "api") { return; } var command = msg.content.split(";",1); if (command == "!Strategy") { StrategyScript.Process(msg); } }); StrategyScript.Process = function(msg){(rest of code)} So basically each is self-contained and so I've had to generate some things (like the terrain array) each time a script is called. So my question: If I were to combine ALL of these into one giant script (yikes, thats gonna be a long one on my screen) then could (?) I have the script, using an on(ready) to create the static data (like terrain polygons) and maybe define functions (I still have to figure out how to write functions as constants instead of just straight functions) and have that data persist until the on(chat) message calls are made? Then I could see making an array of tokens on the map, their locations, units, info etc and updating it using on(change) events and then accessing it when units fire etc. I guess what I am asking is if this massive script would still be 'running'&nbsp; in between times when players (in the same session) use various macros? (or how do I structure it so that it remains running?)&nbsp;&nbsp; The only script I haven't written that I use in this game is ScriptCards - which I am using for lovely output (I know I could write code to do the output, but my CSS/HTML is almost non-existent and power/script cards is so easy to use using a sendChat and building up the output in my own scripts).

I think I answered your question in the post I made at the same time as yours. =D

I think you sent your last response Aaron just as I wrote my last - and I think you've given me the answers! I see a massive re-write in my future...

Aaron - you are the best. Period end. Thanks so much!

Let me clarify how API scripts run.&nbsp; First off, I think it's good to use the mental model of the API Sandbox being "a player without a screen that just reacts to what everyone else is doing."&nbsp; The API is event driven, and the way you latch on to those events is by registering a function with the event in question using the on() function.&nbsp; The first argument is the event, the second argument is the function.&nbsp; In reality, the body of your script is only executed one time, but the event handlers continue on and are called whenever those events occur.&nbsp; Anything your script sets up will remain, which is why each of your scripts have access to that StrategyScript global variable you're creating/using.&nbsp; Here's a list of what happens when you start the API Sandbox: All scripts are concatenated into one "file" and executed. The sandbox loads all the objects in the game, issuing create events for each one. The ready event is fired. Events are fired as they occur (chat events, change events, add events, remove events, etc). Generally speaking, it's best to wrap your scripts in on('ready',...) so that you don't have to deal with all the create events coming in.&nbsp; In practice, most scripts won't have to deal with that because they only react to the chat:message &nbsp;event.&nbsp; Wrapping in on('ready',...) also serves the purpose of creating a closure (private scope) around your script which protects it and other scripts from namespace collisions. So, to further codify that into an answer.&nbsp; If you create your script with all your commands together, it will be a larger file, but you can start to get some better reuse of code and data structures.&nbsp; Using on('ready',...) and the sample code above, you should be able to make all your commands run much, much faster through preloading and caching.

If you want to dig into Javascript and the API more, here's some links to conversations that go over lots of concepts in plain english and great detail: <a href="https://app.roll20.net/forum/post/6605115/namespaces-novice-seeks-help-exploring-the-revealing-module-pattern" rel="nofollow">https://app.roll20.net/forum/post/6605115/namespaces-novice-seeks-help-exploring-the-revealing-module-pattern</a> <a href="https://app.roll20.net/forum/post/6584105/creating-an-object-that-holds-specific-character-dot-id-and-character-name/?pagenum=1" rel="nofollow">https://app.roll20.net/forum/post/6584105/creating-an-object-that-holds-specific-character-dot-id-and-character-name/?pagenum=1</a> <a href="https://app.roll20.net/forum/post/6237754/slug%7D" rel="nofollow">https://app.roll20.net/forum/post/6237754/slug%7D</a>

Ok, so thanks for the input and links, I'm certainly growing in my understanding. I'm re-writing my various scripts into "One Script to Rule them All", and am structuring it using the Revealing Module Pattern described in the link above. I've also read the link that gets into the const/var/let comparison and am eliminating all the (many many many) var declarations... A question I have: I've previously been writing my functions based on what I learned on the web tutorials as function NAME() {body of function} but I see the fat arrow, defining the functions as const: const NAME = () =&gt; {body of function} Are there pros/cons? My 'newer' ones - such as the terrain function, I've written using the fat arrow syntax, and am incorporating the 'burndown queue' where needed to avoid the Infinite Loop error. My debate would be whether to just copy my previously written functions (using the 1st type of syntax) or rewrite them into the fat arrow format. Thoughts?

There are a few differences with the fat arrow syntax, but typically the only one that makes a difference on Roll20 (the only one I've run into having to think about) is that it doesn't bind its own this . Especially with the revealing module pattern and any number of events passing through a script, I've not had a lot of reason to use this except for within a Class. If you have working code, I wouldn't rush to rewrite it unless it made sense to you. For myself, I find that my eyes more easily digest the fat arrow syntax than function naming pattern, but there isn't a speed difference to the code execution, if that is what you are wondering.

What Tim said. =D&nbsp; You will probably never run into the this issue, as modern Javascript has class definitions as first class citizens and their syntax for functions doesn't use fat arrows or the function keyword.&nbsp; The binding of this is really only going to show up with prototypical inheritance (which you'll probably never use) and certain UI libraries you can't use with the Roll20 API anyway. If you want to see some examples of modern Javascript classes, take a look at implementation of libTypes.

I looked at libTypes... I have not seen Classes or Static or Constructor before... and now my brain 'urts... I feel like I have been given homework...

Always something more to learn. =D