API findObjs performance issue in large campaigns

Hi, I've been trying to help someone  debug a performance problem ("Possible infinite loop detected") with the 5eShapedCompanion script. The GM in question has a rather large campaign - he's created characters for most of the entries in the Monster Manual + loads of extra NPCs + PCs from various campaigns + spell templates + a bunch of stuff from various WotC adventure paths. In total he has 750 character objects in the campaign. Now I know this is not entirely advisable - and I've recommended that he takes an alternative approach - but in the grand scheme of computer programming, it's not such a huge amount of data to work with, and it really ought not to result in things timing out > 30seconds. So I decided to do a bit of debugging, on the assumption that I'd probably done something a bit daft in the script somewhere. But after a bit of digging, I couldn't find any smoking guns in my script, so I did a little bit of very simplistic profiling of a basic test case: on('chat:message', (msg) => {     if(msg.type === 'api' && msg.content === '!test') {         const start = Date.now();         const profile = [];         createObj('character', {name:'TestingChar'});         profile.push({name:'charCreated', time:Date.now()});         const char = findObjs({type:'character', name:'TestingChar'})[0];         profile.push({name:'charRetrieved', time:Date.now()});         createObj('attribute', {characterid:char.id, name:'testAttr', current:'myValue'});         profile.push({name:'attrcreate', time:Date.now()});         const attr = findObjs({type: 'attribute', name: 'testAttr', characterid: char.id})[0];         profile.push({name:'attrFound', time:Date.now()});         attr.remove();         profile.push({name:'attrRemoved', time:Date.now()});         char.remove();         profile.push({name:'charRemoved', time:Date.now()});         sendChat('', '/w gm ' + JSON.stringify(profile.map( entry => { return {name:entry.name, time:(entry.time-start)/1000}})));     } }); The results were: [{"name":"charCreated","time":0.004}, {"name":"charRetrieved","time":0.549}, {"name":"attrcreate","time":0.551}, {"name":"attrFound","time":1.184}, {"name":"attrRemoved","time":1.185}, {"name":"charRemoved","time":1.186}] As you can see, findObjs is routinely taking > 0.5 seconds, even when it's looking up by what would be a fully indexed composite key in a normal relational database (attribute name + character id). findObjs being slow for big campaigns would be expected and livable-with if it was an edge case for doing arbitrary lookups across random fields - but at the moment it's the only way we can retrieve and work with character attributes, which is pretty fundamental. I can easily end up doing 50+ attribute lookups for a single script command, which is already pushing me dangerously close to 30 seconds without any other work going on. Is there any way that either (a) some more intelligent indexing can be applied to improve this (I know nothing about your backend datastore so I have no idea if this is feasible) or (b) could we have a special function for looking up character attributes that is optimised? Cheers, Lucian

For reference, on one of my campaigns, which has 33 character objects, those findObjs calls take more like 20ms. I'd really hope that they can be O(log(n)) at worst with sensible indexing, but it looks like it's a lot closer to O(n) at the moment, which is pretty bad for such an important query.

I'd have to put some logs in to check, but I'm pretty sure this is why we went with a deferred recursive call for part of a script Aaron and I were looking at. With the deferred calls in, it can take up to a minute and a half for the api to load the data it needs for 200 or so tables.

Hi Kristin, I think we should be careful not to confuse two separate issues here. There are problems that arise from the total byte size of the campaign, particularly if it's being copied; and this can obviously contribute to poor performance on the client as well. I've recommended to the GM in question that he take an alternative approach for this reason. But this is distinct from the API performance of querying the data, which really shouldn't have a linear relationship to the size of the dataset in a properly designed data store. In this case, we're not talking about data transfer times or memory footprints, we're talking about how long it takes to perform a query with a particular set of criteria against whatever data storage system sits in the back end. And since this is from the API, this is all happening at your end - there's no round trip time involved and no dependency on the performance of the client PC(s). I'll grant you that the Shaped sheet has quite a lot of attributes, but as I mentioned above, in database terms, a few hundred thousand records is relatively small and really shouldn't cause a performance slow down for indexed queries. I've worked with websites that are routinely doing queries with joins across multiple tables containing millions of records for each page view. With sensible indexing and caching, this can be done in a matter of a few milliseconds. A routine query like finding an attribute by name for a given character should never be taking half a second, irrespective of the size of the campaign, and to me this suggests that there is probably some tuning/optimisation that needs to be done on your data store. Cheers, Lucian

The backend is Firebase. &nbsp;This might be pertinent: <a href="https://firebase.google.com/docs/database/security" rel="nofollow">https://firebase.google.com/docs/database/security</a>... Improper indexing on the backend probably would result in similar slowdowns with duplicating the dataset regardless of if it's traversing for findObjs()/filterObjs() or traversing for the intent of copying the data. &nbsp;I commonly run into cases in relational databases where a lack of index can go below the threshold of notice for a long time, until there is enough data to make it apparent. Hopefully adding some indices will have as much benefit in Firebase as it does in relation databases.

Yeah, I was hoping that this might be as simple as putting an index on charId + attribute name and Hey Presto! everything's fast again. But I guess it might not be that easy. I drew the distinction with copying because I know there's a bunch of stuff that happens in/via the client that you might not expect to, in which case the byte size of the data becomes much more relevant. I guess even if there's nothing going over the network, the db writes&nbsp; on copying a few hundred thousand records may well take a while and having the front end waiting on a response could be problematic. That's a case where an asynchronous back-end process with polling for completion is probably warranted. But for the attribute query case there's next to no data being transmitted or written, so hopefully it should just be a case of adding the appropriate index...

Hi, For the avoidance of doubt, I still consider this to be an important problem (and hopefully one with a relatively simple solution) and would like to hear back as to whether something can be done about it - even if that's just a dev taking&nbsp; 5 minutes to look at the Firebase definitions and go "yeah, in theory we should be able to whack on index on that" and then parking it until someone has a bit more time... Cheers,