Hmm, It does indeed look like something has changed in the backend. I setup a new speed test and found that both setAttrs and getAttrs are scaling at a rate of ~1ms/attribute, but getAttrs is wildly variable taking a minimum of 80ms to get 1000 variables, but as long as 2500ms 11000ms (went back and reviewed my speed data and the highest get time was shocking!) to get those same variables. I'm hoping that they get this fixed, or I'm going to need to rewrite the JS for several sheets.
Some notes on your thoughts on how to optimize:
In my "on open" event I call all the data. Everything. So opening a sheet drags a bit, but this is a one-time operation. Then I dump the single object with all the data into a new object, add in the arrays I need from getSectionIDs, and finally JSON.strigify the new object and write it to the sheet via setAttrs(). This approach seemed better and safer than trying to maintain a shadow database, as if there is any data corruption or synching issues then just closing and opening the sheet again refreshes all the legit data.
This is something that I may experiment with depending on how long this backend change goes on. I think you've hit most of the pain points with this sort of method. The only thing I would suggest testing is what happens when you have multiple users making changes at the same time on a sheet and what happens when you make changes to several sheets at the same time. Both of these scenarios are extremely unlikely with just users, but when you throw in the API's ability to manipulate sheets it becomes a near certainty that an API might try to do several successive manipulations to the same character or to several characters. I'd also look into ways to handle APIs that change attributes values, but don't trigger sheetworkers (there's several legacy scripts out there that operate this way) as that will cause a desync between your cache and the actual values.
When my single event listener fires (well, all events except open ;) ) I use getAttrs() to fetch my all-data-JSON-string and parse it into an object using the same name as the "values" object I used prior. Because Roll20 uses a flat data structure I decided to do the same rather than reformatting my data into a hierarchical tree. This allowed me to use the same code in my sheet, like v[`repeating_section_${rowID}_attribute`] without any additional changes. This transition has been A LOT easier and quicker than I originally expected.
This is good to hear. It's the area that I was most worried about with changing to this method.
Additionally, I breakout the sectionIDs into their own arrays that I built in the on open event from getSectionIDs. Any new repeating sections created are pushed into their respective array.
Before performing a setAttrs(), I merge the values object with another object used for capturing all sheet changes (overwriting the values object) along with the sectionIDs arrays, and JSON.stringify it before feeding it to setAttrs(). And...persistency!
Other than the cache object, this is similar to what the K-scaffold does. I'll probably look into a way to incorporate this cache method into the K-scaffold.
I'm not sure if going async/await would provide additional performance benefits, but that may be the next thing I POC once I'm done with a lot of long and sorely needed refactoring. I mention this because there are still a few things I've noticed that I can't yet account for. Such as, the getAttrs() fetch time seems to vary widely depending on the complexity of the script pieces that run following it. I suspect this could be due to the way callbacks work and it would be an interesting exercise to sort out.
Unfortunately, async/await is not possible in character sheets (other than when using startRoll). There is a library of code called Roll20Async that we thought had solved the problem for several months, however it breaks the API sandbox so that any game using a sheet running on Roll20Async can't use API scripts that interact with attributes.
Also, I think CSS might be another point of optimization as my CSS file is about as organized as junkheap. My hide/show/replace events all seem to drag, and I suspect this could be due to the parsing of my junky CSS file.
CSS doesn't care a great deal about organization fortunately, however if you have multiple rules that could apply to an item and are overwriting things several times, I suppose that could cause some performance issues. A bigger performance issue that I find is writing CSS declarations that are too broad. So something like:
div .div-5 .span-6{/*some style*/}can really screw with the CSS efficiency because the CSS parser has to essentially check every div to see if it or any of its children contain .div-5 and then has to check if .div-5 or any of its children contain .span-6. If you've got a reasonably complex element structure, this can really slow down the rendering of the sheet, especially if the repeating sections are involved in the element tree that the CSS needs to inspect.
Another factor I suspected but couldn't determine a way to test for was the availability of Roll20's "getAttrsAPI". I felt like an early astronomer before telescopes. A successive getAttrs() fetch time on the same operation could swing wildly by a few hundred ms. This made me think that good response times were due to low volume of API calls, and poor response times due to high volume of API calls. Like waiting in a queue to be processed.
I suspect that this is the cause of the wild swings in getAttrs times, but it's odd that the setAttrs times are so much more consistent.
For anyone interested, here's my sheet speed test code:
<label style="display:inline;">
<span>Number of attributes</span>
<input type='number' name='attr_num' value='10' style="width:10rem">
</label>
<label style="display:inline;">
<span>Iterations</span>
<input type='number' name='attr_iterations' value='100' style="width:10rem">
</label>
<button type='action' name='act_trigger'>Test</button>
<label style="display:inline;">
<span>Iterations Remaining</span>
<input type='number' name='attr_iterations_remaining' value='0' style="width:10rem">
</label>
<label style="display:inline;">
<span>Duration of Last Set Iteration (ms)</span>
<input type='number' name='attr_last_set_duration' value='0' style="width:10rem">
</label>
<label style="display:inline;">
<span>Duration of Last Get Iteration (ms)</span>
<input type='number' name='attr_last_get_duration' value='0' style="width:10rem">
</label>
<script type="text/worker">
const times = {
/*
general format is:
gets:[{attrNum:20,duration:30},{attrNum:20,duration:40}],getAvgs:{'20':35},lastget:40
duration is in ms
*/
gets:[],getAvgs:{},
sets:[],setAvgs:{}
};
let lastGetDuration = 0;
const calcStats = function(type,start,end,attrNum){
let duration = end - start;
times[`${type}s`].push({attrNum,duration});
let durationArr = times[`${type}s`]
.reduce((memo,timeObj)=>{
if(timeObj.attrNum === attrNum){
memo.push(timeObj.duration);
}
return memo;
},[]);
times[`${type}Avgs`][`${attrNum}`] = durationArr
.reduce((total,duration)=>{
return total + duration;
}) / durationArr.length;
return duration;
};
const iterateTest = function(attrNum,iterations,origIterations,setDuration=0,getDuration=0){
origIterations = origIterations || iterations;
iterations--;
const getArr = [];
const initiateObj = _.range(attrNum).reduce((m,n)=>{
m[`attribute_${n}`] = Math.random();
getArr.push(`attribute_${n}`);
return m;
},{iterations_remaining:iterations,last_get_duration:getDuration,last_set_duration:setDuration});
let setStart = Date.now();
setAttrs(initiateObj,{silent:true},()=>{
let setEnd = Date.now();
let setDuration = calcStats('set',setStart,setEnd,attrNum);
let getStart = Date.now();
getAttrs(getArr,(attributes)=>{
let getEnd = Date.now();
let getDuration = calcStats('get',getStart,getEnd,attrNum);
if(iterations){
iterateTest(attrNum,iterations,origIterations,setDuration,getDuration);
}else{
outputResults(attrNum,setDuration,getDuration,origIterations);
}
});
});
};
const outputResults = function(attrNum,setDuration,getDuration,iterations){
console.log('Stat Summary');
console.table({
runs:iterations,
get:{'Number of Attributes':attrNum,Duration:getDuration,'Average Duration (ms)':times.getAvgs[`${attrNum}`]},
set:{'Number of Attributes':attrNum,Duration:setDuration,'Average Duration (ms)':times.setAvgs[`${attrNum}`]},
});
console.log('Raw Stats');
console.table({runs:iterations,...times});
};
const runSpeedTest = function(){
console.log('Running sheet speed test');
getAttrs(['num','iterations'],(options)=>{
//Initiate the attributes to test
let attrNum = +options.num || 1;
let iterations = +options.iterations;
iterateTest(attrNum,iterations);
});
};
on('clicked:trigger',runSpeedTest);
</script>
