JSON.parse error in sheetworker?

I've got an error that doesn't always replicate and is hard for me to figure out. 

The situation: I have a hidden variable called runeBag with an initial value of "[]". This variable keeps track of the "runes" that have been tied to active abilities and is intended to store those as an array. The runeBag variable can be grabbed by a companion script and then the script does stuff with it. This is the sheetworker code

// This handels keeping a running list of which runes have been used
    const runes = ["void", "fehu", "uruz", "thurisaz","cansuz", "raidho", "kenaz", "gebgift", "wunjo",
    "hagalaz", "naudhneed", "isaice", "jethe", "eihwas", "pertho", "elhaz", "sowsun", "tiwaz", "berkano", 
    "ehwo", "mann", "claguz", "dagaz", "ing", "othala", "physical", "mental", "spiritual"]
    on("change:repeating_actives", (eventInfo) => {
        //setAttrs({"runeBag": [1,1,3]});
        //console.log(eventInfo);
        //console.log(eventInfo.newValue);
        if (_.contains(runes, eventInfo.newValue) || _.contains(runes, eventInfo.previousValue)) {
            getAttrs(["runeBag"], v => {
                //console.log(v.runeBag);
                if (!(_.isArray(v.runeBag))) {
                    //console.log("not array");
                    var bag = JSON.parse(v.runeBag);
                } else {
                    var bag = v.runeBag;
                }
                //console.log(bag);
                //console.log(_.isArray(bag));
                if (eventInfo.newValue == 0) {
                    //console.log("new value 0")
                    var bag = _.difference(bag, [eventInfo.previousValue]);
                    //console.log(bag);
                }
                else if (!(_.contains(bag, eventInfo.newValue)) && eventInfo.newValue != "void") {
                    bag.push(eventInfo.newValue);
                    //console.log(bag);
                }
                //console.log(bag);
                setAttrs({"runeBag": bag});
            });
        }
    })

It seems that for some characters the runeBag variable is being stored correctly as an array while for others it is being stored as a string. In the ones where it is being stored as a string the console outputs:

SyntaxError: Unexpected token e in JSON at position 0
    at JSON.parse (<anonymous>)
    at Object.eval [as -MRLUTf0WkjK39M2CoSI//repeating_actives_-mrmiz2ltsxkbvvnyvpu//0.9079199319327471] (eval at messageHandler (sheetsandboxworker.js?1611688316695:698), <anonymous>:114:36)
    at _fullfillAttrReq (sheetsandboxworker.js?1611688316695:673)
    at messageHandler (sheetsandboxworker.js?1611688316695:705)
sheetsandboxworker.js?1611688316695:734 
SyntaxError: Unexpected token e in JSON at position 0
    at JSON.parse (<anonymous>)
    at Object.eval [as -MRLUTf0WkjK39M2CoSI//repeating_actives_-mrmiz2ltsxkbvvnyvpu//0.9079199319327471] (eval at messageHandler (sheetsandboxworker.js?1611688316695:698), <anonymous>:114:36)
    at _fullfillAttrReq (sheetsandboxworker.js?1611688316695:673)
    at messageHandler (sheetsandboxworker.js?1611688316695:705)

when I change a rune value and trigger the sheetworker. In the ones where it stores properly the console outputs:

Foudn a pre-defined key order!

Yes, it says Foudn, not Found, no idea why.

Anyone have any idea what is going on?

Edit: sometimes the unexpected token is a "w" or an "o" or some other letter, but it seems to be consistent for each character.

JSON.parse is for reading JSON formatted data/ If the data isnt in JSON format, it's likely to generate errors. Since you're reading an attribute from the character sheet, it seems like its never going to be in JSON format (especially if its a player-edited attribute), so what is HJSON.parse doing there?

Your error tests if it's an array, and assumes if its not an array, it must be JSON. And so attempts to convert it into a javascript object (thats what JSON.parse does). That seems like faulty logic to me.

This line  bag.push(eventInfo.newValue); seems doomed to failure, on any data that has been JSON.parsed, because push if for arrays not objects.

There might be other problem with the script, but it seems like your issue here is form mishandling data.

You want to get the data from teh character sheet, then convert into the correrct type of variable for your needs, and make sure that whatever it started out as (string, null, array, object, JSON, etc) it ends up as exactly the same variable type. That doesnt seem to be being done here.

JSON.parse does not create arrays like [0, 1, 2], it creates objects like {something: 0, "something-else": 1, another_thing: 2}, and it is designed to work with JSON objects not arrays.

PS: that Foudn is a typo in roll20's code - weird, but you can ignore it.

I think you probably just want to use .split() and .join() if you're only dealing with an array. As GiGs said, JSON.parse requires a very specific input - you can certainly write a function to get all your R20 attributes into the correct format, but it's a fair amount of work if you don't need nested objects.

Split and join are pretty easy: (the /\s*,\s*/ is just to get rid of pesky spaces that could creep in... you can just use ',')

const runebagGet = '1,1,3';

let runeArray = runebagGet.split(/\s*,\s*/);  // runeArray: ['1', '1', '3']
runeArray.push('5');
let runeOutput = runeArray.join(',');  // runeOutput: 1,1,3,5

JSON.parse needs something like this:

[{"object1":"0"},{"object2":{"value1":"6","value2":""}}]

if you're missing any of those "" double quotes, or have any in the wrong place, it will crash. It's usually easier to build the object directly and push it to an array than worry about a missing quotation mark - unless the string was made directly from an Object with JSON.stringify, that is.

Hey GiG,

I was using JSON.parse because when I was writing the sheet I asked on the forums (https://app.roll20.net/forum/post/9615301/attributes-that-contain-arrays/?pageforid=9615301#post-9615301) how to store an array in an attribute and someone suggested that JSON.parse would work. I tried it out and it seemed to work (still seems to about half the time), but I'm getting these weird errors. Do you agree with Oosh that .split() is the way to go?

GiGs said:

JSON.parse is for reading JSON formatted data/ If the data isnt in JSON format, it's likely to generate errors. Since you're reading an attribute from the character sheet, it seems like its never going to be in JSON format (especially if its a player-edited attribute), so what is HJSON.parse doing there?

Your error tests if it's an array, and assumes if its not an array, it must be JSON. And so attempts to convert it into a javascript object (thats what JSON.parse does). That seems like faulty logic to me.

This line  bag.push(eventInfo.newValue); seems doomed to failure, on any data that has been JSON.parsed, because push if for arrays not objects.

There might be other problem with the script, but it seems like your issue here is form mishandling data.

You want to get the data from teh character sheet, then convert into the correrct type of variable for your needs, and make sure that whatever it started out as (string, null, array, object, JSON, etc) it ends up as exactly the same variable type. That doesnt seem to be being done here.

JSON.parse does not create arrays like [0, 1, 2], it creates objects like {something: 0, "something-else": 1, another_thing: 2}, and it is designed to work with JSON objects not arrays.

PS: that Foudn is a typo in roll20's code - weird, but you can ignore it.

Joshua S. said:

Hey GiG,

I was using JSON.parse because when I was writing the sheet I asked on the forums (https://app.roll20.net/forum/post/9615301/attributes-that-contain-arrays/?pageforid=9615301#post-9615301) how to store an array in an attribute and someone suggested that JSON.parse would work. I tried it out and it seemed to work (still seems to about half the time), but I'm getting these weird errors. Do you agree with Oosh that .split() is the way to go?

I agree completely with Oosh's post in that thread.

I definitely wouldnt have suggested using JSON.parse to a code beginner to encode your data. A simpler solution that doesnt need much manipulation is going to be much easier for you to learn how to work with.

So yes, encoding your data using a separator* of some sort to create a string that can be easily converted to an array seems like a good approach.

* I'm fond of using a pipe (|) as separator rather than a comma in cases like this, because its less likely to be entered by a player. And when using attributes for data storage you have to be careful to handle how players can accidentally enter bad data (getting extra commas in the wrong place is really easy with player-edited data).

Using split to convert your attribute into a true array, and join if you need to convert it back into a string (perhaps for saving back to an attribute) would be a very simple way to handle this.

Note that regardless of what you try to do with attributes, they are always stored as strings in roll20 and cannot really be an array until you process them somehow. Split is perfect for that, but depending on how those attributes are created in the first place, you might need some validation function to make sure the attribute's contents are in the right format.

If I'd seen your original post here's how I'd have answered it (with the benefit of hindsight, admittedly):

Joshua S. said:

How do I create an attribute that contains something that will be read as an array by the sheetworker? I'm looking for something like:

<input type="hidden" name="attr_array" value="[one, two, three]" />
...
...
...

getAttrs(["array"], v => {_.each(v.array, function(){...})}

Basically, I just want to store an array in a hidden attribute and then later get it and do array things to it (like pop and push) and then re-store it. I'm sure this must be like 101 level stuff, but I'm barely a 100 level coder

Since you're planning to use a hidden attribute, this is doable, and protects against accidental manipulation by players. You need two things: 1st a way to store the data, and 2nd, also a way to retrieve it.

First to store the data. lets say you have an array called runes, and want to store it. You'd use code something like:

const data = runes.join('|');
setAttrs({
    runeBag: data
});

This converts an array like [1,2,3] into a string that looks like this "1|2|3"

Then you need to grab that data and convert it back into an array, which you could do like this:

getAttrs(['runeBag'], values => {
    const runeBag = values.split('|');
    // do stuff with the runes
});

You might also want to ensure that the data you are saving contains valid values, and when you retrieve it ensure it contains the correct kind of values in the array.

PS: it looks like the getAttrs in your original post isnt using the correct syntax. The code above is in the proper form.

Huh, that's weird. My companion script grabs the `runeBag` variable and treats it as an array without any processing. Sometimes it works just fine, specifically in those instances where the sheetworker didn't give an error. As far as I can tell, arrays (or something that will be interpreted as an array) can be stored in an attribute, but it has to be an array type variable that is stored using setAttrs, not something that is entered by the user or set by html. Interestingly, the data looks the same under the Attributes & Abilities tab in the character sheet whether or not it is actually stored as an array. I did find that in my companion script I have to import the attribute as a literal, so in

let runes = getAttrByName(theCharacter.id, `runeBag`);

runes works as an array if and only if runeBag was an array variable set by setAttrs, but in

let runes = getAttrByName(theCharacter.id, 'runeBag');

runes doesn't behave as an array no matter how runeBag was stored. That one took me forever to figure out and I still don't understand why it works that way. 

Using split to convert your attribute into a true array, and join if you need to convert it back into a string (perhaps for saving back to an attribute) would be a very simple way to handle this.

Note that regardless of what you try to do with attributes, they are always stored as strings and cannot really be an array until you process them somehow. Split is perfect for that, but depending on how those attributes are created in the first place, you might need some validation function to make sure the attribute's contents are in the right format.

Joshua S. said:

Huh, that's weird. My companion script grabs the `runeBag` variable and treats it as an array without any processing. Sometimes it works just fine, specifically in those instances where the sheetworker didn't give an error. 

This may be where your error is coming from then.

Anything stored in a roll20 attribute is always a string. But Javascript will try to coerce data into the correct format, and can be pretty smart about it - but it's not perfect. It will sometimes fail, so its best to build your functions to ensure that the data is correct, in both directions - when you saving the data, and when you are retrieving it.

As far as I can tell, arrays (or something that will be interpreted as an array) can be stored in an attribute, but it has to be an array type variable that is stored using setAttrs, not something that is entered by the user or set by html.

There might be a difference between the way setAttrs saves data and manual modification of attributes that makes it easier for JS to recognise arrays, but it is still stored as a string on roll20's servers. We know this because roll20 have told us this.

Also because something I said earlier could be easily overlooked:

You might also want to ensure that the data you are saving contains valid values, and when you retrieve it ensure it contains the correct kind of values in the array.

I shouldnt have said might there - for situations like this, where you are saving data in a specific format somewhere not designed for that format, like a roll20 attribute that has no inbuild validation and will accept any value, this is absolutely vital.

Your code should have some function that makes sure you cannot save invalid data, and also when loading that data, that is in the correct format and individual values in the array are valid. You want to eliminate data as a source of errors, otherwise you will inevitably get errors from it.

Unless users go manually modifying the attribute in the attributes and abilities tab then it should only ever have the specific data that I have set in it, but I take your point. I usually deal with this when getting information out of a variable by telling the function to ignore any data that isn't in the pre-determined list, but I should probably do it on the front end too. Most of my limited programming experience is in python where everything is either a list or a dictionary

GiGs said:

Also because something I said earlier could be easily overlooked:

You might also want to ensure that the data you are saving contains valid values, and when you retrieve it ensure it contains the correct kind of values in the array.

I shouldnt have said might there - for situations like this, where you are saving data in a specific format somewhere not designed for that format, like a roll20 attribute that has no inbuild validation and will accept any value, this is absolutely vital.

Your code should have some function that makes sure you cannot save invalid data, and also when loading that data, that is in the correct format and individual values in the array are valid. You want to eliminate data as a source of errors, otherwise you will inevitably get errors from it.

Joshua S. said:

Unless users go manually modifying the attribute in the attributes and abilities tab 

You should be safe there. Hidden attributes dont show up on that tab.

That doesnt protect you from errors in code from creating data in the wrong format, and since this is the most critical part of your code, it's a good idea to make sure you have some sort of validation (especially for something like a roll20 sheet worker, where users have no way to interact with them and cant manually correct them).