_.each returns multiple times the same entry

Hello all, In our API, we have a function that at a certain point lists each repitem in a given fieldset, and sends a button in the chat for each. The list creation goes like below. (Note that SPM is a code we have in repitems name for easier identification of the fieldset it is pertaining to, and for each repitem, we have a _RowID attribute that copies the unvandalized rowID)               _.each( attributes, function ( indexAttributes ) {                 if( indexAttributes.get( "name" ).endsWith( "_SPM_RowID" )) {                   let row = indexAttributes.get( "current" );                   if ( Earthdawn.getAttrBN( po.charID, Earthdawn.buildPre( "SPM", row ) + "Rank", 0, true) >= circle)   // Only list the matrices of a rank ot hold this spell.                     matrices.push( row );                     log("Matrix search attribute name " + indexAttributes.get( "name" ) + " & value " + row);                 }               }); // End for each attribute.               log("found matrixes quantity " + matrices.length); For reasons I don't understand, this code sometimes generate duplicates, (i.e. the same entry is listed twice). Of course a simple safeguard in the if will get rid of this duplicate   && !matrices.includes(row) But I am now worried about the root cause of these duplicates as it could also happen in other areas of the code. See below a log generated by this function on a test character that has 2 elligible repitems "Matrix search attribute name repeating_matrix_-MsXsyjPtRdTxSmBZGqA_SPM_RowID & value -MsXsyjPtRdTxSmBZGqA" "Matrix search attribute name repeating_matrix_-MsXsyjPtRdTxSmBZGqA_SPM_RowID & value -MsXsyjPtRdTxSmBZGqA" "Matrix search attribute name repeating_matrix_-MsY0FkS5DRxkZvEnwoJ_SPM_RowID & value -MsY0FkS5DRxkZvEnwoJ" "Matrix search attribute name repeating_matrix_-MsXsyjPtRdTxSmBZGqA_SPM_RowID & value -MsXsyjPtRdTxSmBZGqA" "Matrix search attribute name repeating_matrix_-MsY0FkS5DRxkZvEnwoJ_SPM_RowID & value -MsY0FkS5DRxkZvEnwoJ" "found matrixes quantity 5 » It can clearly be seen that the repitem with rowID ending in Gga comes three times, while the woJ comes 2 times, when it would be expected that they come only once... I was playing with 2 hypothesis: 1- Can it be a bug from the _.each function that could come several time at a same entry 2- Could it be that in the "database of all attributes" there are duplicates, where the attribute  repeating_matrix_-MsXsyjPtRdTxSmBZGqA_SPM_RowID  is duplicated, with the same name and same value... Maybe at some point when trying to write on this attribute instead of overwriting, it just duplicated it... so I added a log of JSON.stringify(indexAttribute) "indexAttribute {\"name\":\"repeating_matrix_-MsXsyjPtRdTxSmBZGqA_SPM_RowID\",\"current\":\"-MsXsyjPtRdTxSmBZGqA\",\"max\":\"\",\"_id\":\"-MsXsyjnfCDN7Jgew3n_\",\"_type\":\"attribute\",\"_characterid\":\"-MKPtPDNw5d-Rby1qw3U\"}" "indexAttribute {\"name\":\"repeating_matrix_-MsXsyjPtRdTxSmBZGqA_SPM_RowID\",\"current\":\"-MsXsyjPtRdTxSmBZGqA\",\"max\":\"\",\"_id\":\"-MsXsytw5KsJY3Bryrq_\",\"_type\":\"attribute\",\"_characterid\":\"-MKPtPDNw5d-Rby1qw3U\"}" So there is clearly 2 attributes, with different _id, but the same name !?! I have to dig in my code to understand how this could happen, but this is worrying for me as it means: 1- I don't know what happens when you try to read/write such a duplicate... Can they actually hold 2 different values ?? 2- It is potentially now in my released code, so even if I find and fix the part that creates the duplicate, there are campaigns out there that have these duplicates... Is it a known issue that you can have several attributes with same name but different _id ? Any recommandation?

I think you're running into ghost repeating attributes. I ran into this with my XRay script (part of InsertArg). Only one of them actually reports to the sheet, but the others are accessible to the API. Here is a function that retrieves the actual unique ordinal of a sub-attribute within the section (ie, the sub-attribute "name" within the "weapons" list/section.) const repeatingOrdinal = (character_id, section = '', attr_name = '') => { if (!section && !attr_name) return; let ordrx, match; if (attr_name) { ordrx = /^repeating_([^_]+)_([^_]+)_.*$/; if (!ordrx.test(attr_name)) return; // the supplied attribute name isn't a repeating attribute at all match = ordrx.exec(attr_name); section = match[1]; } let sectionrx = new RegExp(`repeating_${section}_([^_]+)_.*$`); let createOrderKeys = [...new Set(findObjs({ type: 'attribute', characterid: character_id }) .filter(a => sectionrx.test(a.get('name'))) .map(a => sectionrx.exec(a.get('name'))[1]))]; let sortOrderKeys = (findObjs({ type: 'attribute', characterid: character_id, name: `_reporder_repeating_${section}` })[0] || { get: () => { return ''; } }) .get('current') .split(/\s*,\s*/) .filter(a => createOrderKeys.includes(a)); sortOrderKeys.push(...createOrderKeys.filter(a => !sortOrderKeys.includes(a))); return attr_name ? sortOrderKeys.indexOf(match[2]) : sortOrderKeys; }; H/T to The Aaron for the original code, slightly modified to the above.

Thanks... I have to read and understand the code, but I understand it allows to find the "live" attribute among the "ghost" ones... Any idea how the ghost happened in the first place (would be good to fix it). I suppose in presence of ghosts, a routine to delete them could be devised, but maybe at risk to delete the "live" one as a casualty. Thanks a lot

That finds one of the duplicates... I don't think it matters, because all of the duplicates update the same data on the backend, and they are all references by the same ordinal position. Alternatively, it's been a while, but I think if you don't supply the attr_name to that function, I believe it returns the whole set of list items from the section, properly ordered and indexed. Reading through the code really quickly, that seems right. As for how it happens, or how long it's been happening, I don't know. It's something on the roll 20 back end. I would not delete any of them, for The same fears you have.

more about the ghost rows: <a href="https://app.roll20.net/forum/post/9643364/bug-ghost-rows-in-repeating-sections-when-a-dash-is-used-in-the-section-name/?pageforid=9643364#post-9643364" rel="nofollow">https://app.roll20.net/forum/post/9643364/bug-ghost-rows-in-repeating-sections-when-a-dash-is-used-in-the-section-name/?pageforid=9643364#post-9643364</a>

Thanks Andreas, I am not sure if it is the same problem as I have, and as described by timmaugh... In the link you sent it is an issue where various repeating sections have "similar name" or using characters like dash/numbers, and the system seems to mix them up... In my case it happens all in the same repeating_section... To give more details, up to now I only detected it on the _RowID attribute of the repeating section. This is an attribute that is managed by the API and linked to the vandalization of rowIDs... Because sheetworkers work (to my understanding) case insensitive, and eventInfo triggers are lower caps (including the rowID), but the API is case sensitive, the API is unable to deal directly with any rowID sent by the sheetworker. We worked around that having a routine in the API that whenever it detects a new repitem, writes the unvandalized RowID in the repeating_XXXX_rowID. This is the one that now gets duplicated, (but there could be others that are not detected... It takes a _.each where we count/accumulate them to see the problem... ), and I wander if this could have something to do with the fact it is the only case I can think of where the APi automatically writes something based on a change of attribute. Do you still think it is the sae bug ? If not I could make a bug report to roll20&nbsp;

I think it's the same problem... perhaps another trigger, but the same problem. I would see if you can figure out a minimum reproducible example that demonstrates your suspicion RE: the cause, then share that with the help desk, referencing the bug report Andreas linked to as a possible sibling/clone.