I don't know that that's the logic I'd use.
The rule, as I understand it, is:
When a creature and at least one of its allies are adjacent to an enemy and on opposite sides or corners of the enemy’s space, they flank that enemy, and each of them has advantage on melee attack rolls against that enemy.
When in doubt about whether two creatures flank an enemy on a grid, trace an imaginary line between the centers of the creatures’ spaces. If the line passes through opposite sides or corners of the enemy’s space, the enemy is flanked.
The "spaces" in the second para should be understood in the context of this other point about creature size:
A Large or larger creature is flanking as long as at least one square or hex of its space qualifies for flanking.
So it really doesn't matter how large the creature is who is doing the flanking AS LONG AS it occupies (in its footprint) the square adjacent to the potentially-flanked token (let's call this Token-F. The question is how to know whether it occupies that position (and from which side of the token it flanks, so that the opposite side can be tested).
To do that, I would take the position and dimension of Token-F and increase it in 70px in all directions (accounting for rotation). This is the footprint within which an enemy token must reside in order to contribute to flanking.
(The size of the footprint could be adjusted based on the size of Token-F, if that's what the rules indicate; however, that's not my read of the rule so I will continue with a flat 70px buffer.)
You now have a box. Let's call it the Flanking Box.
For an enemy token (call these Token-1, Token-2, etc.) to be contributing to a flanking maneuver, it must 1) have some portion of its footprint intersect the Flanking Box, and 2) have an ally token on the opposite side of Token-F.
Intersection with the Flanking Box is a series of tests:
Setup A) Determine the corners of the Flanking Box
Setup B) Determine the corners of Token-1
1) Check every corner of Token-1 for presence in the Flanking Box
a) use the coordinates of the Token-1 corner being evaluated and the center of the Flanking Box as a ilne segment
b) treat every side of the Flanking Box as a segment described by the corners
c) see if the segment described in (a) intersects any of the segments described in (b) (ray casting)
2) Check every corner of the Flanking Box for presence in the Token-1 footprint (same process as above)
3) For intersections where no corners are in the other square, ie...
-------------
| |
| |
---------------------
| |
---------------------
| |
-------------
...check every segment described in (1b) for intersection with (2b) -- that is, the segments of each individual box
4) If there is an intersection, track WHICH SIDE of Token-F was involved
Then, if Token-F has an intersection on opposite sides (by enemy tokens), it is flanked. So, it has to have an enemy at N-S or E-W (or both).
Finally, for good measure (and to make limit the amount of re-figuring I had to do), I would 1) build the set of flanked tokens in the on('ready') event of my script, and 2) track all flanking situations within the closure of the script so that if I moved Token-1, I could look at all flanking maneuvers that token was involved in. If there were no other flanking situations on the targeted token (Token-F), remove the marker. That is, if I move a token away from having a N/S flanking situation on Token-F, Token-F should lose the "flanked" marker UNLESS there was another set of tokens which still had that token flanked. Then I could use the new position of Token-1 to see if a new flanking situation had been entered into.
*catches breath*
However, all of this is a TON of computation, and a prime candidate for a quad tree. The Aaron wrote one for Roll20 (at least before JumpGate dorked with the Path coordinates), and I wrote an interface to make using it in a script easier. I stopped working on it when I had to add ellipse collision, but for your purposes (only needing rectangular and point collision), it would work great. It has all of the math I described (above) as a series of functions you can simply pass your points/shapes into. Let me see if I can find those to share...