here's a crazy developer idea here.
hide the original background map completely in the client's image cache. basically, load it, but don't display it at all. Instead, load a pure black background of exact same dimensions as the hidden background map., which of course scrolls nicely.
Then, based on where the PC is, grab the relevant fragments from the original and replace the appropriate section of black with the original's chunk of visible art.
In this way, the scrolly/sizey functionality is only working with a single background image, one that starts as black, but begins to reveal itself as the original image's sections are copied onto it as the PC moves. This als means you don't have to "remember" what squares the PC has visited. The background map takes care of that for free by turning squares from black to colorful.
You also simplify calculations, as you only deal with the 2 layers when the token moves and you need to crop-copy some x1,y1,x2,y2 content from the original image to the displayed image.
