[Xastir] Xastir on VMWare Segfault

[Tom Russo]

On Tue, Dec 26, 2006 at 11:51:47AM -0700, we recorded a bogon-computron collision of the <jewen at shaw.ca> flavor, containing:
> I had Xastir running from a terminal window, and found out that it seg
> faulted. I can't find a core dump though.
> 
> I also seem to be having problems with the shapefiles. It seemed to
> work pretty fast when I first started playing with them. Now it can
> take up to 2 or 3 minutes to render an image. I thought it was because
> I had created a dbfawk file, and it was taking a while to parse out
> the file, but I removed the dbfawk file, and still get massive delays
> when trying to use shapefiles.

I've got a theory.  I ran the VM this weekend for several days.  It never
crashed or bogged down as you describe, but when I used a bunch of shapefile 
maps it did use a bit more memory than I thought it would have needed.  I 
betcha dollars to donuts what you're seeing is that xastir is starting to take 
up more RAM than has been allocated and eventually the VM starts thrashing.  

You can see if that's the case by running "top" in another terminal 
window and watching the available memory and swap usage (up near the top of
the display).  If you start to see the swap usage go up as you use xastir, 
then yes, xastir is starting to take up more RAM than was reserved to it, and 
is trying to use swap.  If it tries to use swap a lot, then it will slow down 
dramatically.  If it tries to use so much RAM that it tries to use more swap 
than is available, it will crash unceremoniously.  I'm keen to know if that's
what's going on.

If it is, and if you have enough RAM on your computer, you could try bumping 
up the allocation to the virtual machine.  Out-of-the-box the VM is allocated 
only 256MB, and in the instance I had running for a few days I found that it 
liked to use almost all of that, sometimes coming within 2 or 3 MB of the 
available RAM.  This was with a fair number of shapefiles in use, while 
zooming and panning a little.  If I'd added a few more shapefiles that needed 
rtrees, especially large maps that had lots of shapes in them, I can imagine 
that it would start to have trouble finding room for them.  If you're running 
with shapefiles that cover very large areas, and viewing them in screens that 
only cover a small part of those areas, it's possible that your usage is 
causing some hefty memory usage due to all the rtrees.  Bumping up to 300 or 
512MB (if you can spare it on your computer) would probably help, and more
is better.  Stay within the range recommended by vmware and guide your choice
by watching the windows task manager performance meter, though, because if 
you make the VM too big then windows could start having trouble finding space 
for its own bloat, and then *it* would start to thrash to swap.

I saw no obvious signs that there was a memory leak that would cause rtree
to consume memory without bounds --- if I left xastir alone for a few hours
then memory consumption went down so there were a few dozen MB of free space.
But I can't positively rule out that possibility at this point, but am 
reasonably convinced there's none by pointing to a few years of using the 
feature without seeing RAM-related crashes or unbounded growth).

Rtree enhances shapefile speed by caching some information about shapefiles, 
trading RAM for speed.  If you have lots of shapefiles and are doing lots of 
zooming and panning you could cause a fair amount of memory bloat.  There is
some attempt to throw away rtrees that aren't being used, but that only helps
so much. 

  (Details: an rtree spatial index is built for each shapefile the
   first time it's rendered in such a way that it is not wholly contained in
   the current screen view.  The rtree stores information about the bounding
   box of each shape in the shapefile, and allows fast lookups to answer the
   question "which shapes intersect the screen?"  Without rtree, that was
   answered by reading each shape sequentially from the file and checking
   every time the screen is redrawn.  An existing spatial index is used 
   whenever the associated shapefile is again rendered into a window that 
   doesn't wholly contain it, but the spatial index is NOT accessed if the 
   entire shapefile is contained in the window.  Any rtree spatial index that 
   hasn't been accessed in an hour is discarded and its memory freed.  The 
   idea is not to bother keeping around indices that aren't being used often,
   and only use them when they help (i.e. when the answer to the question
   "which shapes intersect the screen?" is not "all of them".  Even with this
   attempt at minimizing memory footprint, rtrees can wind up taking up a 
   fair amount of RAM.)

One of the decisions I had to make when generating the virtual machine was
to pick something that would be small enough to get running quickly and would
cater to the lowest common denominator machine.  If you have enough RAM, you 
should definitely allocate more to the virtual machine than I did for the 
basic version, especially if you're doing heavy-duty shapefile stuff.  Do this 
in the "Player->Troubleshooting-> Change Memory Allocation" dialog.  Changes 
made there don't take effect until after you reboot the virtual machine, 
though.

If this really is what's going on for you, and you don't have enough RAM to
allocate more to the virtual machine, then the memory-speed tradeoff is not 
working out for you.  If that's the case,  you could always rebuild xastir 
with rtree disabled.  The VM comes with every library needed to build xastir, 
and has the cvs directory all set up for quick rebuilds, so it would just be a 
matter of adding "--without-rtree" to the configure line as described in the 
Wiki "Enhancing and changing" section of "HowTo:VMware".  I hope, though, that 
beefing up the memory allocation to the virtual machine is enough to fix it 
for you.

Another approach might be to continue to use the smaller shapefile tiles that
you used to use, and still use rtree.  With smaller tiles, the rtrees might
not grow so big, and you stand more of a chance that they're not needed because
the tiles are always completely contained in the screen.  You might get the
best of both worlds that way.

-- 
Tom Russo    KM5VY   SAR502   DM64ux          http://www.swcp.com/~russo/
Tijeras, NM  QRPL#1592 K2#398  SOC#236 AHTB#1 http://kevan.org/brain.cgi?DDTNM
"And, isn't sanity really just a one-trick pony anyway? I mean all you get is
 one trick, rational thinking, but when you're good and crazy, oooh, oooh,
 oooh, the sky is the limit!"  --- The Tick