[Xastir] rtree purge

[Tom Russo]

On Tue, Mar 01, 2005 at 08:34:24PM -0500, we recorded a bogon-computron collision of the <jerryc at netlab.org> flavor, containing:
> 
> That explaines the issue I had a week ago , before the rtree purge, My Xastir
> grew so large it was always swapping..

If you spend a lot of time zooming in and out over many, many shapefiles,
the rtrees can bloat quite badly.  That's why I implemented the purge ---
after one hour of not being accessed, an rtree is freed.  The point is to 
accelerate access of shapefiles you are rendering frequently at the expense
of some memory.

> 128 MEG AND XASTIR WAS AT 80 MEG.. AFTYER RUNNING SEVERAL DAYS. IT APPEARS OK
> NOW.
> Darn Caps lock Deamon.

That's odd.  The purge happens every hour, and any rtree not accessed in the
previous hour is freed.  It seems strange that it should take several days
to notice the difference.  Are you sure it was rtrees that did it?  Did you
do lots of zooming and panning with a large number of shapefiles displayed?
That's the only thing that would make the rtree implementation bloat.  Letting
it sit unattended or leaving it in one map view wouldn't do that.  Are
you running with an internet feed and getting lots of weather alerts and/or
lots of stations?  That seems a more likely source of bloat like that, and
the time constant for cleaning it out would be much longer.

The last time I saw an rtree-enabled xastir grow to 80meg was before I 
actually committed the code to the repository, when it still used doubles
in the "Rect" structure instead of the floats it uses now.  That nearly 
doubled the size of the RTree data structure --- and my run then grew to 80 
meg because at that time I unconditionally generated rtrees upon rendering a 
shapefile, and displayed 33 counties worth of dense TIGER/Line data (lines and 
polygons) at once.  But I never committed that version, and only unleashed the 
code when I was convinced it would not grow that large under normal use.  Now 
rtrees are only generated when the shapefile is only partly overlapping the 
visible window --- which reduces the bloat that would come from zooming out 
to high zoom levels and rendering tons of small shapefiles that are entirely
contained in the visible region.

Rtree trades memory and processor use for I/O access.  If you don't have the 
memory to spare and/or a slow machine, that's not a good trade.  FWIW, the two 
desktop machines I use with the largest number of shapefiles have over 1GB of 
RAM, and even my laptop has 256Meg so it's a trade I'm willing to make.  I 
don't know that I'd want to display tons of shapefiles with rtree enabled and 
only 64 or 128Meg of RAM --- the very I/O access you'd be saving with the 
spatial indices is probably lost again in swapping. 

> On Tue, 1 Mar 2005, William McKeehan wrote:
> 
> > Swap appears to be my issue...that's what you get when you have a 64 MB
> > Machine running several things :-)
> > 
> > In my case, I have a total of around 25 MB total memory used by xastir - only
> > 9 MB of that is resident.
> > 
> > I also re-enabled the printf statements in shp_hash.c and verified that the
> > search is finding the file in the hash.
> > 
> > Thanks for the help with debugging!
> > 
> > On Tue, March 1, 2005 8:42 pm, Tom Russo said:
> > > On Tue, Mar 01, 2005 at 06:28:26PM -0500, we recorded a bogon-computron
> > > collision of the <mckeehan at mckeehan.homeip.net> flavor, containing:
> > >> I guess it's possible that my machine is slow enough and has such a small
> > >> amount of RAM that the rtree stuff is getting swapped out and when I refresh
> > >> the map, it's not rebuilding the rtree, but swapping?
> > >
> > > I was thinking that might be a possibility, but assumed that since you
> > > said there was minimal memory bloat that it wouldn't have been a problem.
> > > It seems unlikely that swap could account for the rtree fetches "taking
> > > forever" as you noted.  But I guess if there's a lot of thrashing it could
> > > happen.  I'd consider that exactly the kind of excessive memory bloat I was
> > > trying to avoid by purging unused trees.  I did at one time see the rtree
> > > stuff bloat my xastir process to over 80M, but that doesn't happen anymore
> > > both because I no longer use doubles in the rtree structure (thereby halving
> > > the memory requirement), and do purge the unused rtrees.
> > >
> > > Is this on a linux box?  If so, perhaps you could watch "top" and see if there
> > > is a lot of swap activity at such times.  That'd pretty much settle the
> > > question.

-- 
Tom Russo    KM5VY     SAR502  DM64ux         http://www.swcp.com/~russo/
Tijeras, NM  QRPL#1592 K2#398  SOC#236 AHTB#1 
 "When life gives you lemons, find someone with a paper cut."