[Xastir] rtree purge

[Tom Russo]

On Wed, Mar 02, 2005 at 07:40:04AM -0500, we recorded a bogon-computron collision of the <jerryc at netlab.org> flavor, containing:
> 
> I abuse it, usually full Internet feeds, and automatic tracking, using
> tiger/topo maps. It has been fine for a few weeks not, have not had the issue
> since the purge was put in.

The rtree purge has been around for a couple months, I think.  I put it in
very soon after putting in rtree itself, around Christmas eve.  

> On Tue, 1 Mar 2005, Tom Russo wrote:
> 
> > On Tue, Mar 01, 2005 at 08:34:24PM -0500, we recorded a bogon-computron collision of the <jerryc at netlab.org> flavor, containing:
> > > 
> > > That explaines the issue I had a week ago , before the rtree purge, My Xastir
> > > grew so large it was always swapping..
> > 
> > If you spend a lot of time zooming in and out over many, many shapefiles,
> > the rtrees can bloat quite badly.  That's why I implemented the purge ---
> > after one hour of not being accessed, an rtree is freed.  The point is to 
> > accelerate access of shapefiles you are rendering frequently at the expense
> > of some memory.
> > 
> > > 128 MEG AND XASTIR WAS AT 80 MEG.. AFTYER RUNNING SEVERAL DAYS. IT APPEARS OK
> > > NOW.
> > > Darn Caps lock Deamon.
> > 
> > That's odd.  The purge happens every hour, and any rtree not accessed in the
> > previous hour is freed.  It seems strange that it should take several days
> > to notice the difference.  Are you sure it was rtrees that did it?  Did you
> > do lots of zooming and panning with a large number of shapefiles displayed?
> > That's the only thing that would make the rtree implementation bloat.  Letting
> > it sit unattended or leaving it in one map view wouldn't do that.  Are
> > you running with an internet feed and getting lots of weather alerts and/or
> > lots of stations?  That seems a more likely source of bloat like that, and
> > the time constant for cleaning it out would be much longer.
> > 
> > The last time I saw an rtree-enabled xastir grow to 80meg was before I 
> > actually committed the code to the repository, when it still used doubles
> > in the "Rect" structure instead of the floats it uses now.  That nearly 
> > doubled the size of the RTree data structure --- and my run then grew to 80 
> > meg because at that time I unconditionally generated rtrees upon rendering a 
> > shapefile, and displayed 33 counties worth of dense TIGER/Line data (lines and 
> > polygons) at once.  But I never committed that version, and only unleashed the 
> > code when I was convinced it would not grow that large under normal use.  Now 
> > rtrees are only generated when the shapefile is only partly overlapping the 
> > visible window --- which reduces the bloat that would come from zooming out 
> > to high zoom levels and rendering tons of small shapefiles that are entirely
> > contained in the visible region.
> > 
> > Rtree trades memory and processor use for I/O access.  If you don't have the 
> > memory to spare and/or a slow machine, that's not a good trade.  FWIW, the two 
> > desktop machines I use with the largest number of shapefiles have over 1GB of 
> > RAM, and even my laptop has 256Meg so it's a trade I'm willing to make.  I 
> > don't know that I'd want to display tons of shapefiles with rtree enabled and 
> > only 64 or 128Meg of RAM --- the very I/O access you'd be saving with the 
> > spatial indices is probably lost again in swapping. 
> > 
> > > On Tue, 1 Mar 2005, William McKeehan wrote:
> > > 
> > > > Swap appears to be my issue...that's what you get when you have a 64 MB
> > > > Machine running several things :-)
> > > > 
> > > > In my case, I have a total of around 25 MB total memory used by xastir - only
> > > > 9 MB of that is resident.
> > > > 
> > > > I also re-enabled the printf statements in shp_hash.c and verified that the
> > > > search is finding the file in the hash.
> > > > 
> > > > Thanks for the help with debugging!
> > > > 
> > > > On Tue, March 1, 2005 8:42 pm, Tom Russo said:
> > > > > On Tue, Mar 01, 2005 at 06:28:26PM -0500, we recorded a bogon-computron
> > > > > collision of the <mckeehan at mckeehan.homeip.net> flavor, containing:
> > > > >> I guess it's possible that my machine is slow enough and has such a small
> > > > >> amount of RAM that the rtree stuff is getting swapped out and when I refresh
> > > > >> the map, it's not rebuilding the rtree, but swapping?
> > > > >
> > > > > I was thinking that might be a possibility, but assumed that since you
> > > > > said there was minimal memory bloat that it wouldn't have been a problem.
> > > > > It seems unlikely that swap could account for the rtree fetches "taking
> > > > > forever" as you noted.  But I guess if there's a lot of thrashing it could
> > > > > happen.  I'd consider that exactly the kind of excessive memory bloat I was
> > > > > trying to avoid by purging unused trees.  I did at one time see the rtree
> > > > > stuff bloat my xastir process to over 80M, but that doesn't happen anymore
> > > > > both because I no longer use doubles in the rtree structure (thereby halving
> > > > > the memory requirement), and do purge the unused rtrees.
> > > > >
> > > > > Is this on a linux box?  If so, perhaps you could watch "top" and see if there
> > > > > is a lot of swap activity at such times.  That'd pretty much settle the
> > > > > question.
> > 
> > -- 
> > Tom Russo    KM5VY     SAR502  DM64ux         http://www.swcp.com/~russo/
> > Tijeras, NM  QRPL#1592 K2#398  SOC#236 AHTB#1 
> >  "When life gives you lemons, find someone with a paper cut."
> > 
> 
> The Net Lab year 2000 and beyond Internet Education is Science
> http://www.netlab.org
> WA0JRJ - Jerry
> used to be ICQ 6408731
> used to be AIM PappyJerry

-- 
Tom Russo    KM5VY     SAR502  DM64ux         http://www.swcp.com/~russo/
Tijeras, NM  QRPL#1592 K2#398  SOC#236 AHTB#1 
 "When life gives you lemons, find someone with a paper cut."