[Xastir] rtree purge

[Jerry Chamberlin]

I abuse it, usually full Internet feeds, and automatic tracking, using
tiger/topo maps. It has been fine for a few weeks not, have not had the issue
since the purge was put in.

On Tue, 1 Mar 2005, Tom Russo wrote:

> On Tue, Mar 01, 2005 at 08:34:24PM -0500, we recorded a bogon-computron collision of the <jerryc at netlab.org> flavor, containing:
> > 
> > That explaines the issue I had a week ago , before the rtree purge, My Xastir
> > grew so large it was always swapping..
> 
> If you spend a lot of time zooming in and out over many, many shapefiles,
> the rtrees can bloat quite badly.  That's why I implemented the purge ---
> after one hour of not being accessed, an rtree is freed.  The point is to 
> accelerate access of shapefiles you are rendering frequently at the expense
> of some memory.
> 
> > 128 MEG AND XASTIR WAS AT 80 MEG.. AFTYER RUNNING SEVERAL DAYS. IT APPEARS OK
> > NOW.
> > Darn Caps lock Deamon.
> 
> That's odd.  The purge happens every hour, and any rtree not accessed in the
> previous hour is freed.  It seems strange that it should take several days
> to notice the difference.  Are you sure it was rtrees that did it?  Did you
> do lots of zooming and panning with a large number of shapefiles displayed?
> That's the only thing that would make the rtree implementation bloat.  Letting
> it sit unattended or leaving it in one map view wouldn't do that.  Are
> you running with an internet feed and getting lots of weather alerts and/or
> lots of stations?  That seems a more likely source of bloat like that, and
> the time constant for cleaning it out would be much longer.
> 
> The last time I saw an rtree-enabled xastir grow to 80meg was before I 
> actually committed the code to the repository, when it still used doubles
> in the "Rect" structure instead of the floats it uses now.  That nearly 
> doubled the size of the RTree data structure --- and my run then grew to 80 
> meg because at that time I unconditionally generated rtrees upon rendering a 
> shapefile, and displayed 33 counties worth of dense TIGER/Line data (lines and 
> polygons) at once.  But I never committed that version, and only unleashed the 
> code when I was convinced it would not grow that large under normal use.  Now 
> rtrees are only generated when the shapefile is only partly overlapping the 
> visible window --- which reduces the bloat that would come from zooming out 
> to high zoom levels and rendering tons of small shapefiles that are entirely
> contained in the visible region.
> 
> Rtree trades memory and processor use for I/O access.  If you don't have the 
> memory to spare and/or a slow machine, that's not a good trade.  FWIW, the two 
> desktop machines I use with the largest number of shapefiles have over 1GB of 
> RAM, and even my laptop has 256Meg so it's a trade I'm willing to make.  I 
> don't know that I'd want to display tons of shapefiles with rtree enabled and 
> only 64 or 128Meg of RAM --- the very I/O access you'd be saving with the 
> spatial indices is probably lost again in swapping. 
> 
> > On Tue, 1 Mar 2005, William McKeehan wrote:
> > 
> > > Swap appears to be my issue...that's what you get when you have a 64 MB
> > > Machine running several things :-)
> > > 
> > > In my case, I have a total of around 25 MB total memory used by xastir - only
> > > 9 MB of that is resident.
> > > 
> > > I also re-enabled the printf statements in shp_hash.c and verified that the
> > > search is finding the file in the hash.
> > > 
> > > Thanks for the help with debugging!
> > > 
> > > On Tue, March 1, 2005 8:42 pm, Tom Russo said:
> > > > On Tue, Mar 01, 2005 at 06:28:26PM -0500, we recorded a bogon-computron
> > > > collision of the <mckeehan at mckeehan.homeip.net> flavor, containing:
> > > >> I guess it's possible that my machine is slow enough and has such a small
> > > >> amount of RAM that the rtree stuff is getting swapped out and when I refresh
> > > >> the map, it's not rebuilding the rtree, but swapping?
> > > >
> > > > I was thinking that might be a possibility, but assumed that since you
> > > > said there was minimal memory bloat that it wouldn't have been a problem.
> > > > It seems unlikely that swap could account for the rtree fetches "taking
> > > > forever" as you noted.  But I guess if there's a lot of thrashing it could
> > > > happen.  I'd consider that exactly the kind of excessive memory bloat I was
> > > > trying to avoid by purging unused trees.  I did at one time see the rtree
> > > > stuff bloat my xastir process to over 80M, but that doesn't happen anymore
> > > > both because I no longer use doubles in the rtree structure (thereby halving
> > > > the memory requirement), and do purge the unused rtrees.
> > > >
> > > > Is this on a linux box?  If so, perhaps you could watch "top" and see if there
> > > > is a lot of swap activity at such times.  That'd pretty much settle the
> > > > question.
> 
> -- 
> Tom Russo    KM5VY     SAR502  DM64ux         http://www.swcp.com/~russo/
> Tijeras, NM  QRPL#1592 K2#398  SOC#236 AHTB#1 
>  "When life gives you lemons, find someone with a paper cut."
> 

The Net Lab year 2000 and beyond Internet Education is Science
http://www.netlab.org
WA0JRJ - Jerry
used to be ICQ 6408731
used to be AIM PappyJerry