housekeeping datastructures [was Re: Breaking up shapefiles Re: [Xastir] ESRI shapefiles and dbfawk]

[Tom Russo]

On Sun, Dec 19, 2004 at 10:43:20AM -0800, we recorded a bogon-computron collision of the <archer at eskimo.com> flavor, containing:
> On Sun, 19 Dec 2004, Tom Russo wrote:
> 
> > The "RTree" data structures used to do the spatial indexing are permanent ---
> > once a shapefile is loaded, its index sticks around until you shut down
> > the program.
> 
> I expect that many people run Xastir for short periods of time, or
> perhaps a few days at a time.  Others run it for extended periods.
> I have no idea what the ratio is between these two sets of users,
> but if the rtree structs stay around, that could hurt the
> long-running group.

I'm in the long-running group.  Uptime typically a week or more, and only
that short because I update the code from time to time and have to exit
to do the new make install.

> If there were an easy way to add an access timestamp for the rtree,
> perhaps one could waste that rtree or section of an retree if it
> wasn't accessed within the last XX days?  Not sure if that's
> practical with the current code though.  You tell me.

I think it would not only be practical, but trivial.  As long as there's 
some way to run a periodic cleanup operation (say once an hour).  

The current code keeps a private data structure for each shapefile found,
and stores that data structure in a hash table.
I tried to write it so that the implementation details were somewhat
insulated from the map_shp code --- all map_shp cares is that the structure
contains an element called root that has the pointer needed to pass to
the rtree library.  At the moment, the "shpinfo" structure contains only the 
filename and root of the RTree for that shapefile.  It could trivially be 
extended to have a timestamp of last access.  A periodic cleanup routine 
would then only need to run through the hash of shpinfo data, check 
timestamps, and do a recursive destruction of the structure and remove it 
from the hash table if the last.   

I would just need to write a safe destructor for a single shpinfo item
(which would pretty much mean nothing more than migrating 6 lines of code 
from destroy_shp_hash to a separate routine), write a short routine to 
scan the hash table for stale entries and kill them, and get it called 
every so often.  Make a slider in the "timing" dialog to set the period
for purging them, and the job's done. 

Hmmmm.  A bit tied up for the next few hours, but I have some dead time
coming up.  I'll see if I can explore this.

-- 
Tom Russo    KM5VY     SAR502  DM64ux         http://www.swcp.com/~russo/
Tijeras, NM  QRPL#1592 K2#398  SOC#236 AHTB#1 http://www.qsl.net/~km5vy/
 "When life gives you lemons, find someone with a paper cut."