Breaking up shapefiles Re: [Xastir] ESRI shapefiles and dbfawk

[Tom Russo]

On Sun, Dec 19, 2004 at 12:32:40PM -0500, we recorded a bogon-computron collision of the <mckeehan at mckeehan.homeip.net> flavor, containing:
> In my typical situation, I have 4 shapefiles loaded:
>   State Outlines
>   County Outlines
>   Interstates
>   Major Roads
> 
> All of these are nation wide, so the coverage is very large. My most common
> range of viewing with these shapefiles has been 2048 down to 64.
> 
> Once I zoom in closer than that, I use downloaded Tiger Map shapefiles that
> are seperated by county.


Yes, I use those TIGER county shapefiles, too.  But when I'm zoomed out
to state level where all 33 linear feature and 33 polygon feature shapefiles
are visible, my memory usage goes up to about 60MB (from around 20).  And
stays there.

> I did see an increase in the memory used - about an 8MB increase (from 20 to
> 28), but it was only temporary.

The "RTree" data structures used to do the spatial indexing are permanent ---
once a shapefile is loaded, its index sticks around until you shut down
the program.  There are cases (such as I experienced during my initial testing
of this feature) where memory usage can be extreme and permanent.  If you've 
chosen your map min/max zoom levels appropriately so that the county 
TIGER/Line shapefiles are not loaded when you're zoomed out to cover more
than a few counties, that's probably OK.

Since I don't have a "Major Roads" shapefile of any accuracy, I rely on the
major roads in the TIGER shapefiles and the fact that dbfawk filters out the
low-level detail at high zoom.  This loses badly with the RTree if I'm zoomed
at state level (which I rarely do).

I'm glad that so far your typical usage had gotten you the benefits of the
indexing without the drawback.  In my own typical usage it's been a positive,
too.  

I might have to experiment with my own shapefile-extracting tool and get myself
a set of major roads that I can use at high zooms, and turn off county-level
TIGER data when zoomed out that far.  That way, I'll only get RTrees built
for shapefiles that I actually view at low zooms.  Hmmm.  

-- 
Tom Russo    KM5VY     SAR502  DM64ux         http://www.swcp.com/~russo/
Tijeras, NM  QRPL#1592 K2#398  SOC#236 AHTB#1 http://www.qsl.net/~km5vy/
 "When life gives you lemons, find someone with a paper cut."