[Xastir] Looking for serial->IP widget

[Steve Dimse]

On 1/15/03 at 2:36 PM jjeffers at aprsworld.net (James Jefferson) sent:

>> 2) Create an SQL database that's fed by the I/O daemon.
>
>I've been running aprsworld.net for over it year. It collects approximately a 
>million packets a day, parses them, and inserts them into the appropriate 
>table(s). It works, I'm serving thousands of visitors a month that data. But 
>it is *VERY* IO and CPU intensive. SQL databases are general purpose and 
>cannot easily be optomized for the demands that Xastir would place on it.
>
Which approach is best depends on your application. If the important quality is
speedy access to random pieces from a large body of data, you are not going to
beat a database. My parser does similar parsing to Jim's, and the server dishes
up more than 3 million dynamic web pages a month. This takes a rather powerful
system, with dual processors and SCSI drives. Full parsing of all data into a
relational database is overkill for most applications. 

>For short term storage the database works great, mainly because everything is 
>in memory. I have tables of the last position, weather, and telemetry of all 
>stations. They are small and work well. But the big tables with all of the 
>positions for all stations become a nightmare. They are really hard on the IO 
>sub-system and a complete mess when they get corrupted.
>
I'll take exception to this. For short term storage, nothing is going to beat
keeping the data in your own memory, assuming the data structure is properly
chosen and implemented. Even an RDB with tables cached in memory can't compare.
However, for long term storage of very large quantities of data, nothing is
going to beat an RDB. My weather table has every weather report parsed since
findU went online in March 2000, some 100 million, and is 8 GB, with a 4 GB
index. Say I want to know my temperatures on Christmas 2000:

mysql> select temp from weather where call = "k4hg" and time_rx between
20001225000000 and 20001225235959;
+------+
| temp |
+------+
|   66 |
|   66 |
|   66 |
|   66 |
.....
|   67 |
|   67 |
|   67 |
|   67 |
+------+
274 rows in set (0.43 sec)

Less than half a second to get any day's weather in the last three years is not
bad at all! This is on the production system, handling all the other demands on
findU. This sort of thing is where a major RBD shines.

If your big tables are a nighmare, then perhaps you do not have them optimally
indexed, that is the key to keeping them accessible.

Granted, if a table gets corrupted, it can take hours to rebuild, but in the
three years findU has been up, this has happened three times, once due to a hard
disk crash and twice due to power failures that exceeded the UPS's ability to
run the server. 

Steve K4HG