Sustainability of SOTA

Hi all,
The recent downtime of SOTA resources made me wonder if we should think about the sustainability of the whole, great SOTA scheme and its associated data.

As far as I understand, the entire data and IT infrastructure is currently managed by the MT, and the relatively few people behind that. I greatly appreciate what they have been doing and are doing for the community, of course!

However, I think we should have mechanisms in place so that at least the raw data sources, i.e.

  • summit master data
  • activation & log data
  • and maybe other types of data

will remain accessible even in the worst case scenario of the entire MT being hit by a bus or any other catastrophic event.

A simple solution would be to generate dump files from the relevant databases in regular intervals, and publish them as tar/gzip files so that others could mirror and archive them.

While this would not preserve the entire IT ecosystem, it would at least allow for a fresh start in the worst of cases.

WSPR has this:
http://wsprnet.org/drupal/downloads

And so has the RBN:
http://www.reversebeacon.net/raw_data/

This could be complemented by putting at least parts of the underlying software on a Github repository with a suitable (e.g. non-commercial) open-source license.

And again, please see this suggestion in the light of great appreciation of what the MT has been and is been doing for us mountain-loving amateur radio operators!

73 de Martin, DK3IT

2 Likes

We’re already taking steps to do this. Including making the MT physically separate so it’d need several buses trans-continentally :smiley:

We already do that…via summitslist.csv The rest of the data is surprisingly large - in the 100s of Megabytes, but we already maintain a second copy of that at semi-regular intervals.

This issue was largely unavoidable in the short term due to the way the hosting works. In the medium term, we’ll need to put a bit of resilience in at some expense. We’ll work out how to do that in the best manner.

This is my day job, so yeah, we should be fine - it’ll just take time.

Andrew
SOTA IT Group Leader

5 Likes

In terms of putting a bit of resilience into the database/Web site: do you have a goal for time to recover from a database failure such as just occurred? One day, 6 hours, 1 hour?

Thanks for your efforts.

73, Barry N1EU

1 Like

I will reply here, even though some of this pertains to things posted by others.

SOTA is not a financial institution. Instantaneous recovery is NOT a requirement. Very good average reliability, and recovery within days is even more than we NEED. And the difference between need and want is usually the cost. The only requirement for SOTA sustainability is the efforts of dedicated and talented VOLUNTEERS. We could not afford to pay the MT what they are worth. We can not afford a 100% percent up time database.

Whenever something does happen, a data backup was always available. I have to assume that the backup was available to more than one MT member. They are a diverse group spread out over a very wide area. I doubt they will ever find themselves all on the same bus. Having a fail safe plan within the MT is a good plan. Making the data available outside the control of the MT is not a good plan (in my opinion).

If anyone wants to help the MT continue to maintain SOTA, go to the SOTA store and buy stuff. Or make a donation through the store. Or do both. And I am confident that does happen.

I worry more about funding for the SMS capability for SOTA spotting. I think those are personal costs paid for by the individuals that maintain those systems. I would like to hear that SOTA helps to cover some of those costs for SOTA approved SMS systems in different geographic areas, thus making local calling numbers available to such large numbers of hams.

Glenn AB3TQ

2 Likes

Hi Glen,
In case this relates to my original post, I think you misunderstood my intention, and I should have been clearer. I have tons of respect for the great volunteer work done by the MT and others that made SOTA possible and keep it alive.

My only point was that it would be good to have persistent, publicly available archives of the master and transactional data of the SOTA services so that it would be possible to revitalize the SOTA scheme in case the old community fell apart.

I have zero expectations about uptime and reliability; I know how quickly effort and costs rise with service-level assurances, and after all, this is a hobby, for everyone.

But I still hold down the opinion that taking measures to make the whole great SOTA ecosystem more sustainable and more resilient by publishing data dumps and making as much of the software open-source is desirable.

Take for example RBNhole - it is a great service, and when its author Eric June, KU6J, became a silent key, it was, if I understood correctly, quite some effort for Andrew, @VK3ARR to restore the service. Good to know that the new codebase is now available at vk3arr · GitHub.

And I think we should strive for the same level of shared access for other essential parts of the SOTA ecosystem.

Again, the least I wanted to express is ingratitude over the MTs great work.

73 de Martin, DK3IT

1 Like

I’d never put a hard goal on a volunteer run service, but it should be relatively straight forward to put in some single-point-of-failure resilience such that an outage like this could failover with little or no outage. It’s always a cost to performance tradeoff though.

IANAL, but I think we are at a good spot in that with the summitslist.csv file. Different jurisdictions have different requirements around data privacy and individual logs would probably fall into that grey area if they were publicly accessible.

I get the gist of what you are suggesting though. We will publish more via github as time goes on, but if we lost every system tomorrow, it wouldn’t take nearly as long as it would have 1 or 2 years ago to redeploy things.

/me goes to find some wood to touch

2 Likes

I think I understand your intention, Martin. In principle I agree. But when I read of the measures already in place to back up and in the event of a complete meltdown of the hosting services systems and perhaps their viability, the ability to restore services in a reasonable time all seem to be well catered for, especially for a free service!

Having a linked system with a common reference for summit validation does expose the sotawatch/sotadata combination to potential outages in the event of the common database being unavailable. But that was put in place to remove the conflicts between two summit reference tables being independently maintained, which irritated some users occasionally and created more work for the MT. One instance of common reference data is a smarter solution.

It very rarely fails like it did recently.

Perhaps each activator needs to consider how they would operate, alert potential contacts etc in the event of sotawatch being unavailable. Be prepared. Each operator could and should have alternative arrangements.

At the very least, there are the reflector and the mailing lists that exist for every population centre - I know of several in VK and the North American mailing group. I’m sure there are country-specific lists all over the world. [edit: I forgot to mention the forgettable facebook groups - another option]

If all else fails, the activator may be face with an activation that struggles to achieve the 4 contacts required in the usual way (operate on alerted or spotted frequency, sit back and wait for the contacts to come to you). Gadzooks! Does this mean an activator might have to answer ordinary cqs and make ordinary contacts! Yes! (warning: also may require use of tuning knob on radio, considering other bands, calling cq a bit more yourself, having more than one mode option available).

Then there’s the burning issue of uploading the logs. Relax, guys, they can wait. The points will come. You will get your awards.

All up I think SOTA is eminently sustainable and well supported.

Thanks to the MT and especially the IT group…

3 Likes

Based upon what I have read, it seems like this is a pretty good system. Nonetheless, I think it is a good idea to ensure individually that you have a record of your sota activations & chases. Its easy to periodically down load your own logs and I have now done this.

1 Like

Hi all,

I would like to thank the MT and particular Andy for rescuing our data and installing it back onto the server. I have now installed 3 activation logs by csv and one S2S, all went smoothly.

I have also downloaded my activation csv back-up as I do every month.

I hope your nails grow back soon Andy!

Regards
David G0EVV

2 Likes

I’m more concerned about his stocks of Malt Whisky :smiling_imp:

I’m having a Dry January but I didn’t start till Jan 2nd. So I should run till Feb 2nd. So far not one drop has passed my lips.

   _So far not one drop has passed my lips._

Andy, are you saying you are using a drip IV for that purpose… again? Tsk, Tsk. I promise not to tell anyone else for 48 hours, to give you time to get out of town. My lips are sealed

Elliott, K6EL
[/quote]

…nah - he’s snorting it!

Its worse than that! In Scotland they put whisky in everything - marmalade, cheese, ice cream, tea, coffee, chocolate, but I reckon he is having whisky added to black pudding and chips!

Interesting. February 2nd is known for being something of a drinking Holiday in western Pennsylvania (Groundhog Day). Though I suppose Andy can celebrate from a distance. Its the “spirit” that counts.

I have relatives from Michigan which is next door to Pennsylvania I think.

Much as though it would be easy to bask in the reflected glory of my colleagues I did very little. It’s nice to be thanked, but my colleagues were also being proactive in case we ended up in bigger trouble.

They need a vote of thanks more than me. I pushed a few web buttons and raised the tickets. Trying to answer emails and keep people informed was much more effort!

Surely it’s a deep fried Mars bar.

After the problems with the database I realized I have not contributed financially to SOTA despite doing now 32 activations. I logged in this morning to specifically find out about contributing and I found the Donations link. I’ll put in a little more after the bills are paid.

Mark
N0MTN

I don’t like Black Pudding but did have some sausages that contained a small amount of Black Pudding once when staying in Tongue. I can’t remember the name of the B&B but the cooked breakfasts were wonderful as were the sausgages. Still don’t want Black Pudding on its own as part of my breakfast.