G4YSS: YAHOO SOTA Reflector Withdrawal

Online Resources
1

At risk of showing myself up due to general ignorance of I.T. matters, I have received a message from Yahoo in reference to the original SOTA reflector which they provided from 2001. It’s one thing to stop new posts but quite another not to allow the reading of those already in existence which is how I am interpreting it. See link below:

These reflector postings, which date back to 7th August 2001, trace the history of SOTA right from its conception by John, Richard and others. I am very concerned that we may loose this important history and that of all the members from that time. Unthinkable in my opinion! Devising SOTA was a tremendous achievement which I for one will be ever grateful for and we don’t want to loose access to how it was done especially the effort involved. Also there are messages on there from the likes of Shirley, now SK.

The email received 29-October 2019.
Dear Group Moderators and Members,
*Thank you for your commitment to Yahoo and for helping us define the power of digital communities. Eighteen years ago, we combined the functionality of a site called eGroups.com with a precursor community platform called Yahoo Clubs to launch Yahoo Groups. Since then, you and millions of others have helped prove our hypothesis, by creating and joining more than 10 Million groups. *

  • A lot has changed about the Internet since 2001, including the ways most people now use Yahoo Groups. Today, most Yahoo Groups activity happens in your email inbox, not on the bulletin boards where Yahoo Groups started in the pre-smartphone age. Increasingly, people want content and connections coming directly to them, and this is why we continue to invest in Yahoo Mail – including the recent launch of a new Yahoo Mail app that is currently the highest-rated email app in the App Store and Google Play.*

  • So, as our users’ habits have evolved, we have begun the process of evolving our approach to help active Yahoo Groups thrive and migrate to our email platform. To help you plan for these changes, below is the schedule of how this transition will happen. *

  • Beginning October 28, 2019: *
    •Users will be able to join a Yahoo Group only through an invite from the Group Moderator or by submitting a request to join a Group, which requires approval by the Group Moderator.
    *•Since we are moving Group communication from posting on message boards to email distribution, uploading and hosting of new content will also be disabled on the Yahoo Groups website. *

  • Beginning December 14, 2019:*
    •All Groups will be made private and any content that was previously uploaded via the website will be removed. We believe privacy is critical and made this decision to better align with our overall principles.
    •If you would like to keep any of the content you’ve posted or stored in the past within your Yahoo Group, please download it by December 14 by accessing this link.

  • As these dates get closer, we will send follow-up reminders. More information about the upcoming changes can be found here.*

  • While this evolution of Yahoo Groups is inspired by how we see the platform being used today, we know change can be difficult. Here are a few important facts as we make this transition:*

    1. Yahoo Groups is not going away - We know that our users are deeply passionate about connecting around shared interests, and we are evolving Groups to better align with how you use it today.*
    1. New groups can still be formed - Users can continue to connect with others around their common bonds and interests. From animal rescues to sporting and activity groups, civic organizations to local PTAs, members of our Yahoo Groups will remain connected and able to share their activities and interests. All of the content that you have shared previously on the website, can continue to be shared via email. *

  • We know that Yahoo Groups is an important online extension of your real-life group of friends, interests and communities, and we are committed to supporting communities that rely on Yahoo Groups. Thanks for coming along with us this far. We look forward to seeing where the technology – and you – take us in the decades to come.*

  • Sincerely,*

  • The Groups Team*
    …

As a contingency I have cut and pasted the first 100 posts into a word document but it’s a slow process - ongoing.

The email above apparently provides an option to download all my own G4YSS messages but presumably not the whole content. I requested this because scores of my old activation reports and many others of interest are (it seems exclusively) on this platform. Yahoo replied on 31st October as follows:

*Hello, *

  • You requested a download of your Yahoo Groups data for g4yss. We will notify you at g4yss@yahoo.co.uk when your download is ready. *

  • If this wasn’t you, go to Yahoo | Mail, Weather, Search, Politics, News, Finance, Sports & Videos immediately to cancel the download. *

  • For security, we also recommend that you go to account settings to change your password, and review your recovery phone number(s) and/or email address(es) to remove the ones that don’t belong to you. *

  • Regards, *

  • Yahoo Groups*
    …As of today - nothing heard.

I hope somebody comes back saying I’m a buffoon and it’s already been sorted, its on a previous thread or you just do this, this and this etc. That would be good.

So in summary:
Will it be possible to get all this history transferred across to the current SOTA reflector so that we have both an ‘Old, Old Reflector’ as well as the current ‘Old Reflector?’ As well as ‘Conversations’ there are ‘Files’ also and I think photos too.

OR failing that I would urge individual members to pull down their own posts if they want to keep them. It seems we only have until mid December.

As stated above I am awaiting a reply from Yahoo about downloading G4YSS content and will add further information on that subject as and when.

73, John
(GX0OOO/P)

2

If someone really wants to preserve this there is a tool that can do so. http://www.personalgroupware.com/ It will do some downloading without a license but to fully preserve everything an inexpensive license of just $24.95 is required.

3

Hi John,
You need to allow 2-3 days for Yahoo to prepare the download archive and you will get another email when it is ready. In my case a very, very, very old computer related list that I used to run, there was no data found - I think I went through and deleted all the content when I thought I had closed the list down (obviously it was still in Yahoo’s records somewhere).

Hopefully once created you will have all of the data available - in what format it will be and whether that can be used elsewhere remains to be seen I expect.

73 Ed.

P.S. - the email you receive when the data (if there is any) is ready, looks like this:

Hello,

You recently requested a download of your Yahoo Groups data. Your download request has been completed. To download your data, go to this URL: Yahoo | Mail, Weather, Search, Politics, News, Finance, Sports & Videos .

If this wasn’t you, go to the URL above immediately to delete the download. For security, we also recommend that you go to your account settings to change your password, and review your recovery phone number(s) and/or email address(es) to remove the ones that don’t belong to you.

Regards,
Yahoo Groups

4

I requested a download from Yahoo. They could not find any data. I do have data there. I’m owner of a few groups, including the one discussing the new summits and regions for W6.

Yahoo are just incompetent liars as far as I can tell.

wunder

5

I tentatively assumed that scraping the messages from the site non-interactively would be difficult, with lots of authentication hoops to jump through. In the spirit of experiment I tried using curl to access the URL of a specific message, and was stunned to find that it just works. Of course the HTML file you get back contains a whole load of unwanted guff, but critically, it contains the message.

So I’ve put this in a loop just pulling the messages one by one. I’ve deliberately slugged it to 1 per second to keep it under the radar; provided they don’t block my machine it should be done by close of play today. I’ve got the first thousand already. If they do block, I have lots more IP addresses I can use (I’m not signed on, so they don’t know who it is).

This is only the messages and won’t preserve attachments, photos etc, but I think it will do as an emergency archive of the history.

Martyn M1MAJ

5 Likes
6

Excellent work Martyn!

7

I would be surprised if they even notice you. There are dozens of people right now in world wide freecycling groups downloading thousands of messages from hundreds of groups almost round the clock

8

Depends on whether they have automated rate limiting. I did read somewhere or other that some people had found themselves blocked after about 10000 messages in quick succession.

But we only have 14505 messages so there’s no rush.

9

Thanks Martyn - it would be good to have that archive somewhere. There must be some gems.

I don’t suppose that they can be added to the current system somehow?

10

Requesting a download from Yahoo didn’t work the first time, so I’m requesting it again. That downloads groups in mbox format, which is pretty useful. For files, I’ll probably do it the manual way for a couple of groups. Sigh.

This article walks you through requesting a download.

wunder

11

Hi Wunder,
My downlaod request from Yahoogroups didn’t produce any data either the first or second time that I equested it and waited 2 days each time.

Perhaps the download option that yahoo are sending out simply doesn’t do anything?

73 Ed.

P.S. following the process given in the Verge article takes me to the same place as the link in the email from Yahoo sends me to and there’s no data.

12

Several people I know have been able with http://www.personalgroupware.com/ to restart after the 10000 stiop.

13

My simple-minded grab of the messages did not encounter any problems.

So I now have a bunch of files each containing one message deeply embedded in a humungous amount of Yahoo boilerplate. I will at leisure run a script over them to extract the interesting bit.

If anybody would like the raw data now, send me a direct message and I’ll give you a Dropbox link. Be aware that it is 238MB as a zip archive and will expand to not much shy of a gigabyte.

Martyn M1MAJ

3 Likes
14

Nice work, Martyn!

15

Could you use wget? Not sure how that’s different to curl not being a bash expert. I used it to save a copy of g4ilo’s pages when he went sk.

16

It’s essentially the same thing as curl. The issue was/is not which tool you use to fetch the data but the rate at which you keep asking the server for new files. As Martyn said it’s fetched now.

17

Hi to all,
These responses are very heartening as I was really worried that SOTA would loose its history. For anybody who is at all interested in SOTA this early reflector provides a real insight, so absolutely well done to Martyn. If it all goes belly up today, Martyn is already in possession of the necessary even if a lot of ‘silt’ came down with them. Attachments, photos and other files will likely exist on the poster’s hard drives but most of the messages won’t.

Thanks also to Jim for his suggestions and additions from Steve, Wunder, Ed and Richard with encouragement from Tom, Andy and Brian.

Richard said, ‘I don’t suppose that they can be added to the current system somehow?’ That would be really good!

By the way, I still haven’t heard a peep back from Yahoo so if nothing soon, I will follow the example and request again. I now have the first 200 posts copied and pasted but as it’s taken me two hours for that. I couldn’t imagine trying to do the lot - over 14,000 it seems. It now appears I won’t have to so thanks again for the responses.

73, John.

18

Good stuff! Glad to hear it got sorted

19

I did use PG Offline 4 to retrieve photos and files for a number of yahoo groups I am a member of. In several cases the files retrieved have been uploaded to the groups.io group which has been set up as a replacement. The files upload as a structure, but the photos need to be handled one folder at a time. The automated upload offered by groups.io is a far easier method, but now has a fee which not all groups seem to be prepared to pay.

The “conversations” or text messages are stored by PG Offline in its own “database” which has a DB3 extension. I don’t know what that is internally. It offers the option of exporting the texts to various forms including a set of HTML documents, which makes it very easy to view. I don’t know whether there is a neat option to export the texts to a server to make it accessible to online users. Html would do that very neatly but there are limits in the unlicenced version of PG Offline. It maybe worth buying the licence just to process all those texts and produce accessible files for a server.
This may be useful in relation to the old SOTA reflector.
Andrew VK1DA/VK2UH

20

Apologies for picking the tail end up, is this why GQRP have just migrated to groups.io?

Chris
2E0FRU