ADIF upload appears to be broken. It is spinning on the Submit button.
Initially it was spinning on verify log. I logged out and back in and it restored the log and verify worked. But upload just spins.
So I logged out and back in again. This time it let me upload the restored log.
I am glad to see that I am not the only one having issues with the new DB… Never had trouble with the old one…
Hmm, it has just taken me quite a few attempts to enter my latest activation. The system kept randomly throwing out contacts each time that I edited the log. Bizarre to say the least. Finally all 58 QSOs are logged. If any disappear, as Arny said. “I’ll be back!”.
There was a glitch on the backend that is related to the other glitches we’ve had and is also part of the changes I’m testing now. What happens is the backend system connects to a database that contains all the logs. It maintains a bunch of open connections and it appears like when load surges at various points in the day, some of those connections appear to be open, are reused but are not open and there’s a failure. When it’s retried a different connection is used and the request succeeds. At least, that’s what it looks like is happening.
We can’t update the underlying libraries to see if this will help matters because of other software that runs on the same system. To fix this, basically we’re completely changing the backend API to eliminate the potential for these kinds of errors, and to give us much better horizontal scaling. Because it’s a rip and replace, we’re doing lots of testing, automated and manual.
In the process of doing that, I’ve found a bunch of bugs in the older system that have been there since 2018 or before, so these have been fixed in the newer system and are part of the changes noted in my previous post.
This is the backend, so it affects everything, whether you are using the new DB interface or the old site at old.sotadata.org.uk. The “Database” or SOTAData3 consists of three layers - the database itself, where everything is stored, an API that sits in front of that database that controls how the database is accessed, and then a frontend interface that accesses the API to present data to you, the user. What you refer to as the “New DB” is only that front piece. This issue is part of the API so has nothing to do with the “New DB” or “Old DB” distinction.
I will note that you are experiencing issues with the new DB interface at a rate seemingly about 25 times higher than anyone else, so I’m intrigued to see how you are using the DB interface. No one else is encountering the issues you are reporting to the MT, and we can often see the backend returning the correct result. When we test with the same data you are submitting (pulled direct from the logs) this result is then displayed correctly.
Are you using an old browser, or have an ad-blocker or some kind of privacy guard that’s being aggressive in blocking javascript actions?
4 Likes
Update coming
The release I’ve spoken about on this thread, and on others is just about ready to go. It has undergone substantial testing, including via the toughest test suite there is, Andrew @VK1AD who has this uncanny knack of finding code paths as yet unexplored and providing extraordinarily good bug reports. Many thanks to Andrew.
Over the coming days, we will gradually shift people across to the new site (our hosting allows us to scale which traffic goes to old and new sites). This is to allow us to observe how the new systems hold up under increased load (something that’s not as easy to test in a realistic manner) and back things out easily if the brown stuff makes contact with the air conditioning.
So, what is the big noise about?
New backend API
The major driver for this is to bring a new backend into play to eliminate some of the stability issues we’ve been having on the older API. The older API will hang around for a bit, but for those who are using api-db (and long accepted that I can change stuff at will and a moment’s notice), you may need to change hosts and endpoints in the coming weeks and months.
The new API provides for better scalability too, as we can easily add more compute in the event the load is higher than anticipated. We have additional compute as a result of this, which should help with speed under load.
We’ve also implemented a couple of performance approaches that should also speed up uploading and verifying of QSOs into the database. I can’t quantify exactly what percentage improvement we’re getting, but comments from our testers include “Hmm, much snappier than I was expecting”, which from Andy is high praise.
The number of bugs and cosmetic changes identified and fixed is too many to list here - but some of these have been around for a long time, such as the Completes page not working with filters, and the fixes are across both the front end display and the backend API. Some of the backend queries have been optimised and made more performant. I’m reasonably sure I’ve eliminated the horrible [Object object] display bugs (famous last words…). So progress forward, anyway.
FLE import support
We now have the ability to directly import FLE text files into SOTAData3, so you no longer need to convert to ADIF or CSV via FLE or FLEcli. The normal FLE syntax should be fine, with an additional benefit that you can add in more of the header commands (like mysota, date, etc) to enter multiple activations in a single file.
There are also options to copy WWFF/POTA references into the comments field for SOTA, and also any grid references.
To try it out, click on the Import FLE button when uploading:
Needless to say, I expect there to be bugs from this, but it’s been tested by Josh @WU7H and he’s ecstatic about it. If you use features outside of Josh’ normal workflow, we may exercise code paths hitherto unexplored.
New Badges
We’ve added the following badges:
- Additional Star Wars themed badges for activations on the 4th of May. We now have the Padawan, Jedi, Jedi Master and Obi-Wan badges for 2, 3, 4 and 5 years of activations and chases on the 4th of May.
- Height based badges, for chasing and activating summits with heights of 1km, 2km, 3km, 4km, 5km, 6km, 7km and 8km (the latter two not yet having summits in the database to claim), and a Low Rider badge for activating or chasing a summit below 200m in height.
- DXCC badges for chasing. This is based on the DXCC entity of the association, and is awarded for chasing 10, 25, 50, 100 (Century Club) and 150 DXCC entities. If you activate a border summit in another DXCC - say a DL summit from the OE side, you’ll give chasers credit for the summit DXCC (DL), not your actual location. I make no bones about this because it’s much easier to calculate than trying to work out actual DXCC across an entire log, so if anyone has a problem with it, tough - I write the code, I choose the rules :D. There’s also no activator equivalent yet, as that requires me to post-process every QSO in an activator log, and I don’t think that’s a good use of compute at this point, but I’ll probably get there eventually.
These badges will become available to you when you submit your next log via SOTAData3, and only for those using the new version.
Next steps
I’ll make the first adjustment to our traffic on or around 0000 UTC on 31 August. At this point, approximately 20% of people will start using the new system. If everything goes smoothly, no one should notice any difference, other than a new upload option, some new badges and hopefully some more stability. Once you are allocated to the new system, it should be persistent and you get the same branch each time.
If I notice anything off with the rollout, I’ll remove the traffic split and point back at the main system, but if it’s all going smoothly, I’ll increase the ratio of people using the system over the coming days, with the aim that everyone is on the new system around later Saturday.
16 Likes
OK, I’m stopping the rollout, I’ve noticed some quirks that need more investigation.
1 Like
If you need testers 
73 Joe
1 Like
We’re currently sitting at roughly 50% of users are on the new API and excluding a few comparatively minor bugs that escaped testing, we haven’t seen any real major issues after I fixed the one that stopped the initial rollout.
I expect that we will move to 100% tomorrow when I can observe in the morning (and have late night calls to observe as the peak moves in around 0700 UTC)
3 Likes
Hi Andy,
Many thanks for your hard work 
73, Jarek
2 Likes
Forgive me if this is the incorrect place to post a feature request, but I did notice that on sotl.as, they way they list the first activators of a summit is very nice as they list everyone on the first day. For example, Pilot Knob, W6/SS-156 : SOTLAS
First day’s activations: AA6XA & K6ARK & K6TW & KG6HQD & W6RIP on 17 Aug 2019
I would love to see the individual SOTA summit pages list it like this (right now, it’s just the very first person to make a contact, even if it was 1 minute before another op made the 2nd contact). It would be nice to see all activators get credit and might even prevent an argument over who goes first 
Thank you,
Tim
K6TW
3 Likes
I love the upgrades. Managed to get three new badges in one hit!
I feel a further donation to the programme will be on its way soon. A massive thanks to the MT volunteers for the work that goes in to this.
Indeed Fraser. And great to see that metric units are used. Otherwise I would get a headake 
(not a result of the altitude hi )
73 Joe
2 Likes
The clue is in the words “First Activation” vs “First day’s activations” 
1 Like
Understood. My point was that it’s my humble opinion that listing all the first day’s activators is the better way to go.
Everyone is now on the new API, with the exception of logging chases via SW3. That will switch over later, as I mull the next changes to SW3.
5 Likes
A lot of my experiences with the new code has been to test it and find a bug and revert to the old code.
Well today I got to use the production version and it’s just worked so silky smooth.
1 Like
something is wrong here… I was only active on 2 continents – and only in Europe in VHF…
73 Armin
1 Like
Yes, there’s a bug on the Mountain Explorer code, which I’ve rectified but not pushed live yet. A similar bug likely exists on the Mountain Hunter side which uses similar but not quite the same code, so I’ll have to look at that too, probably later tonight.
3 Likes
Fixed now for both MH and ME badges. You will need to submit a new upload to trigger the badge update. (Not you Armin @DL6GCA as I used your account for testing purposes 
)
2 Likes