I don’t yet see the alterations in the SOTA summits listing (e.g. http://www.sota.org.uk/Summit/ZL3/CB-849); nonetheless, I’ve manually “un-transposed” the lats/longs for those few summits in the SMP db, so the mapping is no longer flooded with “nearby group borders” stretching half-way round the world
Also F/CR-265 has picked up a bogus longitude value 41511.0000.
All of these errors could have been discovered earlier by having INSERT and UPDATE constraints in the database checking that -90<lat<=90 and -180<long<=180. It won’t find all errors but it will pick up complete nonsense. Just saying.
Forgive me Simon for appearing obtuse, but I still don’t see the changes you mentioned. I’m wondering whether we actually agree on what “source” means, or should mean.
To my mind, “source” should mean a relational database where the data are stored, and from where data are drawn to supply various services like web pages, as in the example http://www.sota.org.uk/Summit/ZL3/CB-849 link I mentioned earlier.
Upon inspection of this same link this morning, I find the page still shows the wrong, transposed, latitudes and longitudes, along with some other of the ZL3 summits which prompted me to open this topic. The same goes for the global list of summits http://www.sotadata.org.uk/summitslist.csv . Does this mean that the relational database has not been altered or could it possibly be the case that the various web-pages and other services displaying summits data are not filled “on the fly” with data from the database, but are somehow pre-populated with data and exist as static texts (I sincerely hope not!) ?
Or does “corrected at source” mean that the data have been changed in the ZL3 Association/Region Manual, or some other document, and will filter through into the relational database when somebody can find the time to perform this task?
The spreadsheet that feeds the DB has been corrected. However we wait until Andy has a free moment to propogate those fixes (and a couple of others found in the process of checking). Sometimes it takes a whoopsie to put in place better checks to hopefully prevent future whoopsies. Sorry for the inconvenience caused. ZL2AJ (AM for ZL3).
In my opinion the main cause of transposed co-ordinate errors is the stubbornly wrong-headed attitude of the modern digital number crunchers, starting as far as I know with Google. For no discernable reason they have ignored a convention that has been in place for a couple of hundred years, plus the stated order of spherical co-ordinates in ISO31-11, and decreed that longitude comes before latitude. This is screwing with the minds of people who know better but still have to deal with the data. When you are having to reverse a convention taught to you in your childhood when entering data it is all too easy to slip up.
Repeat after me, “latitude comes before longitude!”
I find it doubly interesting to hear you mention “lat before long”, since the global csv export file from the SOTA database itself http://www.sotadata.org.uk/summitslist.csv lists coordinate pairs in the order longitude, latitude, i.e. in the reverse order to that which you claim should be used. This thankfully presents no problems to the sharp-eyed programmer, though - nor indeed, to the average user.
Historically latitude was much easier to determine than longitude, simply needing a sextant or similar way of measuring the angle of the sun to the horizon at midday. Longitude could not be determined until the invention of accurate timepieces made it possible to time the transit of heavenly bodies, this problem did not arise with latitude as the angle could be measured repeatedly until a maximum was determined. The priority of latitude in notation reflected this situation and was codified in a maritime conference held IIRC in Paris in the 19th century. The confusion has arisen in recent decades because programmers (who probably have never had to navigate!) assumed that latitude and longitude is on a Cartesian plane where the x co-ordinate is plotted first. They aren’t, even if a map is! I found this on the web putting it rather neatly:
“We are used to seeing latitude and longitude marked on paper maps, tempting us to think of these as planar (Cartesian) coordinates. However, they are not; the paper (planar) map is a projection of spherical coordinates, and spherical coordinates are generally written as radius, inclination, and azimuth (at least in physics.) In fact, the order radius, inclination, azimuth is codified in ISO 31-11. Geographers don’t need the radius (or to the extent they do, they use elevation/altitude, which is the deviation from the nominal radius of the earth), so we just have inclination (latitude) and azimuth (longitude). From this perspective, latitude/longitude is perfectly rational.”
As for the SOTA database, I stay away from the computer stuff as I have had minimal exposure to it prior to SOTA and am at best a novice, but you can be sure that I have pointed out that they have got lat and long ass backwards!
Not quite sure what donkeys have to do with the discussion, but if you weren’t referring to such animals, might I remind you Brian that this is a public discussion forum, open to young and old, and that such “colourful” language should be avoided in this, or any other, topic in this Forum. We don’t want the Forum moderator closing the discussion down prematurely, now do we?
But on a (marginally) more serious note, I do hope that the CSV export file formats aren’t to be changed: we’ve only just got used to this one!
“The database” is a relational database. Lat and Long are stored in separate columns of a table. There is no concept of “order” amongst the columns of a table in a relational database. SQL statements always refer to them by name.
Order only becomes relevant in an external presentation of the data. The CSV export does indeed put longitude first, but any software processing it should be looking at the column headers to determine which is which and ignoring the order.
There are obviously some issues in the workflow that pushes summit information into the database, but the database itself seems to me to be done entirely right in this respect.
Yep, because the working documents used by AMs to publish to the summits team are not the same as the canonical summit list that gets published via sotadata. Over the years, this process has evolved. It started off as summit lists in ARMs, it then moved to Excel spreadsheets, and now is in Google Sheets documents. The next evolution will depend on what is needed: do we need a separate system simply for uploading and checking summits? Or can we make do using the Sheets API to act as a kind of middleware glue between there and the published summits lists? Open question, but I tend to think the former.
Neither of which questions make the times particularly interesting, in my opinion
Of course better checks will be put in place. Will we ever eliminate whoopsies? No.
You know that I work for a large, open source vendor, and I met the other day with a large customer of ours that has an IT budget in the billions (yes, billions), of which a reasonable percentage (mid-30s) is spent purely on high-availability and disaster recovery, to eliminate outages. They have a better than six 9s uptime requirement. The reason we were meeting? They’d had an outage; human error, of course. So if they can spend hundreds of millions of dollars on avoiding errors and still have a whoopsie, then what hope does SOTA have?
Now I don’t think you’d have been so bold as to assert this, but please don’t assume because we occasionally cock up, we’re not trying to avoid whoopsies whenever we do things.
I’m a big fan of giving SOTA participants a detailed description of what went wrong when RBNHole isn’t working, even if that makes me occasionally look a little dense or not farsighted enough. But, I’ve always believed in the principle, essentially enshrined in the culture of my company, that I am not the smartest person in the room. We learn from our mistakes, and we get better; and the difference in the IT systems between when I started in SOTA, from when I joined the MT, to now is immense: it’s just not all visible, because, if it’s visible, we’ve generally made a mistake.
At the recent SOTA MT meeting, we took further steps to get the IT systems of the SOTA program in better shape: more of that in a later reflector post once we’ve tidied up the message and agreed on the milestones. Continuous improvement; kaizen; it’s all happening. But we’ll still cock up occasionally
My comments regarding the order of columns in the CSV file were merely an attempt to pull Brian’s leg ever so softly - apologies Brian.
Martyn, of course, is entirely right in his comments regarding relational databases, and in general I would agree with his statement that [quote=“M1MAJ, post:17, topic:14906”]
any software processing it should be looking at the column headers to determine which is which and ignoring the order.
[/quote]However, during the daily import of SOTA data from their CSV file, I haven’t bothered to include the extra step necessary to map column names between those used by SOTA and those used in the SMP’s own relational database. When the time comes that the SOTA CSV file structure is changed (and it hasn’t changed in some five years), then I’ll put in the extra effort involved to do that mapping. Room for improvement? of course: in a professional environment, I would make the extra effort.