This is to notify everyone that the SummitsList.csv file location maybe changing.
I’ve been looking into why the update code sometimes fails to run to completion and I’ve narrowed it to a race between the update code and the web server code.
To fix this I am intending to make the SummitsList.csv file dynamic. i.e. when you click the download link the database app will generate the data and send it to you rather than you asking to download a file. The upside should be the data you get is always the absolute freshest data and requests to update and download at the same time will work. The downside is all those apps which request the data via a hardcoded link will fail to get the new data.
When I get this working users will still be able to get the file by clicking the link which will be in the same place, just the underlying guts will be different. I will leave the SummitsList.csv file where it is now but it will not be updated once I switch over. I’ll add some data to the file that says it’s not being updated in such a way that it shouldn’t break things.
If anyone thinks they have a better way of making IIS 7.5 not lock the file that ASP.NET is trying to update, then let me know because this is the only way I can see that is bulletproof. I want to get this fixed because I’m fed up of getting emails from my software saying something is broken for no good reason.
So will our Apps still be able to get a current updated list even if it’s not on the same URL ?
I’m not familiar with IIS or ASP.NET but would it work better if you arranged that the data was exported to a temporary location and then moved or copy it to the final location a little later ?
On Linux when I need to make sure I don’t cause file corruption or lockout I check that the file I want to operate on was last modified a period of time ago (i.e 60 seconds) before doing anything with it.
At present you hit http://www.sotadata.urg.uk/summitslist.csv and the static file is served. I’m hoping you’ll be able to hit http://www.sotadata.urg.uk/summitslist.aspx instead and the file is generated and served but things are being very uncooperative. I have a dynamic button on the web page which works fine. I just can’t get the blank aspx page to trigger the c# code. I can’t see the error for looking.
I’ve tried the temporary file tricks Stewart. I did them a while back and it was much better because the window of opportunity was reduced. But if I’m trying to delete/rename the file and IIS has just grabbed it to serve then I’m stuffed.
p.s. those links have typos as I don’t want to actually trigger anything on the server.
That’s the canonical way of doing it, but I’ve heard NTFS/FAT can sometimes not cooperate, file move should be atomic on most filesystems.
Another ugly hack would be to store the summitslist.csv file in a BLOB in SQL Server and wrap that in transactions, delivering the data out of the BLOB, but I feel dirty just writing that.
The issue appears to be IIS (the web server) locks the file when it serves it. I just happen to try to delete the file whilst it is locked and it fails. I could catch the file exception and repeat it a few times and then fail because that should work but that feels dirty. The correct way in my opinion is for my app to generate the data when needed and not have static files laying about. On my (obsolete) 2013 vintage quad core i7 HT 3,4GHz running VisualStudio, SQL server, IIS Express, the actual app takes under a second to do a 5 table SQL join to fetch the data and then generate 12MB of CSV file in C#. OK the tables are cached but the time for the production webserver to do this from no-cache should be less than most people’s download time of the data.
Sometimes I really hate ASP.NET, mainly when it’s in donkey mode and you can’t make it move no matter how many carrots are offered.
Is there any way of copying it to the final location that could handle locking, so it waits until it can gain it’s own lock on the destination before replacing it in a way that would prevent IIS locking it during the copy ?
This is when the Unix file semantics would be useful, but that is not an option.
How about making timestamped files, then updating a 302 redirect to the current version? A 302 is a temporary redirect, so the next request would come back to the original URL. Assuming that the clients aren’t stupid, but that wouldn’t be your fault.
Updating the 302 should be atomic, so each request would get one complete, valid file, either old or new.
As someone who worked on a web spider for a decade, a client that can’t handle a 302 is broken.
Spiders must deal with all kinds of broken servers, including a nasty bug in the AS/400 web server (did not close the connection after a 404). But a 302? Fix it. No sympathy here.
HTTP had a clearly defined 302 response slightly more than twenty years ago. Feb. 19, 1996.
Lots of ideas and things to try. There are some limits as to what I can do as we’re on a shared IIS server but maybe URL rewriting will let me update one file whilst users get served the other file. There are gotchas that changing some files would restart my application thread. I shall do some digging tomorrow but I still feel that me generating the data is a better way than a static file.
I’m assuming that ‘serves’ really means ‘reads’ as far as the filesystem is concerned, so why does it lock it? You only need to lock a file when you are changing it. It sounds like IIS (or the way you call it) needs to be changed.
Yes, thanks for the links, I’ve done lots of reading on this last night. The problem has been trying to find out why it would fail randomly and now it seems to be an access issue I can try ways of fixing it that works with ASP.NET and IIS on a shared system.
There must be a check if locked, wait, try again solution that works. Just needs implementing. The other idea is URL rewriting then I can point the input URL to any file whilst I update another etc. Or dynamic generation from a new link.
The ability of the app to disable itself whilst it’s running so users get the maintenance message, do updates and then enable itself is an excellent discovery I made yesterday. Before I hard removed the login page if I needed uninterrupted access, this way is elegant. I’ve not needed the facility enough in the past but now I can, I’ll have to use it.
There are read locks and write locks. You can have multiple readers or a single writer. You don’t want a writer to interfere with somebody in the middle of reading, so readers do need to take out a read lock.
There is some suggestion that IIS holds on to read locks for longer than it strictly needs to. It seems to be a fairly common problem for people wanting to update static content.
Whatever solution you end up with, would it not be straightforward to make the existing URL be a redirect to the new one? That way, clients (correct ones, at least) don’t need to know about the change.
I have to admit that until yesterday evening I had never needed to understand the why & the how of URL rewriting. Like many things MS, it can be done many, many ways. All of which are complex and not well explained !
I’m going to try some file lock detection and retrying first as that results in the smallest amount of change.
My program emulates a HTTP/1.0 GET in using Windows Sockets, it simply treats anything other than a 200 response as an error, it does not use any browser engine !
Given time I could expand the code to cope with 302 or other responses, but it has not needed to handle such things so far.
I have a dynamic button on the web page which works fine. I just can’t get the blank aspx page to trigger the c# code. I can’t see the error for looking.
but keep up the good work. I’m sure the solution will be served up in due course 