I would just do a foreach loop, from hell that is, writing the data to a set of tables. On second pass you update if necessary because date has changed.
Doing MD5 compares just seems harder on BIG FILES to me.
On Wed, Aug 17, 2016 at 4:08 PM, Alan Bourke alanpbourke@fastmail.fm wrote:
Regarding FileSystemWatcher in .Net, I don't know how good an idea it is to monitor a whole drive like that.
-- Alan Bourke alanpbourke (at) fastmail (dot) fm
On Wed, 17 Aug 2016, at 07:28 PM, Ted Roche wrote:
That's a great question!
Obviously, since the post's subject didn't include "[NF]" you've already found your solution -- FoxPro! *wink*
I've done some document management systems in VFP, and the recursion, cataloging and checksums is easy, relatively-speaking. But the validation is an interesting twist, and a much more difficult problem.
Triggering the checking is also an interesting feature. Doing a bulk rescan would be slow and intensive, though you could tune it to not consume excessive resources, at a cost of slower checking.
Windows File Systems have some advanced features in the newer servers that would let you hook into a file system event (adding a new file or saving over an old one) to trigger your validation routine. If WinFS had ever been released, (https://en.wikipedia.org/wiki/WinFS) that would have been perfect, but alas, it was another empty vaporware promise of "The Old Microsoft." However some of "Longhorn" did end up in DotNet, like:
filesystemwatcher.changed(v=vs.110).aspx
A simpler solution might be a "Document Management System" but implementing one of these is a tough challenge in technology, politics, and technical support.
"Validity" is a bit nebulous. How are you defining that?
I mean, there are Word95 documents I can't open in Word2007, but can in LibreOffice. And is a Word document with a macro virus valid? How many versions and variations to support? How to handle password-encrypted or restricted files?
VFP would be a great tool for doing the validation, where you can use low-level file functions to read headers and calculate checksums, but complex structured documents, like MS's Compound OLE Documents, and MS's ZIP-encoded XML and JSON DocX documents, get a lot trickier. There's typically a "magic" signature at the beginning of most files that will tell you it's type, but whether all the contents have integrity is a lot tougher to determine. I suspect each format would need to be reviewed to determine if there were internal consistency checks that would tell you of corruption or truncation.
Sounds like an interesting project, though. Will be interested to hear if you find a suitable package, or DIY it.
-- Ted Roche Ted Roche & Associates, LLC http://www.tedroche.com
[excessive quoting removed by server]