File system manipulator thingy
File system manipulator thingy
Hi
So for the last 12 years (no really), I've been working on a file system manipulator program. Kind of a hobby process but also "hey, you might like using this".
Anyway. It's called NUTS (Native Universal Transfer System, and yes, it's a backronym), and the idea is to present file systems as two sides of an FTP-client-like interface (imagine a DFS image on the left, ADFS on the right, for example). But it does other things, like image (as in picture) translation, and so on.
It's also based around a plugin system so that the main program is agnostic to exactly what a "file" is, allowing all sorts of possibilities.
Anyways, obligatory github ink: https://github.com/RebeccaPhu/NUTS/releases
(Please check list of bugs, especially if venturing into ADFS (new map) territory. Take backups to be safe.)
Note: Heavily Windows based. Not interested in porting to other OSes, as it uses a lot of custom controls. But it's open source, so if anyone fancies a challenge....
(Please note - I'm not in competition with anyone. I'll update it as and when I write stuff.)
So for the last 12 years (no really), I've been working on a file system manipulator program. Kind of a hobby process but also "hey, you might like using this".
Anyway. It's called NUTS (Native Universal Transfer System, and yes, it's a backronym), and the idea is to present file systems as two sides of an FTP-client-like interface (imagine a DFS image on the left, ADFS on the right, for example). But it does other things, like image (as in picture) translation, and so on.
It's also based around a plugin system so that the main program is agnostic to exactly what a "file" is, allowing all sorts of possibilities.
Anyways, obligatory github ink: https://github.com/RebeccaPhu/NUTS/releases
(Please check list of bugs, especially if venturing into ADFS (new map) territory. Take backups to be safe.)
Note: Heavily Windows based. Not interested in porting to other OSes, as it uses a lot of custom controls. But it's open source, so if anyone fancies a challenge....
(Please note - I'm not in competition with anyone. I'll update it as and when I write stuff.)
Re: File system manipulator thingy
116 views and not a single comment?
As my project manager at work says "either I've done well and nobody has any criticisms, or I've do so badly none of you get it".
No questions? No "why even bother"s? No questioning of my coding ability?
Not even a crude dig at my sexual orientation...
Nothing.
Oh well.
C'est la vie.
As my project manager at work says "either I've done well and nobody has any criticisms, or I've do so badly none of you get it".
No questions? No "why even bother"s? No questioning of my coding ability?
Not even a crude dig at my sexual orientation...
Nothing.
Oh well.
C'est la vie.
Re: File system manipulator thingy
Well, one person starred your repository on GitHub, so there's that.
But sometimes things take a while to take off, especially when there's lot of other stuff going on.
Have you thought about using GitHub pages to host the documentation? That might give people a starting point to get into what the project offers.
But sometimes things take a while to take off, especially when there's lot of other stuff going on.
Have you thought about using GitHub pages to host the documentation? That might give people a starting point to get into what the project offers.
- TobyLobster
- Posts: 622
- Joined: Sat Aug 31, 2019 7:58 am
- Contact:
Re: File system manipulator thingy
I'm deducing this is a Windows GUI application to allow transferring files between different file systems, including DFS, ADFS and maybe more? But it sounds like it can do other things too? It's hard to understand the scope of what it can do without some documentation .
- dominicbeesley
- Posts: 2212
- Joined: Tue Apr 30, 2013 12:16 pm
- Contact:
Re: File system manipulator thingy
It's not obvious to me what it does. Is this to move files between disk images or actual machines?
Re: File system manipulator thingy
It sounds like some kind of image transfer solution, but I could be wrong about that. To be honest, I did briefly look at this, but the link led to the releases page, and so I wondered if the code was all browsable online, forgetting for the nth time that I needed to click on the code tab to see the repository. And I think I did this during the holidays (or shortly afterwards in an exhausted state) with not much time/energy to spend investigating, so there's that excuse as well!dominicbeesley wrote: ↑Tue Jan 17, 2023 11:33 am It's not obvious to me what it does. Is this to move files between disk images or actual machines?
None of this is to suggest that no-one is interested, though. Some of us pursue projects that are bizarre in comparison to this one, so please don't think we don't welcome such endeavours!
Re: File system manipulator thingy
OK, I get the idea - lacking in description. No problem.....
A little background
Back in 2011 when I first started this thing, the prevailing paradigm for moving files from one disk image to another (e.g. a DFS SSD file to an ADFS ADF file) was to open a program, load the SSD, export the file (complete with INF), then close that, open a different program, load the ADF, import the file (INF and all), and then close it.
To my mind this seemed silly: the file is the same thing to both systems, so why can't be directly transferred? I envisioned something like an FTP client. In this model, you have "local" (your computer) on the left, and "remote" (the site you're dealing with) on the right. Then you can copy files back and forth as required.
So I came up with this:
(Couple of notes: yes, that's the hard drive image found elsewhere on this forum - NUTS can handle it because it doesn't load images into memory, it operates on them directly. Yes, that's fake directories in a DFS disk image. It just makes the whole thing work more intuitively that way. There's going to be a FAQ entry in the documentation explaining this....)
The Application
So here we have a DFS image on the left, and an ADFS hard disk (Risc OS) on the right. Should you so desire you can just select something on the left, hit "Copy" and off it goes. No intermediate step. (In fact, if such a file is under 4MB the "holding store" for the data in transit is in RAM).
OK, so that's the basic idea. Here's what makes it different: everything uses the "agnostic" idea of where stuff is. So for example, the ADFS (Old Map) file system handler works equally well on an ADF/ADL image, as it does directly accessing a CF card. It's the same code that runs, because the data source is separated from the file system.
At present, while there *is* support for floppy drives, it's the standard Windows API, so your mileage will be crappy. But in theory, it'll work.
The whole system is plugin-based, so you could (for example) implement a plugin that provides access to a Kryoflux device, without having to write any file system code. The existing FS handlers will work with the data source component.
So here's a run down on the "killer" (lol) feature list:
* Support for a wide variety of Acorn/Risc OS file systems, Commdore D64s, AmigaDOS, AMSDOS, +3DOS, TRD, TZX, TAP, CDT, and so on.... there's quite a few filesystems in there, some more complete than others.
* Support for "wrappers" - data sources that implement some container over the filesystem. For example, you can (if you feel inclined) put a Risc OS 800K E disk image inside a CPCEMU DSK file.
* Support for unusual types inside other containers (internally called the "Foreign Object Protocol"). For example, you can created Risc OS ISO images that respect the ! naming, along with file types.
* Icon resolving: Where native use of a file system would show custom icons under the right circumstances, NUTS will too (turned off by default because it usually involves extra processing).
* Image "Install": Copies the contents of an image to another file system in one click.
* Tape image player: Plays back files such as TZX and TAP as the audio that they represent. (UEF to come when I figure out the most intuitive way to do it).
* Translators: Read, save, tweak and print images (of the graphic kind), text (including tokenised BASIC) and audio from selected formats.
* Font system: Original characters are faithfully reproduced by using the original font associated with objects. In the screenshot above, you can see the "BBC Micro" font in use for the SSD, while "Risc OS" is in use for the ADFS hard drive. There is not much difference between them, but they do put some characters in different places, especially above ASCII 126. Notably, the £ sign is in a different place.
* Deals with INF files
* Allows image nesting: you can modify the contents of a Risc OS Sprite file while it's inside an ADF image inside a ZIP file without extracting anything (though depending on container sizes and nesting level, it may be slow to do).
* Imaging Wizard: Sector-by-sector copy of a filesystem from one source to another (again, due to the crappy Windows floppy drive support this is unlikely to be of major use until a decent floppy interface/driver plugin appears).
* Image repair tools (selected file systems)
And there's probably more.
But that's all I could manage in 12 years with some considerable breaks.
Any questions?
A little background
Back in 2011 when I first started this thing, the prevailing paradigm for moving files from one disk image to another (e.g. a DFS SSD file to an ADFS ADF file) was to open a program, load the SSD, export the file (complete with INF), then close that, open a different program, load the ADF, import the file (INF and all), and then close it.
To my mind this seemed silly: the file is the same thing to both systems, so why can't be directly transferred? I envisioned something like an FTP client. In this model, you have "local" (your computer) on the left, and "remote" (the site you're dealing with) on the right. Then you can copy files back and forth as required.
So I came up with this:
(Couple of notes: yes, that's the hard drive image found elsewhere on this forum - NUTS can handle it because it doesn't load images into memory, it operates on them directly. Yes, that's fake directories in a DFS disk image. It just makes the whole thing work more intuitively that way. There's going to be a FAQ entry in the documentation explaining this....)
The Application
So here we have a DFS image on the left, and an ADFS hard disk (Risc OS) on the right. Should you so desire you can just select something on the left, hit "Copy" and off it goes. No intermediate step. (In fact, if such a file is under 4MB the "holding store" for the data in transit is in RAM).
OK, so that's the basic idea. Here's what makes it different: everything uses the "agnostic" idea of where stuff is. So for example, the ADFS (Old Map) file system handler works equally well on an ADF/ADL image, as it does directly accessing a CF card. It's the same code that runs, because the data source is separated from the file system.
At present, while there *is* support for floppy drives, it's the standard Windows API, so your mileage will be crappy. But in theory, it'll work.
The whole system is plugin-based, so you could (for example) implement a plugin that provides access to a Kryoflux device, without having to write any file system code. The existing FS handlers will work with the data source component.
So here's a run down on the "killer" (lol) feature list:
* Support for a wide variety of Acorn/Risc OS file systems, Commdore D64s, AmigaDOS, AMSDOS, +3DOS, TRD, TZX, TAP, CDT, and so on.... there's quite a few filesystems in there, some more complete than others.
* Support for "wrappers" - data sources that implement some container over the filesystem. For example, you can (if you feel inclined) put a Risc OS 800K E disk image inside a CPCEMU DSK file.
* Support for unusual types inside other containers (internally called the "Foreign Object Protocol"). For example, you can created Risc OS ISO images that respect the ! naming, along with file types.
* Icon resolving: Where native use of a file system would show custom icons under the right circumstances, NUTS will too (turned off by default because it usually involves extra processing).
* Image "Install": Copies the contents of an image to another file system in one click.
* Tape image player: Plays back files such as TZX and TAP as the audio that they represent. (UEF to come when I figure out the most intuitive way to do it).
* Translators: Read, save, tweak and print images (of the graphic kind), text (including tokenised BASIC) and audio from selected formats.
* Font system: Original characters are faithfully reproduced by using the original font associated with objects. In the screenshot above, you can see the "BBC Micro" font in use for the SSD, while "Risc OS" is in use for the ADFS hard drive. There is not much difference between them, but they do put some characters in different places, especially above ASCII 126. Notably, the £ sign is in a different place.
* Deals with INF files
* Allows image nesting: you can modify the contents of a Risc OS Sprite file while it's inside an ADF image inside a ZIP file without extracting anything (though depending on container sizes and nesting level, it may be slow to do).
* Imaging Wizard: Sector-by-sector copy of a filesystem from one source to another (again, due to the crappy Windows floppy drive support this is unlikely to be of major use until a decent floppy interface/driver plugin appears).
* Image repair tools (selected file systems)
And there's probably more.
But that's all I could manage in 12 years with some considerable breaks.
Any questions?
Re: File system manipulator thingy
Yes.dominicbeesley wrote: ↑Tue Jan 17, 2023 11:33 am It's not obvious to me what it does. Is this to move files between disk images or actual machines?
No really, both.
Well... sort of.
It has the plugin infrastructure for providing a means to move data to another machine. There is support for configuring a serial port for the use, or a suitable plugin could implement it some other way (I had consider a plugin that would allow browsing stairwaytohell directly, but meh), but I haven't written any such plugin yet.
Offers?
Re: File system manipulator thingy
Me sir, me, me!
Can you add Spectrum microdrive images?
I was having a look at the repository a few days ago, I'm not sure if I have a build environment that will build it, and I couldn't find any pre-built binaries. But otherwise, it's something I'd find useful, and would replace half a dozen seperate tools.
Code: Select all
$ bbcbasic
PDP11 BBC BASIC IV Version 0.45
(C) Copyright J.G.Harston 1989,2005-2024
>_
Re: File system manipulator thingy
This looks really excellent. I have to say, the screenshot helped a *lot* - I didn't really understand what you were talking about until I saw that!
I installed the 64 bit version, and have managed to do some simple things very intuitively. There are other tools around to handle disc images, but (I believe) nothing that has the range of features that you have implemented.
I did find that the sprite translator crashed though!
I installed the 64 bit version, and have managed to do some simple things very intuitively. There are other tools around to handle disc images, but (I believe) nothing that has the range of features that you have implemented.
I did find that the sprite translator crashed though!
Re: File system manipulator thingy
I imagine so. I'll put it on my list.
It's a Visual Studio 2010 build and should be self contained. It should build with a newer visual studio. It has no reliance on things like MFC (it's very pure Win32), though it will require the DirectX SDK.
The Releases page has 32-bit and 64-bit MSI installers.
Re: File system manipulator thingy
Muchos gracias.jms2 wrote: ↑Tue Jan 17, 2023 10:18 pm This looks really excellent. I have to say, the screenshot helped a *lot* - I didn't really understand what you were talking about until I saw that!
I installed the 64 bit version, and have managed to do some simple things very intuitively. There are other tools around to handle disc images, but (I believe) nothing that has the range of features that you have implemented.
Uh yeah. It's not very fault tolerant. And it doesn't handle newer true colour sprites.
And in fairness, there are issues in the translator front-end more than anything.
Have I mentioned it's currently an Alpha release?
- geraldholdsworth
- Posts: 1406
- Joined: Tue Nov 04, 2014 9:42 pm
- Location: Inverness, Scotland
- Contact:
Re: File system manipulator thingy
I did spot this originally and thought that it was some nice healthy competition for Disc Image Manager.
I was going to try it out, but it is only for Windows...shame (I'm pretty much exclusively now on Mac). But, well done - the screen shots look good. Keep going - if you need a hand with ADFS New Map, just let me know. I wrote some documentation on it, mainly so I don't forget in years to come. Same goes for the Sprite handler (I've also written a Sprite Convertor, which deals with all sprite formats I could find) - if you need a hand with that.
Incidentally, with regards the INF files - what format are you using?
We (i.e., here on Stardot) worked out a 'standard' for them a while back, which I was involved in.
I was going to try it out, but it is only for Windows...shame (I'm pretty much exclusively now on Mac). But, well done - the screen shots look good. Keep going - if you need a hand with ADFS New Map, just let me know. I wrote some documentation on it, mainly so I don't forget in years to come. Same goes for the Sprite handler (I've also written a Sprite Convertor, which deals with all sprite formats I could find) - if you need a hand with that.
Incidentally, with regards the INF files - what format are you using?
We (i.e., here on Stardot) worked out a 'standard' for them a while back, which I was involved in.
Gerald Holdsworth, CTS-D
Extron Authorised Programmer
https://www.geraldholdsworth.co.uk
https://www.reptonresourcepage.co.uk
Twitter @radiogezza
Extron Authorised Programmer
https://www.geraldholdsworth.co.uk
https://www.reptonresourcepage.co.uk
Twitter @radiogezza
Re: File system manipulator thingy
That's largely for two reasons: 1, Windows is what I know , and C, one of its design goals is direct hardware access, which tends to be awkward to do in a platform independent way.geraldholdsworth wrote: ↑Wed Jan 18, 2023 8:14 am I was going to try it out, but it is only for Windows...shame (I'm pretty much exclusively now on Mac). But, well done - the screen shots look good. Keep going - if you need a hand with ADFS New Map, just let me know. I wrote some documentation on it, mainly so I don't forget in years to come. Same goes for the Sprite handler (I've also written a Sprite Convertor, which deals with all sprite formats I could find) - if you need a hand with that.
I'm very aware of your excellent documentation, and in fact credit is given in the About Box
Also, a cheeky comment here on the ReadBits function, as I hadn't at the time grasped how fragment ID bits were stored in the bitmap:
https://github.com/RebeccaPhu/NUTS/blob ... ap.cpp#L13
Though I've still yet to grasp how the zone cross-check bytes are generated, so information thereof would be fab.
Um... I'm not sure. I had some old INF files to hand and I derived it from that. So there's probably a newer spec somewhere.geraldholdsworth wrote: ↑Wed Jan 18, 2023 8:14 am Incidentally, with regards the INF files - what format are you using?
We (i.e., here on Stardot) worked out a 'standard' for them a while back, which I was involved in.
Re: File system manipulator thingy
That's a nice utility . Exactly what I was looking for about a year ago, when trying to find a quick and simple way to copy multiple ADFS formatted floppy images over to a ADFS formatted SCSI HDD image!
Whilst the utility does seem to support SCSI HDD images, I had to rename my image from scsi0.dat to scsi.adf in order to read the contents. The image was then recognised as an Acorn ADFS Hard Disk (Old Format). SCSI0.dat is a fairly common file name for SCSI disk images used by BeebEm, BeebSCSI & Pi1Mhz, so it would be nice if the file extension was recognised directly by your utility.
Whilst the utility does seem to support SCSI HDD images, I had to rename my image from scsi0.dat to scsi.adf in order to read the contents. The image was then recognised as an Acorn ADFS Hard Disk (Old Format). SCSI0.dat is a fairly common file name for SCSI disk images used by BeebEm, BeebSCSI & Pi1Mhz, so it would be nice if the file extension was recognised directly by your utility.
Re: File system manipulator thingy
I'll add it to the extension registry.
Note that you can right-click an image and specify the filesystem directly via "Enter As ..."
Note that you can right-click an image and specify the filesystem directly via "Enter As ..."
Re: File system manipulator thingy
I was going to ask why you use a custom licence for the work. Although you might consider it unlikely that people would share and contribute to the software - in which case, I encourage you not to undervalue your own work - having a widely recognised licence would be more appealing. I know that for Windows software some of the considerations are often different, but if we were to imagine someone porting it to the Mac or Linux, a custom licence might be an obstacle to having the software included in software distributions like Debian or Fedora, even if your licence mostly seems to give the users the same rights as a more established permissive licence.
I would recommend the REUSE approach to choosing and defining a licence, but the whole exercise can be rather onerous, at least if dealing with code from other people (which you probably aren't). But the "Which license should I choose?" section gives a starting point for just selecting a licence that is more familiar, and therefore feels "safer", to anyone wanting to see your work in more places.
Anyway, this is just a suggestion. It obviously doesn't have much bearing on how things are at the moment.
- geraldholdsworth
- Posts: 1406
- Joined: Tue Nov 04, 2014 9:42 pm
- Location: Inverness, Scotland
- Contact:
Re: File system manipulator thingy
I found that too - that's why DIM doesn't directly access any hardware, nor can it currently drag items out of the application (as each platform handles it differently). I found another way to copy from one image to another.
Ah, thank you very much.
Very amusing, and thank you.
Have a look through this. Hopefully, it's clearer - but feel free to ask me anything specific. Remember, no question is too stupid.
I've documented it on the last section of the above document.
EDIT: A quick glance through the document and I can't see about cross-zone checks...basically, (IIRC) all these check bytes XORed together have to end up as 0xFF. So, the easiest way I've found is to make them all 0x00, except for one which is 0xFF. From what I've seen of ADFS produced discs, that is how the FileCore does it too.
Gerald Holdsworth, CTS-D
Extron Authorised Programmer
https://www.geraldholdsworth.co.uk
https://www.reptonresourcepage.co.uk
Twitter @radiogezza
Extron Authorised Programmer
https://www.geraldholdsworth.co.uk
https://www.reptonresourcepage.co.uk
Twitter @radiogezza