File system manipulator thingy

discuss pc<>acorn file transfer issues and the use of other utils
Post Reply
Phu
Posts: 21
Joined: Wed Jan 27, 2021 11:00 pm
Contact:

File system manipulator thingy

Post by Phu »

Hi 🙂

So for the last 12 years (no really), I've been working on a file system manipulator program. Kind of a hobby process but also "hey, you might like using this".

Anyway. It's called NUTS (Native Universal Transfer System, and yes, it's a backronym), and the idea is to present file systems as two sides of an FTP-client-like interface (imagine a DFS image on the left, ADFS on the right, for example). But it does other things, like image (as in picture) translation, and so on.

It's also based around a plugin system so that the main program is agnostic to exactly what a "file" is, allowing all sorts of possibilities.

Anyways, obligatory github ink: https://github.com/RebeccaPhu/NUTS/releases

(Please check list of bugs, especially if venturing into ADFS (new map) territory. Take backups to be safe.)

Note: Heavily Windows based. Not interested in porting to other OSes, as it uses a lot of custom controls. But it's open source, so if anyone fancies a challenge....

(Please note - I'm not in competition with anyone. I'll update it as and when I write stuff.)
Phu
Posts: 21
Joined: Wed Jan 27, 2021 11:00 pm
Contact:

Re: File system manipulator thingy

Post by Phu »

116 views and not a single comment?

As my project manager at work says "either I've done well and nobody has any criticisms, or I've do so badly none of you get it".

No questions? No "why even bother"s? No questioning of my coding ability?

Not even a crude dig at my sexual orientation...

Nothing.

Oh well.

C'est la vie.
User avatar
davidb
Posts: 3395
Joined: Sun Nov 11, 2007 10:11 pm
Contact:

Re: File system manipulator thingy

Post by davidb »

Well, one person starred your repository on GitHub, so there's that. :)
But sometimes things take a while to take off, especially when there's lot of other stuff going on.

Have you thought about using GitHub pages to host the documentation? That might give people a starting point to get into what the project offers.
User avatar
TobyLobster
Posts: 618
Joined: Sat Aug 31, 2019 7:58 am
Contact:

Re: File system manipulator thingy

Post by TobyLobster »

I'm deducing this is a Windows GUI application to allow transferring files between different file systems, including DFS, ADFS and maybe more? But it sounds like it can do other things too? It's hard to understand the scope of what it can do without some documentation :-).
User avatar
dominicbeesley
Posts: 2210
Joined: Tue Apr 30, 2013 12:16 pm
Contact:

Re: File system manipulator thingy

Post by dominicbeesley »

It's not obvious to me what it does. Is this to move files between disk images or actual machines?
paulb
Posts: 1767
Joined: Mon Jan 20, 2014 9:02 pm
Contact:

Re: File system manipulator thingy

Post by paulb »

dominicbeesley wrote: Tue Jan 17, 2023 11:33 am It's not obvious to me what it does. Is this to move files between disk images or actual machines?
It sounds like some kind of image transfer solution, but I could be wrong about that. To be honest, I did briefly look at this, but the link led to the releases page, and so I wondered if the code was all browsable online, forgetting for the nth time that I needed to click on the code tab to see the repository. And I think I did this during the holidays (or shortly afterwards in an exhausted state) with not much time/energy to spend investigating, so there's that excuse as well!

None of this is to suggest that no-one is interested, though. Some of us pursue projects that are bizarre in comparison to this one, so please don't think we don't welcome such endeavours!
Phu
Posts: 21
Joined: Wed Jan 27, 2021 11:00 pm
Contact:

Re: File system manipulator thingy

Post by Phu »

OK, I get the idea - lacking in description. No problem.....

A little background

Back in 2011 when I first started this thing, the prevailing paradigm for moving files from one disk image to another (e.g. a DFS SSD file to an ADFS ADF file) was to open a program, load the SSD, export the file (complete with INF), then close that, open a different program, load the ADF, import the file (INF and all), and then close it.

To my mind this seemed silly: the file is the same thing to both systems, so why can't be directly transferred? I envisioned something like an FTP client. In this model, you have "local" (your computer) on the left, and "remote" (the site you're dealing with) on the right. Then you can copy files back and forth as required.

So I came up with this:
Image

(Couple of notes: yes, that's the hard drive image found elsewhere on this forum - NUTS can handle it because it doesn't load images into memory, it operates on them directly. Yes, that's fake directories in a DFS disk image. It just makes the whole thing work more intuitively that way. There's going to be a FAQ entry in the documentation explaining this....)

The Application

So here we have a DFS image on the left, and an ADFS hard disk (Risc OS) on the right. Should you so desire you can just select something on the left, hit "Copy" and off it goes. No intermediate step. (In fact, if such a file is under 4MB the "holding store" for the data in transit is in RAM).

OK, so that's the basic idea. Here's what makes it different: everything uses the "agnostic" idea of where stuff is. So for example, the ADFS (Old Map) file system handler works equally well on an ADF/ADL image, as it does directly accessing a CF card. It's the same code that runs, because the data source is separated from the file system.

At present, while there *is* support for floppy drives, it's the standard Windows API, so your mileage will be crappy. But in theory, it'll work.

The whole system is plugin-based, so you could (for example) implement a plugin that provides access to a Kryoflux device, without having to write any file system code. The existing FS handlers will work with the data source component.

So here's a run down on the "killer" (lol) feature list:

* Support for a wide variety of Acorn/Risc OS file systems, Commdore D64s, AmigaDOS, AMSDOS, +3DOS, TRD, TZX, TAP, CDT, and so on.... there's quite a few filesystems in there, some more complete than others.
* Support for "wrappers" - data sources that implement some container over the filesystem. For example, you can (if you feel inclined) put a Risc OS 800K E disk image inside a CPCEMU DSK file.
* Support for unusual types inside other containers (internally called the "Foreign Object Protocol"). For example, you can created Risc OS ISO images that respect the ! naming, along with file types.
* Icon resolving: Where native use of a file system would show custom icons under the right circumstances, NUTS will too (turned off by default because it usually involves extra processing).
* Image "Install": Copies the contents of an image to another file system in one click.
* Tape image player: Plays back files such as TZX and TAP as the audio that they represent. (UEF to come when I figure out the most intuitive way to do it).
* Translators: Read, save, tweak and print images (of the graphic kind), text (including tokenised BASIC) and audio from selected formats.
* Font system: Original characters are faithfully reproduced by using the original font associated with objects. In the screenshot above, you can see the "BBC Micro" font in use for the SSD, while "Risc OS" is in use for the ADFS hard drive. There is not much difference between them, but they do put some characters in different places, especially above ASCII 126. Notably, the £ sign is in a different place.
* Deals with INF files
* Allows image nesting: you can modify the contents of a Risc OS Sprite file while it's inside an ADF image inside a ZIP file without extracting anything (though depending on container sizes and nesting level, it may be slow to do).
* Imaging Wizard: Sector-by-sector copy of a filesystem from one source to another (again, due to the crappy Windows floppy drive support this is unlikely to be of major use until a decent floppy interface/driver plugin appears).
* Image repair tools (selected file systems)

And there's probably more.

But that's all I could manage in 12 years with some considerable breaks.

Any questions?
Phu
Posts: 21
Joined: Wed Jan 27, 2021 11:00 pm
Contact:

Re: File system manipulator thingy

Post by Phu »

dominicbeesley wrote: Tue Jan 17, 2023 11:33 am It's not obvious to me what it does. Is this to move files between disk images or actual machines?
Yes.

No really, both.

Well... sort of.

It has the plugin infrastructure for providing a means to move data to another machine. There is support for configuring a serial port for the use, or a suitable plugin could implement it some other way (I had consider a plugin that would allow browsing stairwaytohell directly, but meh), but I haven't written any such plugin yet.

Offers?
User avatar
jgharston
Posts: 5319
Joined: Thu Sep 24, 2009 12:22 pm
Location: Whitby/Sheffield
Contact:

Re: File system manipulator thingy

Post by jgharston »

Phu wrote: Tue Jan 17, 2023 8:04 pm Any questions?
Me sir, me, me!
Can you add Spectrum microdrive images? :)

I was having a look at the repository a few days ago, I'm not sure if I have a build environment that will build it, and I couldn't find any pre-built binaries. But otherwise, it's something I'd find useful, and would replace half a dozen seperate tools.

Code: Select all

$ bbcbasic
PDP11 BBC BASIC IV Version 0.45
(C) Copyright J.G.Harston 1989,2005-2024
>_
User avatar
jms2
Posts: 3765
Joined: Mon Jan 08, 2007 6:38 am
Location: Derby, UK
Contact:

Re: File system manipulator thingy

Post by jms2 »

This looks really excellent. I have to say, the screenshot helped a *lot* - I didn't really understand what you were talking about until I saw that!

I installed the 64 bit version, and have managed to do some simple things very intuitively. There are other tools around to handle disc images, but (I believe) nothing that has the range of features that you have implemented.

I did find that the sprite translator crashed though!
Phu
Posts: 21
Joined: Wed Jan 27, 2021 11:00 pm
Contact:

Re: File system manipulator thingy

Post by Phu »

jgharston wrote: Tue Jan 17, 2023 9:26 pm Me sir, me, me!
Can you add Spectrum microdrive images? :)
I imagine so. I'll put it on my list.
jgharston wrote: Tue Jan 17, 2023 9:26 pm I was having a look at the repository a few days ago, I'm not sure if I have a build environment that will build it, and I couldn't find any pre-built binaries. But otherwise, it's something I'd find useful, and would replace half a dozen seperate tools.
It's a Visual Studio 2010 build and should be self contained. It should build with a newer visual studio. It has no reliance on things like MFC (it's very pure Win32), though it will require the DirectX SDK.

The Releases page has 32-bit and 64-bit MSI installers.
Phu
Posts: 21
Joined: Wed Jan 27, 2021 11:00 pm
Contact:

Re: File system manipulator thingy

Post by Phu »

jms2 wrote: Tue Jan 17, 2023 10:18 pm This looks really excellent. I have to say, the screenshot helped a *lot* - I didn't really understand what you were talking about until I saw that!

I installed the 64 bit version, and have managed to do some simple things very intuitively. There are other tools around to handle disc images, but (I believe) nothing that has the range of features that you have implemented.
Muchos gracias.
jms2 wrote: Tue Jan 17, 2023 10:18 pm I did find that the sprite translator crashed though!
Uh yeah. It's not very fault tolerant. And it doesn't handle newer true colour sprites.

And in fairness, there are issues in the translator front-end more than anything.

Have I mentioned it's currently an Alpha release?
User avatar
geraldholdsworth
Posts: 1401
Joined: Tue Nov 04, 2014 9:42 pm
Location: Inverness, Scotland
Contact:

Re: File system manipulator thingy

Post by geraldholdsworth »

I did spot this originally and thought that it was some nice healthy competition for Disc Image Manager.

I was going to try it out, but it is only for Windows...shame (I'm pretty much exclusively now on Mac). But, well done - the screen shots look good. Keep going - if you need a hand with ADFS New Map, just let me know. I wrote some documentation on it, mainly so I don't forget in years to come. Same goes for the Sprite handler (I've also written a Sprite Convertor, which deals with all sprite formats I could find) - if you need a hand with that.

Incidentally, with regards the INF files - what format are you using?
We (i.e., here on Stardot) worked out a 'standard' for them a while back, which I was involved in.
Gerald Holdsworth, CTS-D
Extron Authorised Programmer
https://www.geraldholdsworth.co.uk
https://www.reptonresourcepage.co.uk
Twitter @radiogezza
Phu
Posts: 21
Joined: Wed Jan 27, 2021 11:00 pm
Contact:

Re: File system manipulator thingy

Post by Phu »

geraldholdsworth wrote: Wed Jan 18, 2023 8:14 am I was going to try it out, but it is only for Windows...shame (I'm pretty much exclusively now on Mac). But, well done - the screen shots look good. Keep going - if you need a hand with ADFS New Map, just let me know. I wrote some documentation on it, mainly so I don't forget in years to come. Same goes for the Sprite handler (I've also written a Sprite Convertor, which deals with all sprite formats I could find) - if you need a hand with that.
That's largely for two reasons: 1, Windows is what I know :), and C, one of its design goals is direct hardware access, which tends to be awkward to do in a platform independent way.

I'm very aware of your excellent documentation, and in fact credit is given in the About Box :)

Also, a cheeky comment here on the ReadBits function, as I hadn't at the time grasped how fragment ID bits were stored in the bitmap:

https://github.com/RebeccaPhu/NUTS/blob ... ap.cpp#L13

Though I've still yet to grasp how the zone cross-check bytes are generated, so information thereof would be fab.
geraldholdsworth wrote: Wed Jan 18, 2023 8:14 am Incidentally, with regards the INF files - what format are you using?
We (i.e., here on Stardot) worked out a 'standard' for them a while back, which I was involved in.
Um... I'm not sure. I had some old INF files to hand and I derived it from that. So there's probably a newer spec somewhere.
User avatar
KenLowe
Posts: 4675
Joined: Mon Oct 18, 2004 5:35 pm
Location: UK
Contact:

Re: File system manipulator thingy

Post by KenLowe »

That's a nice utility =D> =D> =D>. Exactly what I was looking for about a year ago, when trying to find a quick and simple way to copy multiple ADFS formatted floppy images over to a ADFS formatted SCSI HDD image!

Whilst the utility does seem to support SCSI HDD images, I had to rename my image from scsi0.dat to scsi.adf in order to read the contents. The image was then recognised as an Acorn ADFS Hard Disk (Old Format). SCSI0.dat is a fairly common file name for SCSI disk images used by BeebEm, BeebSCSI & Pi1Mhz, so it would be nice if the file extension was recognised directly by your utility.
Phu
Posts: 21
Joined: Wed Jan 27, 2021 11:00 pm
Contact:

Re: File system manipulator thingy

Post by Phu »

I'll add it to the extension registry.

Note that you can right-click an image and specify the filesystem directly via "Enter As ..."
User avatar
KenLowe
Posts: 4675
Joined: Mon Oct 18, 2004 5:35 pm
Location: UK
Contact:

Re: File system manipulator thingy

Post by KenLowe »

Phu wrote: Wed Jan 18, 2023 12:15 pm Note that you can right-click an image and specify the filesystem directly via "Enter As ..."
Ah, so you can. Thanks for pointing that out.
paulb
Posts: 1767
Joined: Mon Jan 20, 2014 9:02 pm
Contact:

Re: File system manipulator thingy

Post by paulb »

Phu wrote: Tue Jan 17, 2023 8:04 pm Any questions?
I was going to ask why you use a custom licence for the work. Although you might consider it unlikely that people would share and contribute to the software - in which case, I encourage you not to undervalue your own work - having a widely recognised licence would be more appealing. I know that for Windows software some of the considerations are often different, but if we were to imagine someone porting it to the Mac or Linux, a custom licence might be an obstacle to having the software included in software distributions like Debian or Fedora, even if your licence mostly seems to give the users the same rights as a more established permissive licence.

I would recommend the REUSE approach to choosing and defining a licence, but the whole exercise can be rather onerous, at least if dealing with code from other people (which you probably aren't). But the "Which license should I choose?" section gives a starting point for just selecting a licence that is more familiar, and therefore feels "safer", to anyone wanting to see your work in more places.

Anyway, this is just a suggestion. It obviously doesn't have much bearing on how things are at the moment.
User avatar
geraldholdsworth
Posts: 1401
Joined: Tue Nov 04, 2014 9:42 pm
Location: Inverness, Scotland
Contact:

Re: File system manipulator thingy

Post by geraldholdsworth »

Phu wrote: Wed Jan 18, 2023 10:13 am That's largely for two reasons: 1, Windows is what I know :), and C, one of its design goals is direct hardware access, which tends to be awkward to do in a platform independent way.
I found that too - that's why DIM doesn't directly access any hardware, nor can it currently drag items out of the application (as each platform handles it differently). I found another way to copy from one image to another.
Phu wrote: Wed Jan 18, 2023 10:13 am I'm very aware of your excellent documentation, and in fact credit is given in the About Box :)
Ah, thank you very much. :D
Phu wrote: Wed Jan 18, 2023 10:13 am Also, a cheeky comment here on the ReadBits function, as I hadn't at the time grasped how fragment ID bits were stored in the bitmap:
Very amusing, and thank you.
Phu wrote: Wed Jan 18, 2023 10:13 am Though I've still yet to grasp how the zone cross-check bytes are generated, so information thereof would be fab.
Have a look through this. Hopefully, it's clearer - but feel free to ask me anything specific. Remember, no question is too stupid.
Phu wrote: Wed Jan 18, 2023 10:13 amUm... I'm not sure. I had some old INF files to hand and I derived it from that. So there's probably a newer spec somewhere.
I've documented it on the last section of the above document.

EDIT: A quick glance through the document and I can't see about cross-zone checks...basically, (IIRC) all these check bytes XORed together have to end up as 0xFF. So, the easiest way I've found is to make them all 0x00, except for one which is 0xFF. From what I've seen of ADFS produced discs, that is how the FileCore does it too.
Gerald Holdsworth, CTS-D
Extron Authorised Programmer
https://www.geraldholdsworth.co.uk
https://www.reptonresourcepage.co.uk
Twitter @radiogezza
Post Reply

Return to “software & utilities for the pc, mac or unix”