Another Windows vs Linux file system oddity

[Deleted User 9295]

Having previously experienced this issue, which turned out to be due to some 'fine print' affecting the order in which fread and fwrite may legitimately be called, I'm now experiencing another file system problem which I can't explain. The relevant code is this:

Code: Select all

ptr = _ftelli64(context);
fprintf(stderr, "ftell  ptr = %li\n", ptr);
amount = fwrite(buffer, 1, 256, context);
fprintf(stderr, "fwrite amount = %i\n", amount);
ptr = _ftelli64(context);
fprintf(stderr, "ftell  ptr = %li\n", ptr);

In Linux (substituting ftell for _ftelli64) this works as expected, but in (64-bit) Windows 10 it is outputting:

Code: Select all

ftell  ptr = 0
fwrite amount = 256
ftell  ptr = 257

In other words writing 256 bytes is seemingly causing the file pointer to advance by 257! I initially thought this must be as a result of opening the file in a non-binary mode (especially as it seems to be data-dependent), but the code is this:

Code: Select all

context = fopen(path, "w+b");

Any thoughts as to what might be happening? Can you suggest any additional diagnostics?

[tricky]

Does it include the end of file marker?

[Deleted User 9295]

Does it include the end of file marker?

Is that a byte with value 0x1A? If so, no. Here's a dump of the buffer contents:

Code: Select all

00000000  0B 01 00 2E 6C 61 62 65 6C 31 0D 15 02 00 64 62
00000010  20 22 48 65 6C 6C 6F 20 77 6F 72 6C 64 21 22 0D
00000020  0D 03 00 61 64 63 20 61 6C 2C 36 38 0D 10 04 00
00000030  61 64 63 20 61 78 2C 32 39 33 30 34 0D 12 05 00
00000040  61 64 63 20 65 61 78 2C 37 32 31 30 37 36 0D 12
00000050  06 00 61 64 63 20 72 61 78 2C 2D 32 34 33 35 34
00000060  0D 0F 07 00 61 64 63 20 72 38 62 2C 64 69 6C 0D
00000070  0D 08 00 61 64 63 20 61 78 2C 64 69 0D 11 09 00
00000080  61 64 63 20 72 31 33 64 2C 72 31 34 64 0D 0F 0A
00000090  00 61 64 63 20 72 61 78 2C 72 31 35 0D 1C 0B 00
000000A0  61 64 63 20 72 31 35 62 2C 5B 72 31 35 2B 72 31
000000B0  35 2B 32 36 34 37 32 5D 0D 14 0C 00 61 64 63 20
000000C0  72 31 32 77 2C 5B 72 31 32 2A 39 5D 0D 15 0D 00
000000D0  61 64 63 20 65 64 78 2C 5B 72 63 78 2D 31 36 32
000000E0  5D 0D 11 0E 00 61 64 63 20 72 31 32 2C 5B 72 64
000000F0  69 5D 0D 0F 0F 00 61 64 63 20 72 31 34 62 2C 35

[Coeus]

Code: Select all

ftell  ptr = 0
fwrite amount = 256
ftell  ptr = 257

Have you opened the file in "text" mode, i.e. fopen mode "r" or "w"? If so the extra byte is probably a CR, i.e. if the buffer you handed to fwrite contained a newline (LF, '\n') that would have been expanded.

Contrary to popular belief fread and fwrite are not exclusively for binary data, just for anything where you know how many bytes you are reading/writing.

If this is the issue then just add "b" to the fopen mode, i.e. "rb" or "wb".

[Coeus]

P.S. the issue of text vs. binary mode does not apply to Linux or the other Unixes. As they use '\n' natively as their EOL marker there is no translation.

[cmorley]

Your printf format specifiers look wrong. You probably need lld (or lli) for a 64 bit integer depending on the platform. You should really use %p for pointers and intptr_t or uintptr_t types or your code will break when you change systems.

However this is probably separate from your problem.

[Deleted User 9295]

Have you opened the file in "text" mode, i.e. fopen mode "r" or "w"?

See my original post:

I initially thought this must be as a result of opening the file in a non-binary mode (especially as it seems to be data-dependent), but the code is this:

Code: Select all

context = fopen(path, "w+b");

the extra byte is probably a CR, i.e. if the buffer you handed to fwrite contained a newline (LF, '\n') that would have been expanded.

No, the extra byte is not a CR, it's a NUL. What gets written to the file is this:

Code: Select all

00000000  0B 01 00 2E 6C 61 62 65 6C 31 0D 15 02 00 64 62
00000010  20 22 48 65 6C 6C 6F 20 77 6F 72 6C 64 21 22 0D
00000020  0D 03 00 61 64 63 20 61 6C 2C 36 38 0D 10 04 00
00000030  61 64 63 20 61 78 2C 32 39 33 30 34 0D 12 05 00
00000040  61 64 63 20 65 61 78 2C 37 32 31 30 37 36 0D 12
00000050  06 00 61 64 63 20 72 61 78 2C 2D 32 34 33 35 34
00000060  0D 0F 07 00 61 64 63 20 72 38 62 2C 64 69 6C 0D
00000070  0D 08 00 61 64 63 20 61 78 2C 64 69 0D 11 09 00
00000080  61 64 63 20 72 31 33 64 2C 72 31 34 64 0D 0F 0A
00000090  00 61 64 63 20 72 61 78 2C 72 31 35 0D 1C 0B 00
000000A0  61 64 63 20 72 31 35 62 2C 5B 72 31 35 2B 72 31
000000B0  35 2B 32 36 34 37 32 5D 0D 14 0C 00 61 64 63 20
000000C0  72 31 32 77 2C 5B 72 31 32 2A 39 5D 0D 15 0D 00
000000D0  61 64 63 20 65 64 78 2C 5B 72 63 78 2D 31 36 32
000000E0  5D 0D 11 0E 00 61 64 63 20 72 31 32 2C 5B 72 64
000000F0  69 5D 0D 0F 0F 00 61 64 63 20 72 31 34 62 2C 35
00000100  00

[Deleted User 9295]

Your printf format specifiers look wrong. You probably need lld (or lli) for a 64 bit integer depending on the platform.

No, I am compiling for Windows; %I64i is correct. If I change to %lli I get the following warnings (and the issue isn't affected, of course):

Code: Select all

warning: unknown conversion type character 'l' in format [-Wformat=]
 fprintf(stderr, "ftell  ptr = %lli\n", ptr);
                                 ^
warning: too many arguments for format [-Wformat-extra-args]
 fprintf(stderr, "ftell  ptr = %lli\n", ptr);
 

[cmorley]

Microsoft-specific:
The I (uppercase i), I32, I64, and w argument size modifier prefixes are Microsoft extensions and are not ISO C-compatible.

I64 is not portable.

lld is listed in the Microsoft format specification:

Code: Select all

long long int
unsigned long long int 	ll (lowercase LL) 	d, i, o, u, x, or X

On 64-bit systems, an int is a 32-bit value; so, 64-bit integers will be truncated when they're formatted for output unless a size prefix of ll or I64 is used.

Why it doesn't work is beyond my ability to devine since you didn't mention which compiler or C language version you are using.

https://docs.microsoft.com/en-us/cpp/cp ... w=msvc-160

Code: Select all

long long 	8 	none (but equivalent to __int64) 	-9,223,372,036,854,775,808 to 9,223,372,036,854,775,807

long long is 64 bits on Windows so lld should be the correct format specifier to use with a modern MS C compiler.

[Deleted User 9295]

I64 is not portable.

I know. Tell Microsoft, not me! And anyway what's the relevance of this to the issue under discussion?

lld is listed in the Microsoft format specification

It is now, but that's quite recent, and as I showed I get a warning if I try to use it.

On 64-bit systems, an int is a 32-bit value; so, 64-bit integers will be truncated when they're formatted for output unless a size prefix of ll or I64 is used.

I may not be an expert C programmer but I'm not so ignorant that I'm unaware of such basic things. Truncation, even if it was happening (which is isn't) has no bearing because the values being printed are all small.

Why it doesn't work is beyond my ability to devine since you didn't mention which compiler or C language version you are using.

Since the functions concerned (fopen, ftell, fwrite etc.) are ultimately being executed by the Windows C run-time DLL (msvcrt.dll or similar) I hadn't anticipated that the compiler was relevant. But for the record:

Code: Select all

gcc (x86_64-win32-seh-rev0, Built by MinGW-W64 project) 8.1.0
Copyright (C) 2018 Free Software Foundation, Inc.

[cmorley]

I know. Tell Microsoft, not me! And anyway what's the relevance of this to the issue under discussion?

In as much as you mention linux and Windows in the same post and the difference in behaviour of the same code. Portable code seemed to be at the heart of your original post.

Now I know you are using mingw gcc, this stack overflow post answers the warning message you receive:
https://stackoverflow.com/questions/237 ... -long-long

It seems defining __USE_MINGW_ANSI_STDIO would restore ANSI C compatibility in this instance.

[Deleted User 9295]

Now I know you are using mingw gcc, this stack overflow post answers the warning message you receive

I only receive a warning message when I make the (unnecessary) change you suggested! There is no benefit in that code being portable, since it's been added to diagnose a problem which occurs only in Windows.

Have you any suggestions for why writing 256 bytes to a file causes the file pointer to advance by 257 bytes, even though it's been opened for binary access (and anyway the extra byte isn't a CR)? It does seem that it is the presence of an LF (0x0A) in the buffer which provokes this behaviour, since changing just that one byte fixes the fault. But no CR is added before the LF, as would be the case in text mode; rather the pointer is advanced as if it had!

[Coeus]

Since the functions concerned (fopen, ftell, fwrite etc.) are ultimately being executed by the Windows C run-time DLL (msvcrt.dll or similar) I hadn't anticipated that the compiler was relevant. But for the record:

I don't think it is relevant to the problem you originally posted about, but it is relevant to the issue of the printf format specifiers because it is a case of the compiler generating a warning about compatibility between your code and the behaviour of a library function so there has to be a match between the compiler and the library.

And Richard, I assume you know this already, but for anyone else reading another part of the puzzle for msvcrt is that Microsoft no longer sell a C compiler. Their take is that their compiler is a C++ compiler and therefore they don't need to bother about the ISO C standard, only the C++ one. For a long time C was a subset of C++ so compiling C tended to work but some recent standardisation has added features that are not in C++.

But, back to your original questions, the ISO C standard makes no guarantee that the value returned by ftell() is a number of bytes from the beginning of the file, only that if you use it as the value of the offset in fseek() with whence set to SEEK_SET, it will return you to the same place in the file.

What happens after the code you posted, i.e. between there and fclose() of the file 'context'? Does removing the all to _ftelli64 change anything?

[cmorley]

I would suggest a sanity check and use the Visual Studio C compiler for Windows (it is free). Then see if the behaviour is a mingw thing or a Windows thing.

If you post a complete minimal source file I can compile it as a x64 console app with VS2019 if you don't have VS installed.
edit: caveat as coeus points out it would be the C++ compiler

[Deleted User 9295]

I don't think it is relevant to the problem you originally posted about, but it is relevant to the issue of the printf format specifiers

Of course, but why this desire for forensically analysing the code I added purely for diagnostic purposes? The problem occurs without that code even being present.

the ISO C standard makes no guarantee that the value returned by ftell() is a number of bytes from the beginning of the file, only that if you use it as the value of the offset in fseek() with whence set to SEEK_SET, it will return you to the same place in the file.

Fair enough, but I'm only checking by how much the value returned by ftell() changes as the result of an fwrite(). Surely it must be the case that writing 256 bytes to the file should cause the value returned by ftell() to increase by 256?

What happens after the code you posted, i.e. between there and fclose() of the file 'context'?

A few more bytes are written to the file, using another fwrite(). But those bytes appear in the file one byte later than they should, consistent with the file pointer having been advanced too much. I can workaround the problem using this horrible kludge:

Code: Select all

ptr = _ftelli64(context);
amount = fwrite(buffer, 1, 256, context);
_fseeki64(context, ptr + amount, SEEK_SET);

That results in correct operation, the contents of the resulting file are what they should be.

[baz4096]

I've tried a stripped down version of your fwrite statements in VS2019, mingw32, & 64 (both i686 & x86_64 - this one identical to yours, but using default setup) and I always see the same 0/256/256 values.

Are there any other file operations between the fwrite calls you mentioned above?

Does ferror indicate any errors have been set for the file stream?

[Simon]

I also see 0/256/256 testing with your exact version of mingw64's gcc.

This was all tested on windows 10 20H2 19042.985

I used the buffer you specified

What build of windows are you testing on Richard?

Code: Select all

 { 0x0B, 0x01, 0x00, 0x2E, 0x6C, 0x61, 0x62, 0x65, 0x6C, 0x31, 0x0D, 0x15, 0x02, 0x00, 0x64, 0x62, 0x20, 0x22, 0x48, 0x65, 0x6C, 0x6C, 0x6F, 0x20, 0x77, 0x6F, 0x72, 0x6C, 0x64, 0x21, 0x22, 0x0D, 0x0D, 0x03, 0x00, 0x61, 0x64, 0x63, 0x20, 0x61, 0x6C, 0x2C, 0x36, 0x38, 0x0D, 0x10, 0x04, 0x00, 0x61, 0x64, 0x63, 0x20, 0x61, 0x78, 0x2C, 0x32, 0x39, 0x33, 0x30, 0x34, 0x0D, 0x12, 0x05, 0x00, 0x61, 0x64, 0x63, 0x20, 0x65, 0x61, 0x78, 0x2C, 0x37, 0x32, 0x31, 0x30, 0x37, 0x36, 0x0D, 0x12, 0x06, 0x00, 0x61, 0x64, 0x63, 0x20, 0x72, 0x61, 0x78, 0x2C, 0x2D, 0x32, 0x34, 0x33, 0x35, 0x34, 0x0D, 0x0F, 0x07, 0x00, 0x61, 0x64, 0x63, 0x20, 0x72, 0x38, 0x62, 0x2C, 0x64, 0x69, 0x6C, 0x0D, 0x0D, 0x08, 0x00, 0x61, 0x64, 0x63, 0x20, 0x61, 0x78, 0x2C, 0x64, 0x69, 0x0D, 0x11, 0x09, 0x00, 0x61, 0x64, 0x63, 0x20, 0x72, 0x31, 0x33, 0x64, 0x2C, 0x72, 0x31, 0x34, 0x64, 0x0D, 0x0F, 0x0A, 0x00, 0x61, 0x64, 0x63, 0x20, 0x72, 0x61, 0x78, 0x2C, 0x72, 0x31, 0x35, 0x0D, 0x1C, 0x0B, 0x00, 0x61, 0x64, 0x63, 0x20, 0x72, 0x31, 0x35, 0x62, 0x2C, 0x5B, 0x72, 0x31, 0x35, 0x2B, 0x72, 0x31, 0x35, 0x2B, 0x32, 0x36, 0x34, 0x37, 0x32, 0x5D, 0x0D, 0x14, 0x0C, 0x00, 0x61, 0x64, 0x63, 0x20, 0x72, 0x31, 0x32, 0x77, 0x2C, 0x5B, 0x72, 0x31, 0x32, 0x2A, 0x39, 0x5D, 0x0D, 0x15, 0x0D, 0x00, 0x61, 0x64, 0x63, 0x20, 0x65, 0x64, 0x78, 0x2C, 0x5B, 0x72, 0x63, 0x78, 0x2D, 0x31, 0x36, 0x32, 0x5D, 0x0D, 0x11, 0x0E, 0x00, 0x61, 0x64, 0x63, 0x20, 0x72, 0x31, 0x32, 0x2C, 0x5B, 0x72, 0x64, 0x69, 0x5D, 0x0D, 0x0F, 0x0F, 0x00, 0x61, 0x64, 0x63, 0x20, 0x72, 0x31, 0x34, 0x62, 0x2C, 0x35 };

[Deleted User 9295]

I've tried a stripped down version of your fwrite statements in VS2019, mingw32, & 64 (both i686 & x86_64 - this one identical to yours, but using default setup) and I always see the same 0/256/256 values.

You won't be able to reproduce the problem with a simple test of that sort. Like all the best bugs it manifests only in very specific circumstances; if that wasn't the case I would have spotted it long ago! It's only by chance that a BASIC program I was working on today happened to set up just the right state to trigger it. I could attempt to write a standalone C program that reproduces the same conditions that the BASIC program does, but I suspect it wouldn't be easy.

Does ferror indicate any errors have been set for the file stream?

No, ferror() returns zero.

[Deleted User 9295]

I could attempt to write a standalone C program that reproduces the same conditions that the BASIC program does, but I suspect it wouldn't be easy.

To my surprise it was remarkably easy. The attached program exhibits the problem here, compiled with this makefile:

Code: Select all

test.exe: test.c
	gcc -Wall test.c -o test.exe -lmingw32

[cmorley]

I tried your code on VS2019 compiler as x86 and x64 builds.

Code: Select all

ftell  ptr = 0
fwrite amount = 256
ftell  ptr = 256

The test.bin is 256 bytes exactly.

There is one problem with your code but it doesn't affect the outcome. fwrite return type is size_t and amount is an int. The compiler throws a warning in x64 mode because the truncation wasn't explicitly cast.

Your problem must be in your mingw. I googled mingw _ftelli64 and found a host of bug reports and tickets. Skim reading _ftelli64 is emulated in mingw.

for example

2018-11-26 03:15 Updated by: keith
Comment

I committed 09fb4c4, which should suffice to provide robust implementations of _fseeki64() and _ftelli64() for all WinNT derived platforms. They will also work on Win9x, but I'm leaving the ticket open for the time being, because, as I suspected, they will interact badly with fwrite() on these platforms, if the fwrite() is preceded by an _fseeki64() to an offset beyond the physical end of file.

I skim read other things that say fwrite was fixed but maybe you have a broken version or they didn't fix it properly.

Anyway you don't have a C problem here IMO.

You could try the fcntl functions (open etc) in mingw and see if they work...

[baz4096]

Using your source Richard, on mingw-w64 I get the unexpected result. 0/256/257.

Removing one of those dummy file pointers causes correct behaviour, as does using ftell instead of _ftelli64.
Mixing both ftell and _ftelli64 gives conflicting results:

ftell ptr = 0
_ftelli64 ptr = 0

fwrite amount = 256

ftell ptr = 256
_ftelli64 ptr = 257

This has to be a bug within _ftelli64.

[Deleted User 9295]

The compiler throws a warning in x64 mode because the truncation wasn't explicitly cast.

Your compiler may do, mine reports a warning if I specify size_t but no warning if I specify int!

This is peripheral to the main discussion, but in any event I cannot use a Microsoft compiler because BBC BASIC needs support for the 'long double' (80-bit precision float) data type which gcc and Intel compilers provide, but Microsoft compilers don't.

[cmorley]

Your compiler may do, mine reports a warning if I specify size_t but no warning if I specify int!

Standard library. Not "my compiler".
https://en.cppreference.com/w/c/io/fwrite

[Deleted User 9295]

Standard library. Not "my compiler".

Whatever. I have been seeking practical help with a problem, not a lecture.

[cmorley]

You post in a public forum about C. Information on correct use might be of interest to public readers even if you are not concerned.

size_t is common gotcha and therefore worthy of comment.

[Deleted User 9295]

You post in a public forum about C.

It wasn't my intention to post "about C". That happened to be the language I was using, but the issue was always one of library functions which could have been called from any language. I don't believe any feature of the C language could have been responsible for the observed symptoms.

[cmorley]

Did you try the fcntl (POSIX) file functions to see if they also have a problem with your libraries?

If they work then at least you have a work around for fopen, ftell, fwrite with mingw/mingw-w64.

[Deleted User 9295]

Did you try the fcntl (POSIX) file functions to see if they also have a problem with your libraries?

I haven't tried them, no. There was a reason I used the stream functions originally, but for the life of me I cannot remember why now (could it have been to ensure compatibility with non-file devices like serial ports and the terminal?).

Anyway, given that the existing code works perfectly on the other supported platforms (MacOS, Linux) I don't want to make such a drastic change and potentially risk introducing new issues. I'll incorporate the workaround I listed (enabled in Windows only); it seems to be completely effective.

[Simon]

This looks like a bug in the mingw crt.

there's a few posts about it on their mailing list but it's all pretty cryptic.

upgrading to the newest build it works fine, but of course you may have issues introduced by changes in gcc between 8 and 10

[Deleted User 9295]

of course you may have issues introduced by changes in gcc between 8 and 10

Indeed I do (and it's up to version 11 now)!