I want extern "fil-c"
70 points by fzakaria
70 points by fzakaria
I love this idea.
I think it’s possible to create a Rust-Fil-C integration where everything that’s currently unsafe in Rust is governed by Fil-ABI.
It might require dialect-level changes on both the Rust and Fil-C side.
I’ve posted about it on X. I should sit down and write more about it at some point. I would be especially happy to brain dump what I understand about this to anyone who is interested in dedicating time to prototyping this.
I've seen contradictings claims about the possibility of a hybrid fil/native program, so I'll believe it when I see it. FWIW, Zig seems to be going for whole-program fil.
If Rust manages a hybrid mode, it'll be at an FFI boundary (like the article suggests), not for "everything that's currently unsafe". Because unsafe Rust has a smaller granularity than what fil can work on, and the errors that fil can detect are likely to happen outside the unsafe block that caused them. At that stage, you're basically looking at whole-program fil, which doesn't make much sense for Rust.
I think you're right about unsafe granularity, but I want to add that I think whole-program fil for Rust does make sense as an idea. It could serve as something like compiled/fast Miri.
Just now I happened to see that rustc_codegen_jvm recently added raw pointer support, and apparently it does a pretty good job at catching UB. Might be a good alternative, not sure how the perf is compared to a hypothetical Fil-Rust.
Can you think of a Rust bug that Miri doesn't catch but Fil would ? I'm not saying it's impossible, but I haven't seen any example yet, and my guess is that it would be academic but not realistic. If miri finds more bugs, that's what people will test with, even if it's slower.
The only reason for whole-prog Fil would then be that you want Fil for the C parts, but FFI-Fil doesn't work or is too cumbersome for some reason ?
Can you think of a Rust bug that Miri doesn't catch but Fil would ?
Specifically Rust bugs, probably not, since Miri is intended to catch all Rust language UB. That being said, the lack of FFI support is a notable current limitation in Miri with practical implications that a hypothetical Fil-Rust could help with, though to be fair there is work being done to Miri to better support FFI.
If miri finds more bugs, that's what people will test with, even if it's slower.
I think it's going to depend on what tests you have in mind. Fuzzing, for instance, is going to be quite a bit more sensitive to slowdowns.
That being said, the lack of FFI support is a notable current limitation in Miri with practical implications that a hypothetical Fil-Rust could help with
Sure, but we're back to "Fil is interesting at the FFI boundary", not for the whole rust-and-c program.
Fuzzing, for instance, is going to be quite a bit more sensitive to slowdowns.
Fair point.
Sure, but we're back to "Fil is interesting at the FFI boundary", not for the whole rust-and-c program.
Hrm, fair. Maybe there could be some interesting cross-language optimizations that could be done? Fil-C is mostly an LLVM pass so I could imagine it working after both Rust and C sources have been lowered to LLVM IR.
I'd happy to invite you to https://oceansprint.org/ in 2027 that I organize for geeks to socialize for a week and someone from Rust community that wants to pair up.
This sounds great and oceansprint is soon; but I am also selfish and would want us to hack on it for https://tacosprint.org/ next summer lol
I don't know how to move beyond this proposed minimal ABI: Fil-C needs to recompile the entire program in order to track capabilities (which is why it doesn't work on e.g. Windows), so you can't really pass pointers back and forth without having Rust run under Fil-C.
Memory-safety violations panic instead of becoming exploits.
Well, some of them. Fil-C turns undefined behavior into panics or legal behavior (e.g. uninitialized objects are zero-initialized instead). But Fil-C is a full C implementation, and so allows all the memory safety violations that are legal in C - including out of bounds & use-after frees, as long as they happen entirely within one object.
There are also the cases like unsynchronized accesses to the same object from different threads, where Fil-C just says "Yeah this isn't UB anymore" but doesn't actually fix the memory safety issue.
In Fil-C calling free() is optional as it uses a GC to actually reclaim unreachable memory, but if you do call free() then it updates the capability so that use-after-free and double-free are guaranteed to be caught.
Yes, but if you e.g. have a bump allocator and free its contents by resetting the bump pointer to the start so you can use it for a new set of allocations, Fil-C will still let you read from and write to the old allocations long after they've been overwritten because in C that's perfectly legal.
That's why it's so appealing because of https://github.com/mbrock/filnix
Not sure how this related to my comment?
Sorry, I wasn't clear enough.
Without pointers, there's no modifications need to Rust.
While for pointers you'd need fil-c integrated into rust and using the Nix machinery we can then distribute that.
But Fil-C is a full C implementation, and so allows all the memory safety violations that are legal in C - including out of bounds & use-after frees, as long as they happen entirely within one object.
I'm almost certain that what you wrote here was not what you intended to say. Out of bounds accesses are not legal in C whether they are constrained to a single object or not. Use-after-frees are never legal. Both yield undefined behaviour.
I guess from your followup comment below that by "use-after-free" you actually mean reusing storage for a new object (without calling free at all), which is legal, yes, but it's not what is normally meant by use-after-free. Out-of-bounds access is always illegal.
What I mean is that if, for example, a function takes a value by reference, which can look like a normal object being passed in Java/Python/JavaScript/… (languages where references point at objects) or with an explicit reference in Go/Rust/Vale/… (languages where references point at a location). In these languages, the function can only access in bounds of that location.
But in C, you can legally access past the bounds of the specific pointed-to subobject, and this is very common to do (of course if you end up violating type safety along the way it can be UB, but Fil-C doesn't catch that either).
A simple example of this posted by tstack in earlier discussions:
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
struct User {
char name[8];
int is_root;
};
int main(int argc, char*argv[]) {
struct User* user = malloc(sizeof(struct User));
strcpy(user->name, argv[1]);
if (user->is_root) {
printf("User is root!\n");
}
return 0;
}
Compile with filcc, then run it:
> ./fillo AzureDiamond
User is root!
As I've mentioned previously, that is not what Fil-C is trying to address, different hardening technologies address it - specifically FORTIFY_SOURCE. i.e. if that program is compiled under gcc, with FORTIFY_SOURCE, the access is trapped.
The only issue at the moment for Fil-C is that it is based on clang, and the clang implementation of FORTIFY_SOURCE can not yet handle that case; but that can be addressed.
See these two posts:
https://lobste.rs/s/x7jtkt/memory_safety_absolutists#c_v4tfxu
https://lobste.rs/s/x7jtkt/memory_safety_absolutists#c_0xb5sy
But in C, you can legally access past the bounds of the specific pointed-to subobject
No, you cannot legally do that in the manner you seem to be describing. It is undefined behaviour to do so. In C99 (the reference I happen to have on hand), chapter 7.21.1 (String function conventions): in all cases a char * or void * argument points to the initial (lowest addressed) character of the array. If an array is accessed beyond the end of an object, the behavior is undefined.
Compile with filcc, then run it
... and you will see one possible behaviour; this proves nothing about whether it is legal in C, in general, because UB is involved. (It happens that Fil-C tightens the semantics).
In any case "out of bounds access" refers to many other situations other than using string or memory copying functions. It is certainly not legal to attempt access to an array element by an index that is out of bounds of the array, even if the element would (if it existed) correspond to another field in the same containing object. In general, out of bounds access is not legal, and if the example you gave were legal, it would be a special case.
No, you cannot legally do that in the manner you seem to be describing.
There’s an extra step in the logic.
If you are given a pointer to a member of a struct, it’s legal to use something like Linux’s container_of() macro to access other members in the same struct. Therefore Fil-C does not try to catch pointer shenanigans within an allocation; it only traps when the program strays outside an allocation, not necessarily when it accesses an array out of bounds.
The program above is an example of UB that Fil-C fails to catch, not an example of the kind of valid accesses that are technically out-of-bounds. The latter would be something like,
int user_is_root(char *name) {
User *user = container_of(name, User, name);
// access is out of bounds of name
return user->is_root;
}
There’s an extra step in the logic. ... If you are given a pointer to a member of a struct, it’s legal to use something like Linux’s container_of() macro to access other members in the same struct
Doing such would not be considered an "out of bounds" access, and was not what the example that I commented on was doing. As I said:
No, you cannot legally do that in the manner you seem to be describing.
// access is out of bounds of name
While the access is outside the bounds of name, it wouldn't constitute what would be described as an out-of-bounds access; that term normally refers to the logic error.
Right, I was clarifying that what mira intended to describe is not what you thought they described: they were talking about out-of-bounds accesses in a general multi-language sense, not the C rules-lawyering sense you are using; you are still misinterpreting their program as supposedly an example of the kind of out-of-bounds access that C allows, but as I explained, that is not the point the program was illustrating.
It's a mix of both – I was indeed misremembering some parts of the C standard (in particular the rule found as 6.5.7.9 in the latest C23 working draft), and Fil-C's marketing helped me make my mistake. Thank you both for clearing this up ^^
I'm curious about why container_of is valid - or maybe it isn't according to the standard, but gcc defines behavior for it?
You are most welcome. I haven't looked at the specific container_of implementation that Linux uses, but the best-sanctioned variation that I know is to cast the pointer-to-member into an integer and subtract the offset of the member before casting back to the pointer-to-container type. Because you're not doing pointer arithmetic directly you avoid the UB that you normally get from pointer arithmetic which moves outside the bounds of the pointed-to (sub)object.
Technically the result is implementation-defined (casting from integer to pointer and vice versa is implementation-defined) but in practice on typical architectures/compilers it works as it's intended to.
Often casting to (char *) instead of an integer also works because compilers tend to relax the rules for char *. But, it's more likely to be UB (some might say it is UB, I think the language standard isn't completely clear on this).
The original quote was:
But Fil-C is a full C implementation, and so allows all the memory safety violations that are legal in C - including out of bounds & use-after frees, as long as they happen entirely within one object
What you are talking about is not a memory safety violation even if it is an out-of-bounds access "in some sense". If mira meant what you are saying she meant, then she nevertheless had it wrong when she said "memory safety violations are legal in C"; additionally, the code example she gave did not illustrate what she thought it did.
you are still misinterpreting their program as supposedly an example of the kind of out-of-bounds access that C allows, but as I explained, that is not the point the program was illustrating.
From that post:
But in C, you can legally access past the bounds of the specific pointed-to subobject [...] A simple example of this posted by tstack in earlier discussions:
This was the context that example was presented in, that I responded to. It does quite clearly claim that the example is of the kind of out-of-bounds access that C allows.
What you are talking about is not a memory safety violation even if it is an out-of-bounds access "in some sense".
But it is a memory safety violation. Consider a safe implementation that keeps tight bounds on the name array that is passed to user_is_root(): the function would crash at runtime with an out-of-bounds access. Obviously (which is the point of the discussion) this implementation would be too strict in practice, hence Fil-C does bounds checking on allocations not bounds checking on arrays. And the point of the example in mira’s post is that this looseness matters, because a C runtime can’t tell the difference between “good” out of bounds and “bad” out of bounds at this granularity.
And it’s subtle whether that strcpy() is UB or not, since strcpy() accesses its destination as bytes, which is legal for any object. So these should all be equivalent:
strcpy(user->name, argv[1]);
// inline strcpy
memcpy(user->name, argv[1], strlen(argv[1]) + 1);
// (void*) struct is same as (void*) first member
memcpy(user, argv[1], strlen(argv[1]) + 1);
But it is a memory safety violation.
I disagree.
Consider a safe implementation that keeps tight bounds on the name array that is passed to user_is_root(): the function would crash at runtime with an out-of-bounds access
Obviously (which is the point of the discussion) this implementation would be too strict in practice, hence Fil-C does bounds checking on allocations not bounds checking on arrays
Too strict in practice, yes - because it prevents valid accesses. (But, I suspect checking allocation bounds rather than individual subobject bounds is also better for performance).
Consider an implementation that doesn't keep such bounds, and which defines reasonable conversion from integer to pointer as required for a reasonable implementation of container_of. No undefined behaviour is needed for container_of to function correctly, and it's not accessing memory outside of what it is supposed to; there's no safety violation.
And it’s subtle whether that strcpy() is UB or not, since strcpy() accesses its destination as bytes, which is legal for any object
The arguments are required to be char arrays, and the text states that "If an array is accessed beyond the end of an object, the behavior is undefined". The example clearly does so. I think you're confusing the strict aliasing rule (which does have special allowances for access via char) with the requirements on the string functions.
The arguments are required to be char arrays, and the text states that "If an array is accessed beyond the end of an object, the behavior is undefined".
It doesn’t say that they have to be declared as char arrays, they can be “other objects treated as arrays of character type”. There is also wording earlier in the standard that allows any object to be accessed as a character array, but is a bit roundabout: 6.3.3.3 Conversions - Pointers says:
When a pointer to an object is converted to a pointer to a character type, the result points to the lowest addressed byte of the object. Successive increments of the result, up to the size of the object, yield pointers to the remaining bytes of the object.
But incrementing a pointer is only defined for arrays (6.5.7 Additive operators) so putting it together we have to conclude that an expression designating a character array might refer to any type of object or part thereof. Which also implies that the bounds of the array are indeterminate, because it isn’t clear which subobject the pointer is supposed to refer to.
(Perhaps this looseness only applies to structures and their first members (6.7.3.2 “A pointer to a structure object, suitably converted, points to its initial member (…), and vice versa.”) but that is sufficient for our example struct User. However, if you only permit incrementing through the remaining bytes of the object then container_of() is disallowed.)
Hence the <string.h> preamble is vague about which object’s end is the end that matters, and implies that it isn’t necessarily the end of the char array: “the end of an object” could just as well be the end of the enclosing struct User.
The upshot is that C doesn’t allow precise bounds checking on strings embedded in structs, and a buffer overrun that corrupts fields following the string is not UB (tho it might lead to UB when those fields are subsequently used). And container_of() relies on the same kind of permitted but unsafe operation (using pointer arithmetic to access outside the bounds of a subobject) that leads to buffer overrun security vulnerabilities.
Tangentially, it’s maybe worth noting that the meaning of the [ ] operator is changing in the next C standard so that it accesses arrays directly, without first converting them to pointers. So user->name[9] will be unambiguously out of bounds, whereas at the moment it’s arguably permitted. (But in practice compilers are likely to complain, because features like FORTIFY are stricter than the standard requires. So even in current C name[9] probably needs the same kind of casting to hell and back that would make it valid in future C.)
No, you cannot legally do that in the manner you seem to be describing. It is undefined behaviour to do so. In C99 (the reference I happen to have on hand) [...]
It doesn't matter whether some spec/reference calls it illegal/undefined/etc, what matters is that the FilC compiler/runtime lets it happen.
It's a clear OOB access in most languages, but FilC doesn't catch it. I wouldn't mind that hole (it has a valid technical reason) if FilC didn't gutsily claim "total memory safety".
It doesn't matter whether some spec/reference calls it illegal/undefined/etc, what matters is that the FilC compiler/runtime lets it happen.
The FilC compiler/runtime may well let it happen. But if there's a claim made that "the language allows this" (which there was) but it doesn't (which is the case), I'd say that matters.
I wouldn't mind that hole (it has a valid technical reason) if FilC didn't gutsily claim "total memory safety"
It remains a problem that there's no consensus agreement on what "memory safety" actually means. I agree that adding "total" when this hole is present is a little presumptuous.
Yes, the language allows this. It doesn't matter what the spec says about this, because the spec also says that it's up to compilers to decide what to do here, and they all decide to allow this. The C spec is intentionally weak : when it calls something illegal/undefined/etc, that has as much effect as me not allowing swear words for my kids.
Would be good to hear the venerable pizlonator's thoughts on this. Sadly he often resorts to jabs at Rust for marketing Fil-C, but actually a mixed system could make a lot of sense from a performance & safety POVs; the q is whether it's technically viable
A hybrid fil-C/Rust world would yield a somewhat surprising incentive gradient: Rewrite your C in Rust not to make it safer but to make it faster. I wonder how that would pan out.
I'd love to see a Fil-C FFI for the languages that I use that so often depend on C for libraries.