@prologic https://twtxt.net/user/prologic/twtxt.txt
Unterhaltung: 30 Beiträge von @alexonit, @bender, @lyse, @movq, @prologic und @zvava.
@prologic https://twtxt.net/user/prologic/twtxt.txt
Unterhaltung: 30 Beiträge von @alexonit, @bender, @lyse, @movq, @prologic und @zvava.
ok so i have found a genuine twt hash collision. what do i do.
internally, bbycll relies on a post lookup table with post hashes as keys, this is really fast but i knew i'd inevitably run into this issue (just not so soon) so now i have to either:
1) pick the newer post over the other
2) break from specification and not lowercase hashes
3) secretly associate canonical urls or additional entropy with post hashes in the backend without a sizeable performance impact somehow
@zvava Herw you go: https://git.mills.io/yarnsocial/twtxt.dev/pulls/28
@prologic im unsure how i feel about the hash v2 proposal, given it is completely backward incompatible with hash v1 it doesn't really solve any of the problems with it. it only delays collisions, and still fragments threads on post edits
i skimmed through discussions under other the proposals — i agree humans are very bad at keeping the integrity of the web in tact, but hashes in done in this way make it impossible even for systems to rebuild threads if any post edits have occurred prior to their deployment
@zvava It is just completely impossible to make v2 backwards-compatible with v1.
Well, breaking threads on edits is considered a feature by some people. I reckon the only approach to reasonably deal with that property is to carefully review messages before publishing them, thus delaying feed updates. Any typos etc., that have been discovered afterwards, are just left alone. That's what I and some others do. I only risk editing if the feed has been published very few seconds earlier. More than 20 seconds and I just ignore it. Works alright for the most part.
@lyse i dont mind if the hash is not backward compatible but im not sure if this is the right way to proceed because the added complexity dealing with two hash versions isnt justified
regular end users wont care to understand how twt hashes are formed, they just want to use twtxt! so i guess i could work in protecting users from themselves by disallowing post edits on old posts or posts with replies, but i'm not fond of this either really. if they want to break a thread, they can just delete the post (though i've noticed yarn handling post deletes dubiously...)
on activitypub i do genuinely find myself looking through several month or even year old posts sometimes and deciding to edit/reword them a little to be slightly less confusing, this should be trivial to handle on twtxt which is an infinitely simpler specification
@zvava There would be only one hash for a message. Some to be defined magic date selects which hash to use. If the message creation timestamp is before this epoch, hash it with v1, otherwise hammer it through v2. Eventually, support for v1 could be dropped as nobody interacts with the old stuff anymore. But I'd keep it around in my client, because why not.
If users choose a client which supports the extensions, they don't have to mess around with v1 and v2 hashing, just like today.
As for the school of thought, personally, I'd prefer something else, too. I'm in camp location-based addressing, or whatever it is called. There more I think about it, a complete redesign of twtxt and its extensions would be necessary in my opinion. Retrofitting has its limits. Of course, this is much more work, though.
@zvava @lyse I also think a location based reference might be better.
A thread is a single post of a single feed as a root, but the hash has the drawback of not referencing the source, in a distributed network like twtxt it might leave some people out of the whole conversation.
I suggest a simpler format, something like: (#<TIMESTAMP URL>)
This solves three issues:
[ timestamp, url, content ]@alexonit Personally, I find the reversed order of URL first and then timestamp more natural to reference something. Granted, URL last would be kinda consistent with the mention format. However, the timestamp doesn't act as a link text or display text like in a mention, so, it's some different in my opinion. But yeah.
@alexonit @lyse i really don't understand why this was not the solution in the first place, given how simple twtxt is (mean to be), a reply should be as simple as #<https://example.com/twtxt.txt#2025-09-22T06:45Z> with the timestamp in an anchor link. the need for a mention is avoided like this as well since it's already linking to the replied-to feed!
🐀💭 i should just implement it into bbycll and force it into existence
We've been discussing the idea of changing the threading model from Content-based Addressing to Location-based addressing for years now. The problem is quite complex, but I feel I have to keep reminding y'all of the potential perils of changing this and the pros/cons of each model:
With content-addressed threading, a reply points at something that’s intrinsically identified (hash of author/feed URI + timestamp + content). That ID never changes as long as the content doesn’t. Switching to location-based anchors makes the reply target extrinsic—it now depends on where the post currently lives. In a pull-based, decentralised network, locations drift. The moment they do, thread identity fragments.
Here is just a small list of things™ that I'm aware will break, some quite badly, others in minor ways:
@prologic I know we won't ever convince each other of the other's favorite addressing scheme. :-D But I wanna address (haha) your concerns:
I don't see any difference between the two schemes regarding link rot and migration. If the URL changes, both approaches are equally terrible as the feed URL is part of the hashed value and reference of some sort in the location-based scheme. It doesn't matter.
The same is true for duplication and forks. Even today, the "cannonical URL" has to be chosen to build the hash. That's exactly the same with location-based addressing. Why would a mirror only duplicate stuff with location- but not content-based addressing? I really fail to see that. Also, who is using mirrors or relays anyway? I don't know of any such software to be honest.
If there is a spam feed, I just unfollow it. Done. Not a concern for me at all. Not the slightest bit. And the byte verification is THE source of all broken threads when the conversation start is edited. Yes, this can be viewed as a feature, but how many times was it actually a feature and not more behaving as an anti-feature in terms of user experience?
I don't get your argument. If the feed in question is offline, one can simply look in local caches and see if there is a message at that particular time, just like looking up a hash. Where's the difference? Except that the lookup key is longer or compound or whatever depending on the cache format.
Even a new hashing algorithm requires work on clients etc. It's not that you get some backwards-compatibility for free. It just cannot be backwards-compatible in my opinion, no matter which approach we take. That's why I believe some magic time for the switch causes the least amount of trouble. You leave the old world untouched and working.
If these are general concerns, I'm completely with you. But I don't think that they only apply to location-based addressing. That's how I interpreted your message. I could be wrong. Happy to read your explanations. :-)
@lyse @prologic Can't we find a middle ground and support both?
The thread is defined by two parts:
The client/pod generate the hash and index it in it's database/cache, then it simply query the subject of other posts to find the related posts, right?
In my own client current implementation (using hashes), the only calculation is in the hash generation, the rest is a verbatim copy of the subject (minus the # character), if this is the common implemented approach then adding the location based one is somewhat simple.
function setPostIndex(post) {
// Current hash approach
const hash = createHash(post.url, post.timestamp, post.content);
// New location approach
const location = post.url + '#' + post.timestamp;
// Unchanged (probably)
const subject = post.subject;
// Index them all
addToIndex(hash, post);
addToIndex(location, post);
addToIndex(subject, post);
}
// Both should work if the index contains both versions
getThreadBySubject('#abcdef') => [post1, post2, post3]; // Hash
getThreadBySubject('https://example.com#2025-01-01T12:00:00') => [post1, post2, post3]; // Location
As I said before, the mention is already location based @<example https://example.com/twtxt.txt>, so I think we should keep that in consideration.
Of course this will lead to a bit of fragmentation (without merging the two) but I think this can make everyone happy.
Otherwise, the only other solution I can think of is a different approach where the value doesn't matter, allowing to use anything as a reference (hash, location, git commit) for greater flexibility and freedom of implementation (this probably need the use of a fixed "header" for each post, but it can be seen as a separate extension).
@alexonit Yhays kind of love you!! Stance and position on this. If we are going to make chicken changes in the threading model, let's keep content based addressing, but also improve the use of experience. So in fact, in order to answer your question, I think yes, we can do some kind of combination of both.
@prologic That is really great to hear!
If there are opposing opinions we either build a bridge or provide a new parallel road.
Also, I wouldn't call my opinion a "stance", I just wish for a better twtxt thanks to everyone's effort.
The last thing we need to do is decide a proper format for the location-based version.
My proposal is to keep the "Subject extension" unchanged and include the reference to the mention like this:
// Current hash format: starts with a '#'
(#hash) here's text
(#hash) [@nick](/reader/index.php?action=own&url=url) here's text
// New location format: valid URL-like + '#' + TIMESTAMP (verbatim format of feed source)
(url#timestamp) here's text
(url#timestamp) [@nick](/reader/index.php?action=own&url=url) here's text
I think the timestamp should be referenced verbatim to prevent broken references with multiple variations (especially with the many timezones out there) which would also make it even easier to implement for everyone.
I'm sure we can get @zvava, @lyse and everyone else to help on this one.
I personally think we should also consider allowing a generic format to build on custom references, this would allow for creating threads using any custom source (manual, computed or external generated), maybe using a new "Topic extension", here's some examples.
// New format for custom references: starts with a '!' maybe?
(!custom) here's text
(!custom) [@nick](/reader/index.php?action=own&url=url) here's text
// A possible "Topic" parse as a thread root:
[!custom] start here
[custom] simpler format
This one is just an idea of mine, but I feel it can unleash new ways of using twtxt.
I was trying to say (badly):
That's kind of my position on this. If we are going to make significant changes in the threading model, let’s keep content based addressing, but also improve the user experience. Answering your question, yes I think we can do some combination of both.
@prologic I can see the issues mentioned, but I think some can be fixed.
The current hash relies on a url field too, by specification, it will use the first # url = <URL> in the feed's metadata if present, that too can be different from the fetching source, if that field changes it would break the existing hashes too, a better solution would be to use a non-URL key like # feed_id = <UNIQUE_RANDOM_STRING> with the url as fallback.
We can prevent duplications if the reference uses that same url field too or the client "collapse" any reference of all the urls defined in the metadata.
I agree that hashing based on content is good, but we still use the URL as part of the hashing, which is just a field in the feed, easily replicable by a bot, also noting that edits can also break the hash, for this issue an alternative solution (E.g. a private key not included in the feed) should be considered.
For offline reading the source would be downloaded already, the fetching of non followed feeds would fill the gap in the same way mentions does, maybe I'm missing some context on this one.
To prevent collisions there was a discussion on extending the hash (forgot if that was already fixed or not), but without a fallback that would break existing clients too, we should think of a parallel format that maintains current implementations unchanged, we are already backward compatible with the original that don't use threads at all, a mention style format for that could be even more user-friendly for those clients.
We should also keep in mind that the current mention format is already location based (@<example https://example.com/twtxt.txt>) so I'm not that worried about threads working the same way.
Hope to see some other thought about this matter. 🤓
My proposal is taken from https://texudus.readthedocs.io/en/latest/ BTW.