2023-10-02 11:01:00 <KipIngram   > This seems to be the "seminal paper" on log structured file systems:
2023-10-02 11:01:02 <KipIngram   > https://web.stanford.edu/~ouster/cgi-bin/papers/lfs.pdf
2023-10-02 11:03:42 <veltas      > So is a "log file system" sort of the other extreme from an unjournaled filesystem?
2023-10-02 11:05:10 <veltas      > My understanding of journaling is you have a journal/log for new data for speed and reliability but periodically move data into efficient access patterns for long term storage
2023-10-02 11:07:41 <KipIngram   > Yes, the entire job is done by writing sequentially through the disk.  At least that's the "primary strategy."
2023-10-02 11:08:00 <KipIngram   > As we discussed the other day, heat segregation might cause some deviations.
2023-10-02 11:08:14 <KipIngram   > You don't update any data "in place."
2023-10-02 11:09:02 <KipIngram   > One of the assumptions that makes it effective is that you presume a lot of RAM, so that most reads are satisfied from a RAM cache.
2023-10-02 11:09:14 <KipIngram   > That lets you keep the seek head at the write point, so you eliminate most seeks.
2023-10-02 11:09:27 <KipIngram   > In a flash drive seeking isn't really an issue anyway, though.
2023-10-02 11:10:50 <veltas      > Seems like it works fine with modest RAM and assuming you can read the whole disk in one go relatively quickly
2023-10-02 11:11:18 <veltas      > Then you can build a RAM index/tree and you wouldn't even need to store any index structures on disk
2023-10-02 11:11:30 <veltas      > The only advantage of writing index structures would be initial reading cost
2023-10-02 11:16:27 <KipIngram   > Yes, I'd think so too.
2023-10-02 11:17:00 <KipIngram   > Anyway, though, at the time it was proposed spinning disks were still primary, and if you do have to service random reads from the disk you get back a lot of seek time.
2023-10-02 11:17:50 <KipIngram   > The way I'm approaching this, for my project, is that I primarily need the flash to hold my stuff between sessions, when power is off.
2023-10-02 11:18:03 <KipIngram   > During a session my guess is that most often I'll read once and then just "work."
2023-10-02 11:18:39 <KipIngram   > And in my case it's a NOR flash, so reading is a non-issue anyway.
2023-10-02 11:18:45 <KipIngram   > Reads like RAM.
2023-10-02 11:19:29 <veltas      > Yeah I'm thinking for a small storage device like a NOR Flash you can handle something that you just index yourself by reading the whole journal at power-on
2023-10-02 11:19:38 <KipIngram   > Yep.
2023-10-02 11:19:43 <veltas      > It's worth it for how it simplifies the structure/code
2023-10-02 11:20:01 <KipIngram   > Of course, I'm using a calculator as my "target application," so I'll need to store all of my variable values across power offs.
2023-10-02 11:20:10 <veltas      > But for a big capacity disk that would be rubbish
2023-10-02 11:20:25 <KipIngram   > I guess think of it as saving a "workspace."
2023-10-02 11:20:31 <veltas      > You really don't need a filesystem for a calculator though
2023-10-02 11:20:54 <KipIngram   > No, and I'm not thinking quite in those terms.  Still thinking in terms of blocks.
2023-10-02 11:21:00 <veltas      > You probably need similar idea to journal for your wear levelling though
2023-10-02 11:21:04 <KipIngram   > But I'm planning a mapping...
2023-10-02 11:21:08 <KipIngram   > Right - ^ that.
2023-10-02 11:21:34 <veltas      > Also store in a format that increases number of 1's
2023-10-02 11:21:41 <veltas      > 0's cause more wear
2023-10-02 11:21:49 <veltas      > If I remember right
2023-10-02 11:21:50 <KipIngram   > Well, erases cause wear.
2023-10-02 11:22:01 <KipIngram   > When you erase the bits either all go to 0 or all go to 1.
2023-10-02 11:22:12 <KipIngram   > And "writing" is how you switch some of them to the other value.
2023-10-02 11:22:18 <veltas      > I'm thinking of erase-to-1, the programming causes some wear too if I remember right
2023-10-02 11:22:49 <KipIngram   > I don't  know for sure, but I've always thought you could write a cell more than once, so long as you were only taking bits from "erased" to "not erased."
2023-10-02 11:23:03 <veltas      > It's often easy to reduce number of 0's, you can have a bit represent whether a block should be inverted on read
2023-10-02 11:23:15 <veltas      > You can usually program 1 or a few times, depending on the NOR
2023-10-02 11:23:22 <veltas      > But a limited number of programs
2023-10-02 11:23:23 <KipIngram   > It probably varies from device to device.
2023-10-02 11:23:34 <veltas      > Yes that varies a lot
2023-10-02 11:23:41 <veltas      > For NOR especially
2023-10-02 11:24:37 <KipIngram   > I'll need to save my page mapping across power cycles.  I was thinking i'd just write it to whatever is the write point and then save that location somewhere - tricks like we're talking about might be useful when saving that table location in a "special place."
2023-10-02 11:24:53 <KipIngram   > that probably made  no sense.
2023-10-02 11:24:58 <KipIngram   > I didn't phrase it very well.
2023-10-02 11:25:28 <KipIngram   > I just figure I need one piece of data in a known location, to find the stuff that moves.
2023-10-02 11:25:40 <KipIngram   > And in that fixed location, any tricks to extend lifetime would be of interest.
2023-10-02 11:26:16 <KipIngram   > I could just use a signature and a time/date stamp on the mapping table and then just search the whole flash for the latest one at power on.
2023-10-02 11:26:43 <KipIngram   > Since it's only 512k, searching all of it wouldn't be that big of a deal, if you only had to do it once each session.
2023-10-02 11:33:55 <KipIngram   > I'm still pondering exactly how far toward a file system to go.  My wife's calculator (HP-48) supports a "directory structure."  It feels a lot like a typical file system.
2023-10-02 11:34:35 <KipIngram   > And "programs" are stored by name, a lot like just having source code in a string variable.
2023-10-02 11:35:27 <KipIngram   > I think this system I'm planning now may turn out to be fully decompilable, so I might not actually need "source."
2023-10-02 11:35:39 <KipIngram   > Just decompile for screen rendering.  Not 100% sure of that yet, though.
2023-10-02 11:36:48 <KipIngram   > I don't want to get SO focused on a calculator, though, that I overlook other applications completely.
2023-10-02 11:45:27 <rendar      > is ok to have lowercase words and symbols into them? e.g. kd3@C or kd3@O ?
2023-10-02 11:47:20 <KipIngram   > Yes, Forth is fine with that.
2023-10-02 11:47:40 <KipIngram   > "Standard" operation is to be case sensitive, but how you "use that" is entirely up to you.
2023-10-02 11:47:59 <KipIngram   > You can even define fully numeric names, if you want to mess with your own head.
2023-10-02 11:48:06 <KipIngram   > Forth doesn't care.
2023-10-02 11:48:13 <thrig       > programmers might
2023-10-02 11:48:23 <KipIngram   > Sometimes you run into case insensitive ones, but that has to be done "deliberately."
2023-10-02 11:48:23 <GeDaMo      > : 0  1 ; :P
2023-10-02 11:48:32 <KipIngram   > Yep.
2023-10-02 11:49:01 <rendar      > but Forth words are case sensitive usually, right?
2023-10-02 11:49:03 <GeDaMo      > Some older systems would sometimes define : 0 0 ; so it didn't have to parse 0 every time
2023-10-02 11:49:07 <KipIngram   > Yes.
2023-10-02 11:49:14 <KipIngram   > Case sensitivity is the norm.
2023-10-02 11:49:28 <KipIngram   > But Forth has absolutely zero concept of "reserved words."
2023-10-02 11:49:32 <KipIngram   > Nothing is off limits.
2023-10-02 11:50:26 <GeDaMo      > Usually, it's anything but a space
2023-10-02 11:50:47 <KipIngram   > Well, yes - spaces are special.
2023-10-02 11:51:04 <rendar      > indeed
2023-10-02 11:51:37 <KipIngram   > vms14 mentioned multi-word word names the other day, and I've thought a teeny bit about it on occasion.
2023-10-02 11:52:07 <KipIngram   > You could make a dictionary entry with spaces in the name no problem - it's the parsing of the source out of the input stream where the spaces are relevant.
2023-10-02 11:52:21 <KipIngram   > It only ever tries to look up words with no internal spaces.
2023-10-02 11:52:21 <rendar      > multi-word word? a word with spaces into it?
2023-10-02 11:52:37 <KipIngram   > And the standard tools for word creation won't let you make an internal space.
2023-10-02 11:52:43 <rendar      > yeah
2023-10-02 11:52:46 <KipIngram   > But the dictionary as a data structure could hold such a thing.
2023-10-02 11:53:07 <rendar      > i use lisp notation for space, e.g. some-thing
2023-10-02 11:53:23 <KipIngram   > That works fine in Forth.
2023-10-02 11:53:48 <rendar      > yeah
2023-10-02 12:06:54 <rendar      > let's say i have x,y in the stack, and i want to do some math operations on those coordinates, my functions results in `1.0 1.0 dup sin swap cos * swap dup sin swap cos *` where 1.0 1.0 are x y coordinates, but ..isn't much overhead 3 swaps? consider if we do that millions of times..
2023-10-02 12:09:33 <rendar      > basically the operation is x,y => sin(x)*cos(x), sin(y)*cos(y)
2023-10-02 12:13:07 <KipIngram   > Well, sure - anything you do a million times is going to take some amount of time.
2023-10-02 12:13:41 <KipIngram   > However, a couple of stack motions in the midst of a bunch of floating point arithmetic, and particularly including trig functions, isn't going to add much relatively speaking.
2023-10-02 12:13:43 <MrMobius    > rendar: maybe. doing a lot of stack operations can slow your code down if it's not being compiled to optimized machine code. on the other hand, if other things like sin and cos take a lot longer than swap, the overall hit to performance may not be noticeable
2023-10-02 12:14:28 <rendar      > hmm i see
2023-10-02 12:14:46 <rendar      > let's consider that in my case a swap would result in swapping 32 bytes
2023-10-02 12:15:20 <KipIngram   > How would it do that?
2023-10-02 12:15:26 <KipIngram   > Swap swaps a pair of cells.
2023-10-02 12:15:34 <rendar      > in my case 1 cell is 32 bytes
2023-10-02 12:15:37 <KipIngram   > So that would be eight bytes in a 64-bit system.
2023-10-02 12:15:43 <KipIngram   > I see - you're creating something new?
2023-10-02 12:15:48 <rendar      > yeah
2023-10-02 12:16:01 <KipIngram   > You're dealing with 32 BYTE floating point numbers??
2023-10-02 12:16:03 <rendar      > this is not a Forth like we know
2023-10-02 12:16:08 <rendar      > lol nope
2023-10-02 12:16:20 <rendar      > but i have an algebraic type which includes also floating points
2023-10-02 12:16:29 <KipIngram   > I see.
2023-10-02 12:16:45 <KipIngram   > Well, obviously swapping 32 byte items will cost more than swapping 64-bit ones.
2023-10-02 12:17:05 <rendar      > yeah
2023-10-02 12:17:11 <KipIngram   > But worse, I'm guessing you'll have written that swap word in Forth, so you'll have some standard Forth overhead in that.
2023-10-02 12:17:16 <KipIngram   > Normal swap is a fast primitive.
2023-10-02 12:17:45 <GeDaMo      > What's actually on the stack? The whole 32 bytes or a pointer?
2023-10-02 12:17:55 <rendar      > no that swap word executes some code written in Rust
2023-10-02 12:18:08 <rendar      > GeDaMo, the whole 32 bytes thing
2023-10-02 12:19:20 <GeDaMo      > At least it should still be in L1 cache
2023-10-02 12:19:46 <rendar      > indeed
2023-10-02 12:19:55 <rendar      > but i think i'll add some facilities to manipulate stack
2023-10-02 12:20:52 <rendar      > e.g., let's consider this is the stack |1.0 2.0| so, stack-0 is 2.0 and stack-1 is 1.0, now, if *without swapping* i can fake that top of stack is stack-1, i can call cos and sin operations into that
2023-10-02 12:21:36 <rendar      > basically i manipulate top of stack index
2023-10-02 12:22:18 <rendar      > without moving items, that remains just there
2023-10-02 12:26:58 <KipIngram   > Hmmm.
2023-10-02 12:27:15 <KipIngram   > You'd have to guarantee that that code wouldn't push anything onto the stack.
2023-10-02 12:27:44 <KipIngram   > Somehow you have to keep up with what memory is "in use."
2023-10-02 12:27:53 <KipIngram   > And usually the stack0 item is cached in a register.
2023-10-02 12:28:01 <KipIngram   > But your system sounds pretty different.
2023-10-02 12:29:15 <rendar      > indeed
2023-10-02 12:33:47 <KipIngram   > I do still feel that the floating point math and the trig functions are likely to dominate your timing.
2023-10-02 12:34:23 <KipIngram   > Especially since, as GeDaMo pointed out, those stack items are likely in your cache.
2023-10-02 12:46:20 <rendar      > yeah i think so
2023-10-02 14:52:37 <lispmacs[work]> thrig: hi, are you thrig of thrig.me fame?
2023-10-02 14:54:23 <lispmacs[work]> I was reading the Funky Filenames post, I was just curious if you knew: is there some program that lists all the files on a filesystem by inode, without having to walk the directory trees?
2023-10-02 14:55:36 <thrig       > it would probably be a very low level tool that knows how to find where the filesystem puts inodes
2023-10-02 14:56:04 <lispmacs[work]> I can't think of a compelling reason that would be necessary, but was curious
2023-10-02 14:57:04 <lispmacs[work]> I couldn't find one in a shallow duckduckgo search
2023-10-02 14:57:27 <lispmacs[work]> maybe I should search for filesystem specific utilities
2023-10-02 14:58:53 <lispmacs[work]> e2fsprogs, that sounds promising
2023-10-02 14:59:56 <thrig       > you dd over some portion of your disk and want the inodes back would be one case
2023-10-02 15:03:53 <lispmacs[work]> I just found fstransform package ... automatically converts file system to another file system type without copying the files
2023-10-02 15:04:18 <lispmacs[work]> that is fancy
2023-10-02 15:04:57 <lispmacs[work]> a few file recovery tools available in Guix, maybe one of them has option to list the inodes