2023-03-29 10:09:10 <KipIngram   > vms14: It's not that block "files" are particularly "cool" - it's more just that they offer an interface to the drive that matches the drive's native capabilities.  In most Forth's you run under an OS you're not really doing raw disk operations - you're keeping your blocks in an OS file and just providing the block *interface*.  But in a bare metal system, this is the "bottom level" of your disk interface,
2023-03-29 10:09:12 <KipIngram   > and it matches the hardware capabilities.
2023-03-29 10:09:27 <KipIngram   > Then you'd use those capabilities to build whatever other disk capability you wanted, such as a file system, etc.
2023-03-29 10:09:51 <KipIngram   > In an os-based Forth, blocks are sort of an optional feature, but in a bare metal system that feature is where you "start" dealing with the drive.
2023-03-29 10:19:08 <KipIngram   > Forth's general philosophy is to provide "lowest level" ability to interact with the hardware.  Block files in an os-Forth don't really truly give you that; they just emulate the functionality.
2023-03-29 15:01:46 <veltas      > Downside to blocks is the lines are short and it wastes an insane amount of space
2023-03-29 15:02:21 <veltas      > And the typical approach to blocks complicates line-end termination of things like single-line comments
2023-03-29 15:02:30 <veltas      > Because the whole block goes in the input buffer
2023-03-29 15:03:13 <veltas      > Those are my complaints. I think every good forth system deserves a block interface regardless because it's easy to use, easy to implement, easy for the computer to grok. You can design good databases for it etc
2023-03-29 15:04:16 <veltas      > And you can design a simple filesystem for a block interface easily, an example is in... I think the polyforth manual
2023-03-29 15:04:28 <veltas      > Or swiftforth manual if not
2023-03-29 15:06:42 <veltas      > I'm sort of interested in the concept of having one-block-files, i.e. you limit each 'file' to 1024 characters. Then the lines can be longer and you can get more out of your block space
2023-03-29 15:07:59 <veltas      > It's very retro having blocks, they feel like punchcards. It's a nice feeling and a lot of the point of Forth is that retro aesthetic so it's very legit
2023-03-29 15:08:15 <veltas      > But it just annoyed me too much last time I tried it
2023-03-29 15:15:23 <Zarutian_iPad> what was that thing called that basically had object database on tape but with the ?class? definations at the front
2023-03-29 15:31:34 <veltas      > Zarutian_iPad: You mean like this sort of thing? https://i.imgur.com/lAIpW0o.png
2023-03-29 15:33:17 <veltas      > That's from the polyforth manual you can get from the greenhills website
2023-03-29 15:33:38 <Zarutian_iPad> nope
2023-03-29 15:33:56 <veltas      > I mean greenarrays
2023-03-29 15:34:15 <KipIngram   > veltas: The line length and space usage aspects you mention aren't features of "blocks" - they're features of how you use the blocks.
2023-03-29 15:34:41 <Zarutian_iPad> this was in USA navy reserarch on pdp11 or such eight track data tapes
2023-03-29 15:34:43 <KipIngram   > You can certainly use newlines in blocks, and for that matter you can use compression.
2023-03-29 15:35:06 <veltas      > Compression feels too far
2023-03-29 15:35:32 <KipIngram   > I think fretting over the space efficiency of stored text isn't really required these days - the only thing that really takes up significant s pace is video and other "big" data.
2023-03-29 15:35:55 <KipIngram   > Well, I agree, but you *could* do it - it's not precluded by using blocks directly.
2023-03-29 15:37:15 <KipIngram   > In our drives at work the post-compression, post ECC unit that we actually store is referred to as a "code page," and it's a fixed size right at 8 kB.
2023-03-29 15:37:53 <veltas      > When we talk about source blocks we usually mean this kind of setup where each location in a 64x20 or similar grid is one offset of a block
2023-03-29 15:38:15 <veltas      > Yeah that's a flash term
2023-03-29 15:38:18 <KipIngram   > So the whole strategy winds up storing fixed size blocks in the flfash.  Unfortunately, those blocks aren't the same size as the physical flash pages, so we have to stagger them.
2023-03-29 15:38:29 <KipIngram   > That makes some code pages twice as expensive to read out as others.
2023-03-29 15:39:49 <veltas      > When they said forth has the only truly portable storage... "blocks", it's not really true for flash. Because with flash really you want some notion of discarding which you don't get with typical forth block interfaces
2023-03-29 15:40:25 <KipIngram   > Yes, ideally you're offered an "unmap" or "trim" capability.
2023-03-29 15:41:10 <KipIngram   > You never overwrite data in place with flash; every write (even re-writes) writes to a brand new place, which tacitly "unmaps" the previously written data that had been associated with that page.
2023-03-29 15:41:20 <KipIngram   > But being able to explicitly unmap is useful too.
2023-03-29 15:42:15 <veltas      > An SSD will happily pretend to be a linear disk but you're not using it optimally if you do that
2023-03-29 15:43:02 <veltas      > linear array I should say
2023-03-29 15:43:13 <KipIngram   > I agree - that's the interface you're usually given, but I've always thought you could get better efficiency and performance if you coupled the file system more directly to the underlying structure.
2023-03-29 15:43:44 <KipIngram   > I mean, you still need to do wear leveling, and space recovery, and all that, but I think putting the file system and those things together would offer opportunities for gain.
2023-03-29 15:43:52 <KipIngram   > Instead of drawing a hard boundary between them.
2023-03-29 15:44:14 <veltas      > Wear leveling etc can happen transperently quite easily
2023-03-29 15:44:29 <veltas      > But lack of a 'discard' operation has an impact
2023-03-29 15:44:43 <KipIngram   > It does.
2023-03-29 15:45:14 <KipIngram   > And indeed wear leveling isn't that hard - to some extent the way we do our writes it kind of takes care of itself.  But it has to at least be considered.
2023-03-29 15:45:33 <veltas      > And also you need to just straight up tell the OS "this is actually random access, don't bother defragging or worrying about platters"
2023-03-29 15:45:57 <KipIngram   > They also try to group together data that has similar "write heat," and that is a trade off against "perfect" wear leveling.
2023-03-29 15:46:37 <veltas      > That's beyond my understanding, I just know the basics of NAND flash
2023-03-29 15:46:43 <KipIngram   > Perfect wear leveling comes from just going round robin around the physical pages and always writing to "the next one."  Any step you take away from that for other reasons lessens the perfection of the wear leveling.
2023-03-29 15:46:51 <veltas      > Yeah
2023-03-29 15:46:54 <KipIngram   > But there's a lot of room for trade-offs in there.
2023-03-29 15:47:14 <KipIngram   > The gain in the other direction is lower write amplification.
2023-03-29 15:48:45 <KipIngram   > I really do want to make a storage system from scratch one of these days.  All the various bits of it are actually very interesting.
2023-03-29 15:49:27 <veltas      > I agree
2023-03-29 15:49:59 <KipIngram   > When I took this job, I thought I "knew flash," because a whole lot of the devices we programmed at that programmer company were "flash."
2023-03-29 15:50:16 <KipIngram   > But that was NOR flash - NAND flash turns out to be a whole different, much more involved world.
2023-03-29 15:50:58 <veltas      > Yeah I've only worked with NOR, I do know about NAND but not to the level a drive manufacturer would
2023-03-29 16:04:26 <veltas      > KipIngram: what brand drives do your company make?
2023-03-29 16:06:01 <veltas      > I think I can find anything in computing interesting honestly, I'm a pretty boring person :P
2023-03-29 16:11:47 <KipIngram   > They're our own drives - IBM brand.  But we don't sell them on the open market - we use them in rack-mount storage products that we sell.
2023-03-29 16:12:37 <KipIngram   > https://en.wikipedia.org/wiki/Flash_Core_Module
2023-03-29 16:12:54 <KipIngram   > https://base.imgix.net/files/base/ebm/electronicdesign/image/2018/08/www_electronicdesign_com_sites_electronicdesign.com_files_FMS2018_Fig2.png?auto=format&fit=max&w=1440
2023-03-29 16:13:52 <KipIngram   > We're not the fastest gun in Dodge.  But we kick ass on capacity.
2023-03-29 16:14:36 <KipIngram   > Currently we have as much as 38.4TB of physical capacity in that form factor, and it offers hardware compression, so depending on what you're storing you can get quite a lot more than that on it.
2023-03-29 16:15:17 <KipIngram   > For English text you'd be talking about a 100 TB drive, barely bigger than a deck of cards.
2023-03-29 16:16:24 <veltas      > Nice
2023-03-29 16:16:27 <veltas      > I'm quite enjoying designing my Forth word processor
2023-03-29 16:17:46 <Zarutian_iPad> just from the image, KipIngram, these pcbs look like 4 to 8 layers
2023-03-29 16:20:16 <Zarutian_iPad> and the smallest passive component looks like 0306 or so sized smd
2023-03-29 16:21:10 <Zarutian_iPad> s/0306/0603/
2023-03-29 16:21:55 <Zarutian_iPad> it doesnt matter which way around as these are the two dimensions of the thing in thou
2023-03-29 17:57:33 <KipIngram   > I'm not caught up on all those details.  I don't work a lot with the PCB designers.  Mostly with the firmware and FPGA code developers.
2023-03-29 17:57:44 <KipIngram   > By the time I get my hands on them they're all buttoned up.
2023-03-29 18:25:03 <KipIngram   > The first round of boards for the latest model generation had some signal integrity issues.  My "old way" of testing these drives was to use a "Y-cable" arrangement.  The drive presents a pair of 2-bit lanes, that function as single connections, rather than one 4-bit lane.  So you have to make independent connections to the thing for each connection, and that Y cable allowed me to "plug it into two PCIe
2023-03-29 18:25:05 <KipIngram   > slots," so to speak.
2023-03-29 18:25:17 <KipIngram   > It showed up as two /dev/nvme devices in Linux, which I would test in parallel.
2023-03-29 18:25:48 <KipIngram   > But that cable exacerbated any signal integrity shortcomings, and when I tried to test this latest model that way it just wasn't happening.
2023-03-29 18:26:00 <KipIngram   > I couldn't even get a test off the launch pad without it failing.
2023-03-29 18:26:24 <KipIngram   > So now I use one of those rack-mount product boxes that I mentioned earlier, with the drives "properly installed" in it.
2023-03-29 18:26:57 <KipIngram   > I wiped the commercial software (which does a lot of storage management stuff) and installed Linux on each of the two "canisters" it has inside.
2023-03-29 18:27:07 <KipIngram   > So it's like having a pair of Linux servers in one box.
2023-03-29 18:27:32 <KipIngram   > I run the test software on both of them, use ssh to synchronize things, and aggregate the performance measurements when I'm done.
2023-03-29 18:27:59 <KipIngram   > I've seen a couple of issues that still may be signal integrity related, but it's a rare thing now.
2023-03-29 18:28:16 <KipIngram   > I need to bug my manager to swap my drives out for the ones with the new boards.
2023-03-29 18:30:42 <KipIngram   > Oh, and I set the first of those boxes up late last year.  But today I took delivery of two new ones.  Got them into my rack and on network controllable AC power, and established a couple of serial connections I need to them from another server.
2023-03-29 18:30:58 <KipIngram   > There's some proprietary hardware in there that prevents the drives from showing up on initial power up.
2023-03-29 18:31:12 <KipIngram   > I have to send a command to that hardware over the serial connecction to get it to "mount" them.
2023-03-29 18:31:53 <KipIngram   > That all went together quite nicely.  Tomorrow I'll take a shot at getting the Linux OS installs done.  But the first go round a guy helping me did that - I wans't able to figure it out.  That has to be done over the serial connection too.
2023-03-29 18:32:04 <KipIngram   > (There's no monitor connection on this box).
2023-03-29 18:32:28 <KipIngram   > He's on vacation this week, so if it doesn't make nice with me tomorrow I'll have to wait until he's back.
2023-03-29 18:32:46 <KipIngram   > This will bring me up to 36 drive bays.