2022-05-14 03:23:26 <veltas      > I don't disagree with that
2022-05-14 03:24:45 <veltas      > But I think you should make it dirt simple, because it's so critical to get right and performance isn't significantly impacted by a 'less efficient' layout in many cases, especially if you cache directory blocks
2022-05-14 03:25:07 <veltas      > And if it's simple you can add journaling easier too, more important than log n lookup
2022-05-14 03:27:01 <veltas      > Classically a filesystem is not expected to have database-like performance, but it *is* expected to protect the integrity of data in the face of bad blocks and power loss
2022-05-14 03:34:29 <veltas      > Also really if you're designing a new filesystem today, and especially if it's aimed at embedded, you should design it to optimise performance and reduce wear for NAND
2022-05-14 03:35:21 <veltas      > When I say 'performance' here I mean the performance of writing a file out to disk, for which the resolution and trees and all that are sort of irrelevant
2022-05-14 03:36:34 <veltas      > But it's actually *natural* for NAND to use journaling because you want to erase a page and write out a journal to blocks until it's 'full', and then try and commit the changes properly when you've done that
2022-05-14 03:37:24 <veltas      > And you want to try and keep the oldest pages the longest so you can try and recover file contents if something gets corrupted, maybe you can at least recover an older version of the file
2022-05-14 03:39:28 <veltas      > The big 'interface' change, if you're not working with low-level NAND, is that you need to tell the device when to discard a block. Classically you just ignore a block when it's 'dead', but now you need to say to the low-level device "it's okay to lose this data"
2022-05-14 03:39:34 <dave0       > did i mention littlefs the other day?
2022-05-14 03:42:05 <veltas      > https://github.com/littlefs-project/littlefs/blob/master/DESIGN.md
2022-05-14 03:42:15 <dave0       > ah nice
2022-05-14 03:42:20 <veltas      > Can't remember if you mentioned it but I'm aware of it, I considered it for a project at work
2022-05-14 03:42:34 <dave0       > i read about it on hackernews
2022-05-14 03:43:02 <veltas      > And then ended up writing my own solution. LittleFS is good but I didn't want a 'little' implementation, I wanted something that would be 'compact' in an actually tiny storage area (and have a small implementation)
2022-05-14 03:43:23 <veltas      > I found it with google because I was trying to research possibilities for our data storage
2022-05-14 03:43:52 <veltas      > But truth is that our requirements turned out to be wanting something more like a 'registry' than a filesystem, i.e. with small blocks or no blocks
2022-05-14 03:44:04 <veltas      > We went with really tiny blocks like in the order of 32 bytes
2022-05-14 03:44:10 <veltas      > 16-32
2022-05-14 03:44:52 <veltas      > It pains me that it's proprietary because it's something I'm proud of, and IMO it's overcomplicated for our purposes so I'm going to try and kill it internally
2022-05-14 03:45:00 <dave0       > one thing in littlefs was that it was bounded memory
2022-05-14 03:45:05 <dave0       > that was interesting
2022-05-14 03:45:14 <veltas      > Such is the life of the embedded engineer lol
2022-05-14 03:46:10 <dave0       > veltas: even though its for flash, do you think a floppy disk would benefit from a "littlefs" ?
2022-05-14 03:46:25 <dave0       > the wear levelling is interesting
2022-05-14 03:47:06 <veltas      > I think it's a super-set of the requirements for magnetic disk
2022-05-14 03:47:11 <veltas      > So it should work fine on floppy
2022-05-14 03:47:21 <dave0       > cool
2022-05-14 03:47:24 <veltas      > Main issue it probably doesn't handle is optimising seeking
2022-05-14 03:47:36 <veltas      > Because it's designed for flash which is random access
2022-05-14 03:47:39 <dave0       > yeah i have a floppy where the first couple of tracks are worn out because the FAT wore out
2022-05-14 03:48:03 <dave0       > ah
2022-05-14 03:48:10 <veltas      > I didn't honestly realise floppy could 'wear out' like that!
2022-05-14 03:48:36 <dave0       > it's like 30 years old :-p
2022-05-14 03:48:46 <veltas      > But the flash access patterns, copy-on-write, etc, could probably be easily optimised for seeking if not already
2022-05-14 03:49:47 <dave0       > i don't like unix filesystems where there is inodes separate from the name of the file
2022-05-14 03:49:59 <veltas      > Yeah it's not a good model
2022-05-14 03:50:41 <veltas      > It's a shame because ext4 is really well designed IMO
2022-05-14 03:50:57 <dave0       > oh is it?
2022-05-14 03:51:58 <dave0       > learning how filesystems work at the disk level is on my TODO list lol
2022-05-14 03:52:39 <veltas      > Yeah but it's not aimed at flash. Doesn't hurt it too much but as a Forther I think you should design for one or the other to minimise requirements
2022-05-14 03:52:44 <dave0       > i found a microsoft FAT sepcification, and also exFAT specification
2022-05-14 03:53:38 <veltas      > FAT is horrid, loads of fields where it doesn't need them, bad limitations on size and weird conventions on working around size limits
2022-05-14 03:53:48 <veltas      > exFAT has a lot less support than people seem to think
2022-05-14 03:54:08 <veltas      > Like i.e. your car MP3 player of yesterday didn't understand it even though exFAT was old when it came out
2022-05-14 03:54:17 <veltas      > And a lot of USB sticks come formatted with exFAT
2022-05-14 03:54:29 <veltas      > So people are like "why doesn't my USB stick work with my car" and throw it away
2022-05-14 03:54:39 <veltas      > Try explaining that to a normal user!
2022-05-14 03:54:46 <dave0       > hm
2022-05-14 03:55:20 <veltas      > It's sad that computers are so complicated, I think Forth mindset has the right idea because we're actually trying to make it simpler
2022-05-14 03:55:32 <veltas      > Because you can't run away from the complexity, it's there, waiting, dormant
2022-05-14 03:59:47 <veltas      > The mission of desktop Forths IMO: to provide applications that replace existing desktop standards from text editors, to Excel, to web browsers
2022-05-14 03:59:57 <veltas      > Simplifying the requirements, making it dumber/simpler
2022-05-14 04:00:11 <veltas      > And then demonstrate the power of knowing *everything* about your software
2022-05-14 04:00:40 <veltas      > For example, "my text editor doesn't support X". Your text editor is like 200 lines of code, go add it.
2022-05-14 04:02:31 <veltas      > Okay I'm reaching the IRClog stage, I need to make a blog and just post links to that instead, sorry lol
2022-05-14 11:06:31 <KipIngram   > I am interested in wear-proteting NAND.  My day job is with a team at IBM that designs NAND SSDs for use in enterprise storage gear.  However, that doesn't mean I know file systems - as I said yesterday, we present a flat logical address space.  Under the hood, though, there is a logical-to-physical mapping designed for wear leveling, various types of heat optimization, and so on.
2022-05-14 11:06:55 <KipIngram   > I probably should say too much about details, but I know a lot of things on that front.
2022-05-14 11:07:07 <KipIngram   > I definitely intend to make my own nand storage system at some p oint.
2022-05-14 11:07:28 <thrig       > maybe you could sell pre-stressed ssd to fashionable hipsters
2022-05-14 11:09:17 <KipIngram   > We've "big on capaicty" - pun definitely intended.  This is a standard 2.5" drive form factor, but the biggest one has 38.4 TB of physical capacity and includes hardware compression.  So if you were storing English text on there you could probably tet 100 TB on it.
2022-05-14 11:09:24 <KipIngram   > Palm of your hand.
2022-05-14 11:10:16 <KipIngram   > Actually problaby not quite 100 TB - I think the limit on the logical size it will present for you is like 88 TB or something like that.
2022-05-14 11:10:47 <KipIngram   > Just because the logical-to-physical map has to be stored somewhere, and there's a size limit on that.
2022-05-14 11:11:25 <KipIngram   > Most of it is kept in RAM, but there's not enough RMA for the whole thing so it gets paged to and from flash.
2022-05-14 11:12:22 <KipIngram   > s/RMA/RAM/
2022-05-14 11:15:10 <KipIngram   > I do performance testing on the things.  In my "younger life" I'd have been plop in the middle of the design group, but I showed up at the little company the place used to be before IBM bought it too late for that.
2022-05-14 11:16:25 <KipIngram   > In that little company I ran the embedded software team, but IBM brought in legacy management, so the existing management all got new work.
2022-05-14 11:17:04 <KipIngram   > I was a little bummed about that intially, but there was a retention bundle that was too good to leave, and by the time that expired I'd realized I didn't want to be an IBM manager anyway.  Those guys never seem to touch the technology.
2022-05-14 11:17:18 <KipIngram   > And sit in meetings 85% of the time.
2022-05-14 11:17:20 <KipIngram   > Yuck.
2022-05-14 11:17:55 <thrig       > they seem to be in touch with whatever they've read recently in PC Magazine
2022-05-14 11:17:56 <KipIngram   > The great part of it was that they lowered no salaries when they took over, so I get the same money and a much lower-stress job.
2022-05-14 11:18:24 <KipIngram   > I'd fail as a manager in IBM, I'm quite sure.  The culture is just too different.
2022-05-14 11:18:49 <KipIngram   > Very "let's all hold hands and come to agreement on everything."  Sometimes it's hard to tell who's actually in charge of something.
2022-05-14 11:22:35 <KipIngram   > The work is a lot less hard now, and that's kind of good, because I'm a lot older.  I don't know if I still have it in me to push as hard as I did in the old days.  I took it really seriously and usually overworked myself quite a lot.  It's a wonder my wife didn't leave me along the way, but I got lucky there too.
2022-05-14 11:26:28 <KipIngram   > But those small companies... the owner or board or whatever trusted me with major responsibilities, and I felt a compulsion to warrant that trust.  It was as big or bigger a motivator than salary.  Big companies, on the other hand, seem hell-bent on making everyone as expendable as possible.
2022-05-14 11:27:01 <KipIngram   > I've had about 15 different bosses since 2012 - there's constant change and roll.
2022-05-14 11:29:07 <KipIngram   > So I do my one little piece of things, which is occasionally creatively stimulating (at least I got to roll my own tools and processes, and performance testing is a little more reliant on knowing the product than other kinds of "black box" testing), but most of the time it's turning a crank.  If I hadn't gotten to build the crank I would have been bored to tears, probably.
2022-05-14 11:29:52 <KipIngram   > But - same money, easier work.  Leaves me a lot more time to be a "family first" sort of person.
2022-05-14 11:30:11 <KipIngram   > So over the years I've come to feel I'm pretty lucky.
2022-05-14 11:32:16 <KipIngram   > It's actually a good thing they bought that little company - I had only been there five months when the acquisition went down, and during htose five months I learned enough about the fellow that owned that little company to now believe I wouldn't have been able to stand him long term.
2022-05-14 11:32:46 <KipIngram   > Most of his guys he'd hired right out of college - some of them had been with him like 20 years.  And he pretty much had them under his thumb.
2022-05-14 11:32:57 <KipIngram   > I was too old when I joined to let anyone do that to me.
2022-05-14 11:33:20 <KipIngram   > But by the time I got a read on him I knew the acquisition was coming, and that it would mean some money for me, so I gritted my teeth and stuck.
2022-05-14 11:34:40 <KipIngram   > He got a lot of $$$ out of that deal, but he didn't get what he "expected."  He thought that a big part of what IBM was buying was "his leadership" of the place, but they rendered him irrelevant pretty quickly when they actually came in in October of 2012, and he left by the end of the year.
2022-05-14 11:35:15 <KipIngram   > But he really didn't need an income anymore, so I don't feel bad for him or anything.  He was set.
2022-05-14 11:35:44 <KipIngram   > He tried to start another company later, but it didn't "go."
2022-05-14 11:35:58 <KipIngram   > These days he mostly finances his wife's political aspriations.
2022-05-14 11:36:10 <KipIngram   > Which have also not gone anywhere so far.
2022-05-14 11:37:27 <KipIngram   > I actually knew his wife before I ever met him - back when I was a VP at one of those other little companies she was an outside sales person for an electronics distributor, and would come around trying to peddle parts.
2022-05-14 11:37:50 <KipIngram   > Apparently she visited his company as well and wound up getting more than business there.  :-)
2022-05-14 11:38:55 <veltas      > KipIngram: You can talk about this "under the hood" levelling, it's not novel, all SATA SSDs do this
2022-05-14 11:39:04 <veltas      > Well to some extent anyway
2022-05-14 11:39:35 <veltas      > "Very "let's all hold hands and come to agreement on everything."" God I hate that kind of culture
2022-05-14 11:39:37 <KipIngram   > Well, that's why I said as much as I did - I felt like those parts weren't "novel," as you say.  I can try, but I'll want to be careful.  They're fairly sensitive.
2022-05-14 11:39:46 <KipIngram   > What kind of thing do you want to know?
2022-05-14 11:39:49 <veltas      > Very Soviet-style "we need consensus on everything"
2022-05-14 11:40:00 <KipIngram   > If you ask a question I'll see how far I can feel comfortable answering.
2022-05-14 11:40:07 <veltas      > I don't want to know I'm just saying I don't think it's quite as sensitive as you might think
2022-05-14 11:40:16 <KipIngram   > Yes - I imagine it's part of most really large companies.
2022-05-14 11:40:18 <veltas      > I'm just saying everyone does this kind of logical-to-raw stuff
2022-05-14 11:40:51 <veltas      > Every single SATA SSD does it, otherwise it wouldn't be able to obey the SATA protocol without killing it really quickly
2022-05-14 11:40:56 <veltas      > (I think?)
2022-05-14 11:41:04 <KipIngram   > What I was used to from the old days was "owner told me what he wanted and left the rest to me, I discussed architecture with the team but made final decisions, and they made it happen."
2022-05-14 11:41:14 <KipIngram   > I'd make the plan, the work schedule, etc.
2022-05-14 11:41:36 <KipIngram   > And I held a lot of influence over their promotions, raises, continued employement, and so on.
2022-05-14 11:41:45 <veltas      > I've said before, you need someone to steer the ship. Someone who can listen to council but ultimately has taste and will go against consensus occasionally
2022-05-14 11:41:47 <KipIngram   > None of that is the case for managers at IBM - it's all very "fuzzy."
2022-05-14 11:42:01 <KipIngram   > Yes - that exactly how I'd learned to operate.
2022-05-14 11:42:08 <veltas      > Or who can just make simple decisions without involving 10 people
2022-05-14 11:42:14 <KipIngram   > If I'd tried to do that at IBM I'd have crashed and burned.
2022-05-14 11:42:27 <KipIngram   > So it's sort of good I didn't have that opportunity.
2022-05-14 11:42:29 <veltas      > And with accountability ... like people need to admit they made the wrong decision and learn from it
2022-05-14 11:42:38 <KipIngram   > Right on.
2022-05-14 11:42:56 <veltas      > But these days too often if something goes wrong we say "well we followed procedure, nothing we could have done" even when I'm screaming at them to not do it that way in the first place
2022-05-14 11:42:59 <KipIngram   > Anyway, I do think most of this is just the way people approach NAND flash storage.
2022-05-14 11:43:34 <KipIngram   > Writes go sequentially through physical storage, period - the logical to physical mapping is just done to handle the rest.
2022-05-14 11:43:41 <KipIngram   > So wear leveling is automatic.
2022-05-14 11:44:19 <KipIngram   > Garbage collection attempts to keep up with the most "stale" blocks so that the smallest amount of still valid stuff has to be moved before erasing a block.
2022-05-14 11:44:29 <KipIngram   > I suspect all flash SSD vendors do it just like that.
2022-05-14 11:44:48 <veltas      > Probably yeah
2022-05-14 11:44:50 <KipIngram   > The manner in which they select blocks for garbage collection and erasure probably varies.
2022-05-14 11:45:07 <KipIngram   > Because you can't really track that "perfectly" - it would require too much RAM.
2022-05-14 11:45:13 <KipIngram   > So it's a heuristic.
2022-05-14 11:45:32 <KipIngram   > Especially when you then roll in read and write heat optimization.
2022-05-14 11:45:43 <veltas      > The erase size is much larger than a block usually I think
2022-05-14 11:45:57 <KipIngram   > QLC flash stores four bits of data in each flash cell, and those bits have different latencies.
2022-05-14 11:46:05 <KipIngram   > So we try to optimize over that too.
2022-05-14 11:46:36 <KipIngram   > A physical page holds four different logical pages, with four different read latencies.
2022-05-14 11:46:50 <KipIngram   > Well, sorry - compression comes in there too.
2022-05-14 11:47:04 <KipIngram   > There's not an exact one-to-one correspondence between logical and physical pages.
2022-05-14 11:47:21 <KipIngram   > But four streams get written into the physical pages, and recovering those streams carry different latencies
2022-05-14 11:47:50 <veltas      > Interesting
2022-05-14 11:47:52 <KipIngram   > I assume it's because it's faster to read out a voltage from a cell to "MSB" precision than to learn what the less significant bits are.
2022-05-14 11:48:15 <KipIngram   > After all, that voltage is split into 16 bins, for the four bits.
2022-05-14 11:48:34 <KipIngram   > If you only need to know you're in the top or bottom half, and you don't need to know WHERE in those halves, that's faster.
2022-05-14 11:48:46 <KipIngram   > If you need to know the exact bin (to get that fourth bit), that takes longer.
2022-05-14 11:49:33 <KipIngram   > A lot of the most sensitive stuff I probably don't even know, because it involves extremely intimate knowledge of the flash chips.
2022-05-14 11:50:01 <KipIngram   > It's ironic - the newer, higher density flash chips are really "crappier."  We do a lot of things to fight that crappiness and still get enterprise grade reliability.
2022-05-14 11:50:40 <KipIngram   > If you filled up one of our drives with data and then turned it off and put it on a shelf for a few months, it would probably have accrued more error than we can recover from and be unusable.
2022-05-14 11:51:03 <KipIngram   > The things are meant to be under power all the time, and have background algorithms running to continuously detect and correct bit errors.
2022-05-14 11:51:26 <KipIngram   > Before they get so many errors we can't remove them with our ECC stuff.
2022-05-14 11:51:41 <KipIngram   > I joke that our job is to make armor out of wet tissue paper.
2022-05-14 11:52:04 <KipIngram   > The first flash devices, that stored one bit per cell, were much more reliable.
2022-05-14 11:52:22 <KipIngram   > It just gets harder with every density step.  SLC, MLC, TLC, and now QLC.
2022-05-14 11:53:16 <spoofer     > One of my engingeering profs quoted someone, "Anyone can make a bridge that stands, but it takes an engineer to make a bridge that barely stands."
2022-05-14 11:53:17 <KipIngram   > After all, the whole way the things work is that when you write them you trap some charge in little "pockets" in the device, and later read that charge level back out.
2022-05-14 11:53:23 <KipIngram   > Well, they drift over time.
2022-05-14 11:53:38 <KipIngram   > The more bins you try to pack in there, the more that drift hurts you.
2022-05-14 11:54:11 <KipIngram   > The less drift it takes to push you into another bin.
2022-05-14 12:00:15 <KipIngram   > Anyway, an earlier job had made me very familiar with NOR flash (I was VP for a company called BP Microsystems - made device programming equipment).  But I was rather amazed in the new gig as I learned how many "NAND crappy things" we had to engineer around.
2022-05-14 12:01:10 <KipIngram   > You usually figure that in any given sector, the underlying ICs just get better and better over time, but in the NAND storage case they're really getting worse and worse.  Better on capacity, but worse in a lot of other ways.
2022-05-14 12:01:17 <KipIngram   > More hoops to jump.
2022-05-14 12:01:30 <KipIngram   > But capacity is king, so...
2022-05-14 12:02:03 <KipIngram   > So there are more engineering marvels to accomplish on each round, but yet the market still expects the storage to never become slower.
2022-05-14 12:02:25 <KipIngram   > I don't develop here, but I work very closely with the devleopers, and I've watched them struggle with that.
2022-05-14 12:02:25 <veltas      > NOR is a lot simpler
2022-05-14 12:02:28 <veltas      > But humbler
2022-05-14 12:02:31 <KipIngram   > Yes.
2022-05-14 12:02:54 <KipIngram   > A lot of the devices BP Micro's equipment knew how to program were NOR flash chips.
2022-05-14 12:03:13 <KipIngram   > Lots of other stuff too, but flash programming was an important "customer interest."
2022-05-14 12:03:20 <veltas      > BP Micro... rings a bell
2022-05-14 12:03:30 <KipIngram   > They're still out there.
2022-05-14 12:03:54 <KipIngram   > But they've struggled - there was a window of time when they were in the cat bird's seat, because programming chips was a sophisticated process.
2022-05-14 12:04:12 <KipIngram   > But the chips just kept getting easier to program, and that expertise became less valuable.
2022-05-14 12:04:21 <KipIngram   > I feel like I was there during the hayday.
2022-05-14 12:04:31 <KipIngram   > Most stuff more or less programs itself these days.
2022-05-14 12:05:18 <KipIngram   > But we made not only "engineering" units like you might have on your desk, but also big floor standing things that were used in manufacturing operations and programmed like 1400 parts per hour and things like that.  That was where the big money was.
2022-05-14 12:05:49 <KipIngram   > For years our customers needed to program more parts, bigger parts, this year than last year, so we sold a lot of those things.
2022-05-14 12:06:20 <KipIngram   > Then around 2001/2002 the industry downturned - as far as the customers were concerned the equipment they already had was enough, and our revenues just CRATERED.
2022-05-14 12:06:48 <KipIngram   > In the late 1990s revenues went up 50% a  year, year over year.
2022-05-14 12:07:15 <KipIngram   > Then in we went from $50 million in 2000 to $12 million in 2001.  :-|
2022-05-14 12:07:28 <KipIngram   > And the hayday was over.
2022-05-14 12:07:49 <KipIngram   > I gave the owner a letter in summer 2002 and tried my hand at consulting.
2022-05-14 12:08:09 <KipIngram   > Pro tip: don't go become a consultant in the middle of a tech downturn.  :-)
2022-05-14 12:08:18 <KipIngram   > I did ok - well enough to keep doing it for five years or so.
2022-05-14 12:08:33 <KipIngram   > But I never did as well as I had working for someone, so in 2007 and went back and found a new job.
2022-05-14 12:08:37 <KipIngram   > In the seismic industry.
2022-05-14 12:10:22 <KipIngram   > Most of my consulting gigs were related to motion control and machine vision - stuff I'd picked up at BP in the making of those big floor standing units.
2022-05-14 12:11:13 <KipIngram   > https://bpmmicro.com/bpm310/
2022-05-14 12:11:29 <KipIngram   > Fun story behind the BP Micro -> BPM Micro name change.
2022-05-14 12:12:06 <KipIngram   > Somewhere in the late 1990's BP (the big BP) decided to launch a new marketing campaign - promoting themselves as "into the clean energy future."
2022-05-14 12:12:13 <KipIngram   > Adopted a little sunburst logo, etc.
2022-05-14 12:12:30 <KipIngram   > Lawyer shows up at our door and tells us we're going to have to stop calling ourselves BP Microsystems.
2022-05-14 12:12:52 <KipIngram   > Basically just said "you've got the law on your side, but we have more lawyers and we will win."
2022-05-14 12:13:00 <KipIngram   > Just muscled us into compliance.
2022-05-14 12:13:22 <KipIngram   > The original name came from the names of the guys that started the company -  Bill and Peter.
2022-05-14 12:13:40 <KipIngram   > By the time I hired on Bill had forcibly bought Peter out.
2022-05-14 12:13:58 <KipIngram   > Told him that if he didn't sell his interest to him he'd just quit and start a new company.
2022-05-14 12:14:15 <KipIngram   > I never met Peter, but I do know Bill was a thoroughly brilliant engineer.
2022-05-14 12:14:21 <KipIngram   > One of the best I've ever worked around.
2022-05-14 12:14:51 <KipIngram   > This is where my deep-seated belief that technology companies should be led by technologists comes from.
2022-05-14 12:16:19 <thrig       > three envelope CEOs are another option
2022-05-14 12:16:44 <KipIngram   > So, yeah - those were my "glory days."  I was basically Bill's right hand so far as the tech stuff went.  He told me what his pet ideas were - I made the details pan out.  And there was another guy, the EVP, who ran all the sales and marketing stuff.
2022-05-14 12:17:26 <KipIngram   > I had to answer to both of them - to Bill and the EVP.  But the EVP never tired to tell me how to run technical things - he knew he had no clue.
2022-05-14 12:18:02 <KipIngram   > So I basically listened to what little architectural things Bill wanted to see come into the products, and let the EVP set the business priorities, and took it from there.
2022-05-14 12:19:51 <KipIngram   > The EVP got fired in early 2001.  He had the highest salary, and money had become tight for the owner.
2022-05-14 12:20:24 <KipIngram   > And I realized over the next few months that in spite of staying out of the tech stuff, the EVP had also "buffered me" (and the rest of us) from many of Bill's quirkiest quirks.
2022-05-14 12:20:52 <KipIngram   > By summer I'd had enough of doing without that buffer, and was starting to see that I had the next highest salary and Bill was still under financial pressure, so I quit.
2022-05-14 12:21:05 <KipIngram   > By the end of 2001 the entire top management layer had either quit or been let go.
2022-05-14 12:21:26 <KipIngram   > I feel sure Bill would have axed me before too much longer, just because of finances.
2022-05-14 12:21:37 <KipIngram   > He certainly could have done my job - I was really just a convenience to him.
2022-05-14 12:22:37 <KipIngram   > On the other hand, he really was NOT capable of doing the EVP's job - I expect it hurt the company to lose him.
2022-05-14 12:22:51 <KipIngram   > He was one of these guys that could sell ice cubes to Eskimos.
2022-05-14 12:28:09 <KipIngram   > He was a pretty shady guy, though.  My two predecessors running engineering there tried to not cooperate with him, and he crafted their departure from the company.
2022-05-14 12:28:18 <KipIngram   > I made a particular point to get along with him.
2022-05-14 12:29:26 <KipIngram   > It wasn't really hard - like I said, he never tried to actually call technical shots.  All he wanted was support for his business priorities, and that actually didn't seem "wrong" to me.  I sure didn't know how to sell the stuff.
2022-05-14 12:30:54 <KipIngram   > Ok - I'm going to try to shut up now; I got my memory lane button pressed.  veltas - was there anything more about the under the hood stuff you were interested in?
2022-05-14 12:31:30 <veltas      > As I said earlier I wasn't asking for it
2022-05-14 12:31:39 <veltas      > Although the bit I read was interesting :P
2022-05-14 12:31:44 <KipIngram   > Oh - well, I didn't catch that either time.  Sorry.
2022-05-14 12:32:04 <KipIngram   > I'm only now about half into my first cup of coffee.
2022-05-14 12:32:19 <KipIngram   > I used to wake up ready to roll, but these days it takes a little caffeine.
2022-05-14 12:33:08 <KipIngram   > The heater in the coffee maker fritzed out yesterday.  New one should come today.  But yesterday I heated up a pot of water on the stovetop and poured it through.
2022-05-14 12:33:55 <KipIngram   > I've got one of those sous vide immersion things, so I was at least able to get it to exactly the right temperature.