2023-05-31 08:37:20 Had a slight error in the above - this is corrected: 2023-05-31 08:37:22 https://imgur.com/a/492dPzd 2023-05-31 09:30:46 You know, that nice clean beautiful builds> does> solution I worked up a couple of weeks ago? 2023-05-31 09:30:49 It's no good. 2023-05-31 09:31:25 builds> leaves an item on the stack. 2023-05-31 09:31:53 Well, at least I think that's the problem. 2023-05-31 09:33:05 Anyway, it's got some sort of bug. Looking into it now. 2023-05-31 09:33:23 Interferes with using a passed in parameter when the defining word is used. 2023-05-31 09:33:47 It leaves an address on the stack, which does> patches. 2023-05-31 09:38:37 KipIngram: I slept on our chat yesterday about the block-allocated dictionary where you shared the idea of encoding a jump to the next block if a definition hits the end of a block. I can see how that would work for executable definitions, but what about definitions that call `allot`? Maybe re-define `allot` to allocate the next block and move the definition there if it computes there's not enough space left. Seems like that would work, but t 2023-05-31 09:38:37 definitions made up of `,` would have a hard time since `,` only appends one cell at a time... Any suggestions? 2023-05-31 09:38:40 Anyway, that's unacceptable. 2023-05-31 09:39:20 Yes, that is an issue. Any code you write will have to either be able to get continuous memory or know how to deal with the jumps. 2023-05-31 09:39:24 No way around that. 2023-05-31 09:39:36 If something needs to allot a big array, you have to give it the memory to do that. 2023-05-31 09:39:56 It's just not something I need very commonly, but stuff I plan to do in the future will need it. 2023-05-31 09:41:22 Maybe I could make all words that write to the dictionary detect when they're at the end of a block, and then copy the current definition to the next block. Maybe typing all that out was the process i need to go through to get to that conclusion. 2023-05-31 09:42:05 Sure. You could do that - a definition that runs into the end could copy its partial progress to a new block. 2023-05-31 09:46:25 Anyway, the code between builds> and does> expects to be able to process stack parameters and does not expect anything to be in the way. 2023-05-31 09:46:36 So I'm going to have to work on that a little. 2023-05-31 09:49:14 Also, I seem to have a bug in my output handling when base is not set to 10. 2023-05-31 09:49:39 do not set base to not 10 WONTFIX RESOLVED 2023-05-31 09:49:41 Things misbehave once I've typed more than 10 characters into EXPECT. 2023-05-31 10:33:37 I believe this works: 2023-05-31 10:33:55 https://pastebin.com/B0KzMMee 2023-05-31 10:34:47 It encodes the instruction in little endian form - if you stored the number at some location, those three bytes would get set right. 2023-05-31 10:35:17 So here ! 3 allot would compile it. 2023-05-31 10:36:22 Please excuse the rot in the builds> clause - that's a work-around for my problem discussed earlier. builds> has actually left TWO items on the stack that does> will do something with, so I have to dodge those at the moment. 2023-05-31 10:36:32 Totally unacceptable. 2023-05-31 10:37:15 It was slightly annoying to need both . 2023-05-31 10:37:28 But the most significant bit of the desitnation register has to move to the right. 2023-05-31 10:37:50 And unfortunately << doesn't work with negative shift counts. :-) 2023-05-31 10:38:16 Anyway, I regard that as a nice starting point for assembly. 2023-05-31 10:43:22 If I recall correctly from my past work of this type, mov is entirely different and required separate treatment. 2023-05-31 10:43:41 Though it may turn out that it's an easy modification of one of these. 2023-05-31 10:44:09 That's the thing - I think I'm not going to have these words actually compile the code - I'll do that separately, because some of these results could be foundations for further instructions. 2023-05-31 10:45:10 Or something like that - maybe it will just mean factoring those starting points into (rrop) word and have : rrop (rrop) ; 2023-05-31 10:45:24 then 2023-05-31 10:45:35 : something-else (rrop) ; 2023-05-31 10:48:56 OR it might make more sense to modify in ram. 2023-05-31 10:49:09 : something-else rrop ; 2023-05-31 10:49:55 Anyway, these are all correct, per the defuse site: 2023-05-31 10:49:57 rax rax rradd C00148 ok 2023-05-31 10:49:59 rax rax rror C00948 ok 2023-05-31 10:50:01 rax rax rradc C01148 ok 2023-05-31 10:50:03 rax rax rrsbb C01948 ok 2023-05-31 10:50:06 rax rax rrand C02148 ok 2023-05-31 10:50:07 rax rax rrsub C02948 ok 2023-05-31 10:50:09 rax rax rrxor C03148 ok 2023-05-31 10:50:11 rax rax rrcmp C03948 ok 2023-05-31 10:54:34 A similar test starting with rax rax rradd and walking the destination register through the whole set is right. 2023-05-31 10:56:39 is rrxor spoken with a trilled r? 2023-05-31 10:58:29 If you please. 2023-05-31 10:58:41 There's a bug with the source reg delta. Chasing it down now. 2023-05-31 10:59:27 I'm moving the high order bit of the source inccorectly. 2023-05-31 11:06:18 Oh, of course. It actually only needs to move one bit. 2023-05-31 11:06:27 Fixed - pastebin is corrected now. 2023-05-31 11:06:56 So, that works along all the "axes" of the situation - I'm not sure if there's any easy way to explicitly check the entire output. 2023-05-31 11:07:27 Guess I could write a python script that generated the whole matrix of instructions and a Forth program to do the same and compare. 2023-05-31 11:12:57 Now I've got to figure out what to do about builds> polluting the stack. 2023-05-31 11:13:10 And scratch my head over why I didn't notice that in the first place. 2023-05-31 11:13:44 That was definitely a "forest, trees" kind of slip up. 2023-05-31 11:32:56 Ok, one of the items is easy to move completely into does> - stacking it in builds> just avoide a little stack noodling. 2023-05-31 11:33:40 The other one is the address of the 2nd pfa field which needs to get patched in does>. I guess I need to just bite the bullet and recover it from the dictionary. More verbose - it was so easy to get back there in builds>. 2023-05-31 11:33:48 But the problem has to be solved. 2023-05-31 11:34:35 Instead of a dp.h @ in builds> I'll need curr @ @ 8- in does> 2023-05-31 11:34:41 So, not "awful." 2023-05-31 12:20:22 I'm looking into op reg [reg] now. 2023-05-31 12:21:37 This has some conditional behavior in it. It looks like an easy step whenever the source register isn't rsp or rbp or their higher register partners, r12 and r13. For those four source cases, though, there is a pattern deviation. 2023-05-31 12:21:53 Apparently you can't encode [rbp] without at least a one-byte offset. 2023-05-31 12:21:59 Or [r13]. 2023-05-31 12:22:34 I had actually noticed that in earlier work when trying to minimize the length of next and so on. 2023-05-31 12:25:21 It would be interesting to know why that is - maybe they had already used those particular bit patterns in some other historical context, so they weren't "available." Which means that special case logic probably appears in the processor microcode too. 2023-05-31 12:26:50 The nice thing, though, is that the bit patterns for registers that appear in op reg, reg also appear in op reg, [reg] - it's just that another byte gets added and another bit set for those special cases. 2023-05-31 12:27:21 So I'll need to check the source reg, and if it's rsp/r12 do a special case and if it's rbp/r13 do a special case. 2023-05-31 12:27:56 But those "special cases" just look like "additions" - I'll still have a base logic case to start with. 2023-05-31 12:46:03 I'm thinking along the lines of rrop []rop r[]op [i]rop r[i]op to get various addressing modes. 2023-05-31 12:50:08 I'm hoping all of those aren't unreachably far from what I've already got. Then I'll need mov instructions and jumps, and that will start to be something I might call an assembler. 2023-05-31 13:55:31 Another possibility is something like s[] d[] s[i] d[i] that comes after a standard reg reg op and makes the right modifications. 2023-05-31 13:56:07 "source indirect" possibly with indexing, etc. 2023-05-31 13:56:51 It might be useful to have the rrop words save the source and destination register specs for possible later consultation during modification work. 2023-05-31 14:11:52 Anyway, for example, I might windup writing add rcx, [r13+0x08] as rcx r13 add 8 [s+] 2023-05-31 14:12:40 So [s] [d] [s+] [d+] 2023-05-31 14:13:07 And if I can get all of THAT working for the binary operations and moves, then jumps will probably be the only thing more I need. 2023-05-31 14:18:35 Except for "special case odds and ends," like rep movsq and cld and things like that. 2023-05-31 14:20:08 I hate it when cld gets set wrong 2023-05-31 14:20:21 :-) 2023-05-31 14:20:39 My philosophy on that is to always have it clear - if I need to set it, I make sure and clear it back w hen I'm done. 2023-05-31 14:26:36 Oh, and if I use those "post" instructions to adjust addressing mode, then I won't need "rr" at the beginning of all the ops. 2023-05-31 14:26:52 I had that because I expected I might have other modes called out in similar words. 2023-05-31 14:27:18 reg reg op 2023-05-31 14:30:22 Oh, that still leaves immediate operations. 2023-05-31 14:30:46 Probably just will previs an i for that. 2023-05-31 14:31:29 Guess I could have the register words bump a counter or something. 2023-05-31 14:34:03 So I knew at op time whether one or two registers had been called out. 2023-05-31 14:35:13 Or three, I guess - there are cases like mov rax [rbp+8*rdx] etc. 2023-05-31 14:35:56 I use those very rarely, though - I might decide just to allow a few corner cases that I code by hand. 2023-05-31 14:36:18 Really just want a tool for "the bulk" of the coding. 2023-05-31 14:46:23 Looks like mov reg, reg is just or reg, reg with one additional bit set. 2023-05-31 14:49:07 xchg reg, reg looks to be a two-byte instruction. 2023-05-31 14:50:13 But looks to me like I can extend the op code to four bits and pick up mov right along with the others, depending on how things go with the other addressing modes. 2023-05-31 15:53:51 Well, I added that extra op bit and now have reg/reg moves and reg/reg "test" instruction "in the fold." 2023-05-31 15:54:33 I can't think of any other operations that might fit that same pattern. Anyone think of any beyond add or adc sbb and sub xor cmp mov test? 2023-05-31 15:56:20 do you mean other instructions that feature reg,reg operands? 2023-05-31 15:57:07 Yes. 2023-05-31 15:57:13 Other "major" ones particularly. 2023-05-31 15:57:44 The instructions in the set I listed there appear to be all the same bit patterns, except specifically for the operator bits. 2023-05-31 15:58:01 I.e., the register bits get processed the same way for all of them as far as I can tell. 2023-05-31 15:58:25 I honstly think the indirect register addressing modes are within reach too. 2023-05-31 15:58:55 you've got them all that i'm aware of for 8086 2023-05-31 15:58:57 xchg reg, reg exist, but it's "different." 2023-05-31 15:59:03 Doesn't fit the pattern. 2023-05-31 15:59:18 Yeah, I can't think of any others either. 2023-05-31 15:59:38 there are two variants of mov reg,reg though, the one you'd expect and then there's the seg,reg type 2023-05-31 16:00:17 I'll do a register axis check for mov. I may get surprised. 2023-05-31 16:00:54 That is, I'll start with mov rax, rax and then run first the source register through all the possibilities, and then the dest register. 2023-05-31 16:01:11 But not all the "exploded" possibilities. 2023-05-31 16:02:19 Next step will be to try to pick up "register only" indirect addressing. I think that looks easy for 12 of the 16 registers, and the other four will have to get special handling, on top of the standard processing. 2023-05-31 16:02:35 And I haven't really looked into register indirect with offset yet. 2023-05-31 16:05:03 It does look like, though, that first running those four register cases (the indirect exceptions) through the same as the others and then tweaking the result is probably the right way to go. 2023-05-31 16:05:48 what are you using a reference for the instruction encoding format? 2023-05-31 16:05:57 https://defuse.ca/online-x86-assembler.htm#disassembly2 2023-05-31 16:06:05 Just punching cases in and running them. 2023-05-31 16:06:10 Staring at the result. 2023-05-31 16:06:28 I've found no "instruction manual" for this at all - I'm just digesting it and trying to see what I can see. 2023-05-31 16:06:55 I'm already happy with what I believe to be working now - if I can get more, all the better. 2023-05-31 16:07:33 Oh, btw, I see how to fix builds> does>. 2023-05-31 16:07:47 It's not "hard" - just uglies my code a little. 2023-05-31 16:08:02 And I was so pleased with those nice short little definitions. :-( 2023-05-31 16:09:24 KipIngram: intel and amd have programmer manuals 2023-05-31 16:09:26 might help 2023-05-31 16:10:34 i also had a hard time finding an encoding reference for x86 that's easy to digest, 8086 upto 80286 is consistent enough, but the additions that came with 80386 (like SIB byte) made things even more complicated 2023-05-31 16:10:52 alternatively, using something like ghidra or rizin for doing reverse engineering to see how it looks under the hood 2023-05-31 16:13:26 i have an 8086/186/286 assembler i wrote in shell script that sums up those encodings as best as i could tell from pages and processor manuals i found online. i definitely could have documented the code better, but i factored out the encoding patterns by their types at least 2023-05-31 16:14:45 Well, I do find this kind of work fun. 2023-05-31 16:15:01 So hopefully I'll be able to produce something here you guys might find useful. 2023-05-31 16:15:08 I'll certainly share it all. 2023-05-31 16:22:50 if it helps you identify other low hanging encoding patterns -> https://github.com/jhswartz/xfasm/blob/master/arch/x86/xfasm-8086 (code) and https://github.com/jhswartz/xfasm/blob/master/doc/xfasm-8086.txt (doc) <- these instruction mnemonics take the pattern [instruction-type][transfer-size][source-operand-type][dest-operand-type] and operands are expected as SOURCE DESTINATION - so 'andbir 47 al' amounts to: AND bytes { Immediate (47) -> Register 2023-05-31 16:43:51 for a while, i've been interested in the idea of an assembly lanaguage written entirely in a macro processor. a set of macros that which are named according to generic instructions that you'd find in some form on most/all micro{controllers,processors} and then implemented to expand to actual assembly for the respective instructions on each target of interest 2023-05-31 16:46:10 it wouldn't be possible to define everything in a portable way, but the idea would be to minimise the platform-specificness as much as possible. and in some cases mimic CISC operations with compound instructions in RISC instruction sets 2023-05-31 16:49:57 using the C preprocessor as an example: amd64.h might have '#define MOVAR(address, register) lea address, register;' while arm.h might have '#define MOVAR(address, register) ldr register, =address;' 2023-05-31 16:51:01 assuming you had something like symlink called target.h that pointed at the target arch definition of interest. doesn't have to be CPP + GAS, but that's what i tried this idea out with once. 2023-05-31 16:55:14 my go-to reference is https://www.felixcloutier.com/x86/ 2023-05-31 16:56:38 "Low hanging" really says it all - I doubt you could even pay me to attempt a full x64 instruction set tool. 2023-05-31 16:56:48 I mean, I guess you could, but it would take an awful lot. 2023-05-31 16:57:32 some of the intel PDFs ran to 6,000+ pages, last I looked 2023-05-31 16:57:52 I know how I plan to approach jumps, so really the main remaining goal is gathering up more addressing modes. And immediates, but I don't expect them to be particularly painful. 2023-05-31 16:58:08 Oh yeah - those things are monsters. 2023-05-31 16:59:51 I guess there are the 16 and 8 bit operations to look at, but I don't really need full coverage on those. 2023-05-31 17:10:44 I'm proposing to myself at the moment that destination indirection just turns off two bits in the instruction, and source indirection turns those two bits off and sets another one. 2023-05-31 17:11:09 So I'll try having d[] and s[] do those things. 2023-05-31 17:13:01 Looks like dest indirect with an offset turns one of the two bits back on and adds the index byte (for small indices, which is all Forth really needs). 2023-05-31 17:13:51 With bigger indices it turns the OTHER of those two bits back on and adds a 32-bit index. 2023-05-31 17:16:52 This is looking pretty clean - if it holds. 2023-05-31 17:24:17 why not just make bit masks? 2023-05-31 17:24:58 I did that for a MIPS disassembler. very easy to wrangle and curate in a spreadsheet which you can also use to generate code that you copy and paste in 2023-05-31 17:26:09 So you get something like "1011aaaaa0111ssssssrrrrrr01" which is both the mask and what bits to extract for registers or addresses 2023-05-31 17:26:21 bless you 2023-05-31 17:48:09 I want something that runs within my Forth. No spreadsheet in sight. 2023-05-31 17:48:24 You mean for my studying? 2023-05-31 17:48:37 I guess I could, but this process is comfortable enough. 2023-05-31 17:48:51 There's something a bit pleasing about just "gray mattering" it. 2023-05-31 17:51:33 No I mean when something like X86 is too complicated to cleanly decode, you can save yourself time by using the spreadsheet to document the encoding which solves the complexity problem then use string handling within the spreadsheet to formulate code that is directly pasted into your program 2023-05-31 17:52:07 makes this whole tangled mess very easy to deal with. creating any kind of IF or CASE structure instead of that would be awful 2023-05-31 17:53:09 kind of like a huge state machine. it's a nightmare trying to keep it all straight and make adjustments when you have 500 nested if statements 2023-05-31 18:24:18 I kind of see what you mean, but probably not as well as you do, since I haven't actually done much of that. I tend to do things like this mostly just in my head. 2023-05-31 18:24:32 I couldn't always actually tell you how I arrive at solutions. 2023-05-31 18:24:48 I just let stuff slurp around in there and eventually figure something out. 2023-05-31 18:24:58 It's how I designed digital circuits back in the day, too. 2023-05-31 18:25:58 I always felt particularly well able to see what WASN'T going to work, and just mentally close off those paths. 2023-05-31 18:26:21 Eventually some path would lead all the way to the goal, and then I'd start drawing that up. 2023-05-31 19:58:12 https://github.com/ratfactor/meow5 might be mildly interesting to folks here 2023-05-31 20:06:05 its a take on a simpler forth implementation meant for running on newer machines 2023-05-31 20:06:13 1+ for generating binaries without an external linker 2023-05-31 20:06:17 since hardware has significantly improved in capabilities 2023-05-31 20:06:25 unjust: yes. 2023-05-31 20:06:58 is it your project? 2023-05-31 20:07:02 no. 2023-05-31 20:07:03 i wish. 2023-05-31 20:07:07 i'm still working on my thing 2023-05-31 20:08:13 being able to generate binaries without needing a linker is a boon 2023-05-31 20:13:46 it'll be interesting to see if he embeds the interpreter in every binary, or turns it into a dynamic linker and references it in every generated binary's .interp section 2023-05-31 22:18:57 This is weird. 2023-05-31 22:19:24 If the destination register is an indirect reference, then the sorce and destination registers encode the same as with reg/reg ops. 2023-05-31 22:19:44 But if the SOURCE is indirect, the source and dest register fields get SWAPPED in the emitted code. 2023-05-31 22:19:51 They just swap places. 2023-05-31 22:20:13 I can kind of see how that might contribute to efficient hardware, back when they actually built hardware to run the instructions directly. 2023-05-31 22:20:38 It gets the "memory address" into the same spot either way, and some other bit controls whether you're reading or writing. 2023-05-31 22:21:27 I.e., you have one spot in the logic where the RAM address is going to come from, regardless of which direction data is moving. 2023-05-31 22:21:45 So now I'm thinking of a syntax like this: 2023-05-31 22:21:52 reg reg op 2023-05-31 22:21:56 reg reg s[] op 2023-05-31 22:22:00 reg reg d[] op 2023-05-31 22:22:02 etc. 2023-05-31 22:22:17 s[] would among other things SWAP THE TWO REG ARGS. 2023-05-31 22:22:26 And also set a mode that op would be able to consult. 2023-05-31 22:22:42 And then have 2023-05-31 22:22:47 reg reg s[i] op 2023-05-31 22:22:52 reg reg d[i] op 2023-05-31 22:22:54 for indexed. 2023-05-31 22:23:30 I guess the index would go right before s[i] or d[i] 2023-05-31 22:24:37 Just swapping those arguments would totally take care of that wrinkle - the bits would land in the right places. 2023-05-31 22:27:18 If I'm going to be setting modes, then I think when each instruction is generated it should "wipe a slate clean." Then each register word executed will bump a counter as well as put that reg's value on the stack. 2023-05-31 22:27:40 That way I know how many register args I've seen, and can use that to tell the difference between reg/reg, reg/imm, and so on. 2023-05-31 22:33:31 Those special case regs will need for their reg words to set some flags too. 2023-05-31 22:33:46 I think I've just about got my head around the whole problem now. 2023-05-31 23:51:53 the nice thing about this register swapping is that it means it is ALWAYS the register spec deeper in the stack that drives whether we have a special case register or not. 2023-05-31 23:52:05 That's the "ram register." 2023-05-31 23:52:41 I *think* I have it working for the indirect register memory registers too, EXCEPT for those special cases. 2023-05-31 23:52:56 Things like sub rax, [rax] are working. 2023-05-31 23:53:11 Along with everything that was working earlier. 2023-05-31 23:56:43 : s[] swap ram 1+! x:c00200 frame toggle ; 2023-05-31 23:56:44 : d[] ram 1+! x:c00000 frame toggle ;