Greetings everyone !
Since Google closed down comp.ach on google groups, I had been using
Real World Technologies as a portal. About 8 weeks ago it crashed for
the first time, then a couple weeks later it crashed a second time, apparently terminally, or Dave Kanter's interest has waned ...
For Usenet you were using i2pn2.org server, probably via www.novabbs.com
web portal created with Rocksolid Light software. The server and portal
were maintained by Retro Guy (Thom). 2025-04-26 Thom passed away from pancreatic cancer. His Usenet server and his portal continued to work
without maintenance until late July. But eventually they stopped.
So it goes.
Greetings everyone !
Since Google closed down comp.ach on google groups, I had been using
Real World Technologies as a portal. About 8 weeks ago it crashed for
the first time, then a couple weeks later it crashed a second time, apparently terminally, or Dave Kanter's interest has waned ...
With help from Terje and SFuld, we have located this portal, and this
is my first attempt at posting here.
Anyone familiar with my comp.arch record over the years, understands
that I participate "a lot"; probably more than it good for my interests,
but it is energy I seem to have on a continuous basis. My unanticipated
down time gave my energy a time to work on stuff that I had been neglect-
ing for quite some time--that is the non-ISA parts of my architecture.
{{I should probably learn something for this down time and my productivity}}
My 66000 ISA is in "pretty good shape" having almost no changes over
the last 6 months, with only specification clarifications. So it was time
to work on the non-ISA parts.
----------------------------
First up was/is the System Binary Interface: which for the most part is
"just like" Application Binary Interaface, except that it uses supervisor call SVC and supervisor return SVR instead of CALL, CALX, and RET. This method gives uniform across the 4 privilege levels {HyperVisor, Host OS, Guest OS, and Application}.
I decided to integrate exception, check, and interrupts with SVC since
they all involve a privilege transfer of control, and a dispatcher.
Instead of having something like <Interrupt> vector table, I decided,
under EricP's tutelage, to use a software dispatcher. This allows the
vector table to be positioned anywhere in memory, on any boundary, and
be of any size; such that each Guest OS, Host OS, HyperVisor can have
their own table organized anyway they please. Control arrives with a re-entrant Thread State and register file at the dispatcher with R0
holding "why" and enough temporary registers for dispatch to perform
its duties immediately.
{It ends up that even Linux "signals" can use this means with very
slight modification to the software dispatcher--it merely has to
be cognizant that signal privilege == thread waiting privilege and
thus "save the non-preserved registers".}
Dispatch extracts R0<38:32>, compares this to the size of the table,
and if it is within the table, CALX's the entry point in the table.
This performs an ABI control transfer to the required "handler".
Upon return, Dispatcher performs SVR to return control whence it came.
The normal path through Dispatcher is 7 instructions.
In My 66000 Architecture, SVR also checks pending interrupts of higher priority than where SVR is going; thus, softIRQ's are popped off the
deferred call list and processed before control is delivered to lower priority levels.
----------------------------
Next up was the System Programming model: I modeled Chip Resources after
PCIe Peripherals. {{I had to use the term Peripheral, because with SR-IOV
and MR-IOV; with physical Functions, virtual Functions, and base Functions and Bus; Device. Function being turned into a routing code--none of those terms made sense and required to many words to describe. So, I use the term Peripheral as anything that performs an I/O service on behalf of system.}}
My 66000 uses nested paging with Application and Guest OS using Level-1 translation while Host OS and HyperVisor using Level-2 translation.
My 66000 translation projects a 64-bit virtual address space into a
66-bit universal address space with {DRAM, Configuration, MM I/O, and
ROM} spaces.
Since My 66000 comes out of reset with the MMU turned on. Boot software assesses virtual Configuration space, which is mapped to {Chip, DRAM,
and PCIe} configuration spaces. Resources are identified by Type 0
PCIe Configuration headers, and programmed the "obvious" way (later) assigning a page of MM I/O address space to/for each Resource.
Chip Configuration headers have the Built-In Self-Test BIST control
port. Chip-resources use BIST to clear and initialize the internal
stores for normal operation. Prior to writing to BIST these resources
can be read using the diagnostic port and dumped as desired. BIST is
assumed to "take some time" so BOOT SW might cause most Chip resources
to BIST while it goes about getting DRAM up and running.
In all cases:: Control Registers exist--it is only whether SW can access
them that is in question. A control registers that does not exist, reads
as 0 and discards any write, while a control register that does exist
absorbs the write, and returns the last write or the last HW update. Configuration control registers are accessible in <physical> configuration space, The BAR registers in particular are used to assign MM I/O addresses
to the rest of the control registers no addressable in configuration space.
Chip resources {Cores, on-Die Interconnect, {L3, DRAM}, {HostBridge,
I/O MMU, PCIe Segmenter}} have the first 32 DoubleWords of the
assigned MM I/O space defined as a "file" containing R0..R31. In all
cases:
R0 contains the Voltage and Frequency control terms of the resource,
R1..R27 contains any general purpose control registers of resource.
R28..R30 contains the debug port,
R31 contains the Performance Counter port.
The remaining 480 DoubleWords are defined by the resource itself
(or not).
Because My 66000 ISA has memory instructions that "touch" multiple
memory locations, these instructions take on special significance
when using the debug and performance counter ports. Single memory instructions access the control registers themselves, while multi-
memory instructions access "through" the port to the registers
the port controls.
For example: each resource has 8 performance counters and 1 control
register (R31) governing that port.
a STB Rd,[R31] writes a selection into the PC selectors
a STD Rd,[R31] writes 8 selections into the PC selectors
a LDB Rd,[R31] reads a selection from a PC selectors
a LDD Rd,[R31] reads 8 selections from the PC selectors
while:
a LDM Rd,Rd+7,[R31] reads 8 Performance Counters,
a STM Rd,Rd+7,[R31] writes 8 Performance Counters,
a MS #0,[R31],#64 clears 8 Performance Counters.
The Diagnostic port provides access to storage within the resource.
R28 is roughly the "address" control register
R29 is roughly the "data" control register
R30 is roughly the "other" control register
For a Core; one can access the following components from this port:
ICache Tag
ICache Data
ICache TLB
DCache Tag
DCache Data
DCache TLB
Level-1 Miss Buffer
L2Cache Tag
L2Cache Data
L2Cache TLB
L2Cache MMU
Level-2 Miss Buffer
Accesses through this port come in single-memory and multi-memory
flavors. Accessing these control registers as single memory actions
allows raw access to the data and associated ECC. Reads tell you
what HW has stored, writes allow SW to write "bad" ECC, should it
so choose. Multi-memory accesses allow SW to read or write cache
line sized chunks. The Core tags are configured so that every line
has a state where this line neither hits nor participates in set
allocation (when a line needs allocated on miss or replacement.)
So, a single bad line in a 16KB cache 4-way set looses 64-bytes
and one line becomes 3-way set associative.
----------------------------
By using the fact that cores come out of reset with MMU turned on,
and BOOT ROM supplying the translation tables, I was able to achieve
that all resources come out of reset with all control register flip-
flops = 0, except for Core[0].Hypervisor_Context.v = 1.
Core[0] I$, D$, and L2$ come out of reset in the "allocated" state,
so Boot SW has a small amount of memory from which to find DRAM,
configure, initialize, tune the pin interface, and clear; so that
one can proceed to walk and configure the PCIe trees of peripherals. ----------------------------
Guest OS can configure its translation tables to emit {Configuration
and MM I/O} space accesses. Now that these are so easy to recognize:
Host OS and HyperVisor have the ability to translate Guest Physical {Configuration and MM I/O} accesses into Universal {Config or MM I/O} accesses. This requires that the PTE KNOW how SR-IOV was set up on
that virtual Peripheral. All we really want is a) the "routing" code
of the physical counterpart of the virtual Function, and b) whether
the access is to be allowed (valid & present). Here, the routing code contains the PCIe physical Segment, whether the access is physical
or virtual, and whether the routing code uses {Bus, Device, *},
{Bus, *, *} or {*, *, *}. The rest is PCIe transport engines.
Anyway: School is back in session !
Mitch
Greetings everyone !
Chip resources {Cores, on-Die Interconnect, {L3, DRAM}, {HostBridge,
I/O MMU, PCIe Segmenter}} have the first 32 DoubleWords of the
assigned MM I/O space defined as a "file" containing R0..R31. In all
cases:
R0 contains the Voltage and Frequency control terms of the resource,
R1..R27 contains any general purpose control registers of resource.
R28..R30 contains the debug port,
R31 contains the Performance Counter port.
The remaining 480 DoubleWords are defined by the resource itself
(or not).
Because My 66000 ISA has memory instructions that "touch" multiple
memory locations, these instructions take on special significance
when using the debug and performance counter ports. Single memory >instructions access the control registers themselves, while multi-
memory instructions access "through" the port to the registers
the port controls.
For example: each resource has 8 performance counters and 1 control
register (R31) governing that port.
a STB Rd,[R31] writes a selection into the PC selectors
a STD Rd,[R31] writes 8 selections into the PC selectors
a LDB Rd,[R31] reads a selection from a PC selectors
a LDD Rd,[R31] reads 8 selections from the PC selectors
while:
a LDM Rd,Rd+7,[R31] reads 8 Performance Counters,
a STM Rd,Rd+7,[R31] writes 8 Performance Counters,
a MS #0,[R31],#64 clears 8 Performance Counters.
The Diagnostic port provides access to storage within the resource.
R28 is roughly the "address" control register
R29 is roughly the "data" control register
R30 is roughly the "other" control register
For a Core; one can access the following components from this port:
ICache Tag
ICache Data
ICache TLB
DCache Tag
DCache Data
DCache TLB
Level-1 Miss Buffer
L2Cache Tag
L2Cache Data
L2Cache TLB
L2Cache MMU
Level-2 Miss Buffer
Accesses through this port come in single-memory and multi-memory
flavors. Accessing these control registers as single memory actions
allows raw access to the data and associated ECC. Reads tell you
what HW has stored, writes allow SW to write "bad" ECC, should it
so choose. Multi-memory accesses allow SW to read or write cache
line sized chunks. The Core tags are configured so that every line
has a state where this line neither hits nor participates in set
allocation (when a line needs allocated on miss or replacement.)
So, a single bad line in a 16KB cache 4-way set looses 64-bytes
and one line becomes 3-way set associative.
----------------------------
By using the fact that cores come out of reset with MMU turned on,
and BOOT ROM supplying the translation tables, I was able to achieve
that all resources come out of reset with all control register flip-
flops = 0, except for Core[0].Hypervisor_Context.v = 1.
Core[0] I$, D$, and L2$ come out of reset in the "allocated" state,
so Boot SW has a small amount of memory from which to find DRAM,
configure, initialize, tune the pin interface, and clear; so that
one can proceed to walk and configure the PCIe trees of peripherals.
----------------------------
Guest OS can configure its translation tables to emit {Configuration
and MM I/O} space accesses. Now that these are so easy to recognize:
Host OS and HyperVisor have the ability to translate Guest Physical >{Configuration and MM I/O} accesses into Universal {Config or MM I/O} >accesses. This requires that the PTE KNOW how SR-IOV was set up on
that virtual Peripheral.
MitchAlsup <user5857@newsgrouper.org.invalid> writes:
Greetings everyone !
Chip resources {Cores, on-Die Interconnect, {L3, DRAM}, {HostBridge,
I/O MMU, PCIe Segmenter}} have the first 32 DoubleWords of the
assigned MM I/O space defined as a "file" containing R0..R31. In all
cases:
R0 contains the Voltage and Frequency control terms of the resource, >R1..R27 contains any general purpose control registers of resource. >R28..R30 contains the debug port,
R31 contains the Performance Counter port.
The remaining 480 DoubleWords are defined by the resource itself
(or not).
I'd allow for regions larger than 4096 bytes. It's not uncommmon
for specialized on-board DMA engines to require 20 bits of
address space to define the complete set of device resources,
even for on-chip devices (A DMA engine may support a large number
of "ring" structures, for example, and one might group the
ring configuration registers into 4k regions (so they can be assigned
to a guest in a SRIOV-type device)).
I've seen devices with dozens of performance registers (both
direct-access and indirect-access).
Because My 66000 ISA has memory instructions that "touch" multiple
memory locations, these instructions take on special significance
when using the debug and performance counter ports. Single memory >instructions access the control registers themselves, while multi-
memory instructions access "through" the port to the registers
the port controls.
That level of indirection may cause difficulties when virtualizing
a device.
For example: each resource has 8 performance counters and 1 control >register (R31) governing that port.
a STB Rd,[R31] writes a selection into the PC selectors
a STD Rd,[R31] writes 8 selections into the PC selectors
a LDB Rd,[R31] reads a selection from a PC selectors
a LDD Rd,[R31] reads 8 selections from the PC selectors
while:
a LDM Rd,Rd+7,[R31] reads 8 Performance Counters,
a STM Rd,Rd+7,[R31] writes 8 Performance Counters,
a MS #0,[R31],#64 clears 8 Performance Counters.
The Diagnostic port provides access to storage within the resource.
R28 is roughly the "address" control register
R29 is roughly the "data" control register
R30 is roughly the "other" control register
For a Core; one can access the following components from this port:
ICache Tag
ICache Data
ICache TLB
DCache Tag
DCache Data
DCache TLB
Level-1 Miss Buffer
L2Cache Tag
L2Cache Data
L2Cache TLB
L2Cache MMU
Level-2 Miss Buffer
Accesses through this port come in single-memory and multi-memory
flavors. Accessing these control registers as single memory actions
allows raw access to the data and associated ECC. Reads tell you
what HW has stored, writes allow SW to write "bad" ECC, should it
so choose. Multi-memory accesses allow SW to read or write cache
line sized chunks. The Core tags are configured so that every line
has a state where this line neither hits nor participates in set
allocation (when a line needs allocated on miss or replacement.)
So, a single bad line in a 16KB cache 4-way set looses 64-bytes
and one line becomes 3-way set associative.
----------------------------
The KISS principle applies.
By using the fact that cores come out of reset with MMU turned on,
and BOOT ROM supplying the translation tables, I was able to achieve
that all resources come out of reset with all control register flip-
flops = 0, except for Core[0].Hypervisor_Context.v = 1.
Where is the ROM? Modern SoCs have an on-board ROM, which
cannot be changed without a re-spin and new tapeout. That
ROM needs to be rock-solid and provide just enough capability
to securely load a trusted blob from a programmable device
(e.g. SPI flash device).
I'm really leary about the idea of starting with MMU enabled,
I don't see any advantage to doing that.
Core[0] I$, D$, and L2$ come out of reset in the "allocated" state,
so Boot SW has a small amount of memory from which to find DRAM,
configure, initialize, tune the pin interface, and clear; so that
one can proceed to walk and configure the PCIe trees of peripherals.
You don't need to configure peripherals before DRAM is initialized
(other than the DRAM controller itself). All other peripheral initialization should be done in loadable firmware or a secure
monitor, hypervisor or bare-metal kernel.
----------------------------
Guest OS can configure its translation tables to emit {Configuration
and MM I/O} space accesses. Now that these are so easy to recognize:
Security. Guest OS should only be able to access resources
granted to it by the HV.
Host OS and HyperVisor have the ability to translate Guest Physical >{Configuration and MM I/O} accesses into Universal {Config or MM I/O} >accesses. This requires that the PTE KNOW how SR-IOV was set up on
that virtual Peripheral.
This seems unnecessarily complicated.
Every SR-IOV capable device
is different and aside the standard PCIe defined configuration space registers, everything else is device-specific.
scott@slp53.sl.home (Scott Lurndal) posted:
Host OS and HyperVisor have the ability to translate Guest Physical
{Configuration and MM I/O} accesses into Universal {Config or MM I/O}
accesses. This requires that the PTE KNOW how SR-IOV was set up on
that virtual Peripheral.
This seems unnecessarily complicated.
So did IEEE 754 in 1982...
What I have done is to virtualize Config and MM I/O spaces, so Guest OS
does not even see that it is not Real OS running on bare metal--and doing
so without HV intervention on any of the Config or MM I/O accesses.
Every SR-IOV capable device
is different and aside the standard PCIe defined configuration space
registers, everything else is device-specific.
Only requires 3 bits in the MM I/O PTE.
Only requires 1 bit in Config PTE, a bit that already had to be there.
On 8/22/2025 11:17 AM, MitchAlsup wrote:
scott@slp53.sl.home (Scott Lurndal) posted:
<snip>
Host OS and HyperVisor have the ability to translate Guest Physical
{Configuration and MM I/O} accesses into Universal {Config or MM I/O}
accesses. This requires that the PTE KNOW how SR-IOV was set up on
that virtual Peripheral.
This seems unnecessarily complicated.
So did IEEE 754 in 1982...
Still is...
Denormals, Inf/NaN, ... tend to accomplish relatively little in
practice; apart from making FPUs more expensive, often slower, and
requiring programmers to go through extra hoops to specify DAZ/FTZ in
cases where they need more performance.
Likewise, +/- 0.5 ULP, accomplishes little beyond adding cost; whereas
+/- 0.63 ULP would be a lot cheaper, and accomplishes nearly the same effect.
Well, apart from the seeming failure of being unable to fully converge
the last few bits of N-R, which seems to depend primarily on sub-ULP bits.
But, there is a tradeoff:
Doing a faster FPU which uses trap-and-emulate.
Still isn't free, as detecting cases tat will require trap-and-emulate
still has a higher cost than merely not bothering in the first place
(and now requires trickery of routing FPSR bits into the instruction
decoder depending on whether they need to be routed in a way that will
allow the FPU to detect violations of IEEE semantics).
And finding some other issues in the process, ...
...
What I have done is to virtualize Config and MM I/O spaces, so Guest OS does not even see that it is not Real OS running on bare metal--and doing so without HV intervention on any of the Config or MM I/O accesses.
Still seems unnecessarily complicated.
Could be like:
Machine/ISR Mode: Bare metal, no MMU.
Supervisor Mode: Full Access, MMU.
User: Limited Access, MMU
VM Guest OS then runs in User Mode. and generates a fault whenever a privileged operation is encountered. The VM can then fake the rest of
the system in software...
And/Or: Ye Olde Interpreter or JIT compiler (sorta like DOSBox and similar).
Nested Translation? Fake it in software.
Unlike real VMs, SW address translation can more easily scale to N
levels of VM, even if this also means N levels of slow...
Every SR-IOV capable device
is different and aside the standard PCIe defined configuration space
registers, everything else is device-specific.
Only requires 3 bits in the MM I/O PTE.
Only requires 1 bit in Config PTE, a bit that already had to be there.
On 8/22/2025 11:17 AM, MitchAlsup wrote:
scott@slp53.sl.home (Scott Lurndal) posted:
<snip>
Host OS and HyperVisor have the ability to translate Guest Physical
{Configuration and MM I/O} accesses into Universal {Config or MM I/O}
accesses. This requires that the PTE KNOW how SR-IOV was set up on
that virtual Peripheral.
This seems unnecessarily complicated.
So did IEEE 754 in 1982...
Still is...
Denormals, Inf/NaN, ... tend to accomplish relatively little in
practice; apart from making FPUs more expensive, often slower, and
requiring programmers to go through extra hoops to specify DAZ/FTZ in
cases where they need more performance.
Likewise, +/- 0.5 ULP, accomplishes little beyond adding cost; whereas
+/- 0.63 ULP would be a lot cheaper, and accomplishes nearly the same effect.
Well, apart from the seeming failure of being unable to fully converge
the last few bits of N-R, which seems to depend primarily on sub-ULP bits.
scott@slp53.sl.home (Scott Lurndal) posted:
MitchAlsup <user5857@newsgrouper.org.invalid> writes:
Greetings everyone !
Because My 66000 ISA has memory instructions that "touch" multiple
memory locations, these instructions take on special significance
when using the debug and performance counter ports. Single memory
instructions access the control registers themselves, while multi-
memory instructions access "through" the port to the registers
the port controls.
That level of indirection may cause difficulties when virtualizing
a device.
These are on-Die resources not PCIe peripherals.
By using the fact that cores come out of reset with MMU turned on,
and BOOT ROM supplying the translation tables, I was able to achieve
that all resources come out of reset with all control register flip-
flops = 0, except for Core[0].Hypervisor_Context.v = 1.
Where is the ROM? Modern SoCs have an on-board ROM, which
cannot be changed without a re-spin and new tapeout. That
ROM needs to be rock-solid and provide just enough capability
to securely load a trusted blob from a programmable device
(e.g. SPI flash device).
ROM is external FLASH in the envisioned implementations.
----------------------------
Guest OS can configure its translation tables to emit {Configuration
and MM I/O} space accesses. Now that these are so easy to recognize:
Security. Guest OS should only be able to access resources
granted to it by the HV.
Yes, Guest physical MM I/O Space is translated by Host MM I/O Translation >tables. Real OS setup its translation tables to emit MM I/O Accesses, so >Guest OS should too, then that Guest Physical is considered Host virtual
and translated and protected again.
As far as I am concerned, Guest OS thinks it has 32 Devices each of which >have 8 Functions all on Bus 0... So, a Guest OS with fewer than 256 Fctns >sees only 1 BUS and can short circuit the virtual Config discovery.
These virtual Guest OS accesses, then, get redistributed to the
Segments and Busses on which the VFs actually reside by Level-2 SW.
Host OS and HyperVisor have the ability to translate Guest Physical
{Configuration and MM I/O} accesses into Universal {Config or MM I/O}
accesses. This requires that the PTE KNOW how SR-IOV was set up on
that virtual Peripheral.
This seems unnecessarily complicated.
So did IEEE 754 in 1982...
What I have done is to virtualize Config and MM I/O spaces, so Guest OS
does not even see that it is not Real OS running on bare metal--and doing
so without HV intervention on any of the Config or MM I/O accesses.
Every SR-IOV capable device
is different and aside the standard PCIe defined configuration space
registers, everything else is device-specific.
Only requires 3 bits in the MM I/O PTE.
Only requires 1 bit in Config PTE, a bit that already had to be there.
BGB wrote:
On 8/22/2025 11:17 AM, MitchAlsup wrote:
scott@slp53.sl.home (Scott Lurndal) posted:
<snip>
Host OS and HyperVisor have the ability to translate Guest Physical
{Configuration and MM I/O} accesses into Universal {Config or MM I/O} >>>> accesses. This requires that the PTE KNOW how SR-IOV was set up on
that virtual Peripheral.
This seems unnecessarily complicated.
So did IEEE 754 in 1982...
Still is...
Denormals, Inf/NaN, ... tend to accomplish relatively little in
practice; apart from making FPUs more expensive, often slower, and requiring programmers to go through extra hoops to specify DAZ/FTZ in cases where they need more performance.
Likewise, +/- 0.5 ULP, accomplishes little beyond adding cost; whereas
+/- 0.63 ULP would be a lot cheaper, and accomplishes nearly the same effect.
Well, apart from the seeming failure of being unable to fully converge
the last few bits of N-R, which seems to depend primarily on sub-ULP bits.
Having spent ~10 years (very much part time!) working on 754 standards I strongly believe you are wrong:
Yes, there are a few small issues, some related to grandfather clauses
that might go away at some point, but the zero/subnorm/normal/inf/nan
setup is not one of them.
Personally I think it would have been a huge win if the original
standard had defined inf/nan a different way:
What we have is Inf == Maximal exponent, all-zero mantissa, while all
other mantissa values indicates a NaN.
For binary FP it is totally up to the CPU vendor how to define Quiet NaN
vs Signalling NaN, most common seems to be to set the top bit in the mantissa.
What we have been missing for 40 years now is a fourth category:
None (or Null/Missing)
This would have simplified all sorts of array/matrix sw where both
errors (NaN) and missing (None) items are possible.
The easiest way to implement it would also make the FPU hardware simpler:
The top two bits of the mantissa define
11 : SNaN
10 : QNaN
01 : None
00 : Inf
The rest of the mantissa bits could then carry any payload you want, including optional debug info for Infinities.
Terje
BGB wrote:
What we have been missing for 40 years now is a fourth category:
None (or Null/Missing)
This would have simplified all sorts of array/matrix sw where both
errors (NaN) and missing (None) items are possible.
| Sysop: | DaiTengu |
|---|---|
| Location: | Appleton, WI |
| Users: | 1,064 |
| Nodes: | 10 (0 / 10) |
| Uptime: | 149:56:42 |
| Calls: | 13,691 |
| Calls today: | 1 |
| Files: | 186,936 |
| D/L today: |
438 files (115M bytes) |
| Messages: | 2,410,967 |