Status: 12 August 2021 - Rack re-assembled and all O350 kit and IR4 now racked and powered. L2 Display cabling completed and front door reinstalled. Onyx 350 with InfiniteReality4 now up, running and complete.
The Onyx 350 was the last of SGI's IRIX / MIPS graphics super computers built with SGI's geometry engine graphics pipeline. It uses the modular Chimera architecture, so can be scaled out/in by adding/removing modules. An Onyx 350 is an Origin 350 with added graphics option. It was available in short and tall rack configurations with either V12 (InfinitePerformance) or InfiniteReality (InfiniteReality4) graphics.
The Onyx 350's successor was the Onyx4 UltimateVision systems, with ATI not SGI graphics.
Having collected numerous "vintage" O350 compute hosts and Numalink'ed then into a single system, I had the good fortune to arrange a swap of various SGI modules and parts for an InfiniteReality4 graphics "brick".
The InfiniteReality4 "brick" is a large 18RU unit, so making space required me to re-rack and re-stack the entire system. In doing this I learnt something about:
- 19" racks (bolts, nuts and screws types),
- SGI Origin 3000 tall (RU 39) "Hour Glass" rack disassembly and reassembly,
- SGI O350 specific cabling
- SGI O350 specific racking parts,
- PDUs (Power Distribution Units),
- Alternate power plug standards and
- Installing into back of rack.
As this is pretty rare setup amougst SGI collectors, I am documenting all the details in case others need to do similar exercise.
Origin / Onyx 3000 Series Tall "Hour Glass" rack
Given that the IR4 is kind of rare and special, I thought it would be nice to put it into similar era SGI rack. The SGI Origin / Onyx 3000 series computers came installed in tall "Hour Glass" (my name due to their shape) racks. A few of these have appeared over the last year at prices below USD $3000, depending on what was installed. Mine was shipped via sea freight from Canada as an empty rack, with only shelving installed. Perfect for me to install all the O350 kit into.
The rack is very heavy (around 250 kg) and large. As I installed this in basement the rack had to be disassembed. Once in pieces I could gets theses down the stairs and reassemble the rack in situ.
First job was to strip down rack to structural frame by removing the: doors, shelves, sliders, panels and plastics:
The roof cover consists of two interlocking panels, which need retaining screws to be removed and then slid apart. The frame is riveted together with 1/4 steel structural rivets. So disassembly requires removal of all the rivets:
And backside of riveted section:
The rack has over 150 rivets. The quickest way to remove the rivets was by:
- Punching Out - the central pin (mandrel) section from front of rivet using a steel punch and hammer, typically five blows (if you cannot punch it out, then drill a bit first...)
- Drilling Out - the rivet body (6mm drill), until it starts to spin around in the hole and so you are not able to drill any more, to reduce body thickness
- Cold Chisel Off - the remaining front of rivet. In most cases this works with two blows.
With so many rivets this takes some time, I disassembled mine over three sessions.
I found the rivet that most closely matched the orginial was "Huck Magna-Lok" 1/4" (6.4mm) Steel Dome Head 2.0 - 9.5 Grip Range Structural Rivets.
As these are "big" structural rivets you need a rivet tool that is able to pull the rivets with enough force. There are a few alternates including:
As I wanted to be able to pull the rack apart again, I choose to re-bolt it together rather than fully rivet it back together.
To bolt it back together I used: 1/4" x 28 TPI x 1/2" (Full Thread) UNF Fine Button Head Socket Screw Stainless Steel Bolts and 1/4" x 28 TPI UNF Hex Flange NYLOC nuts.
NOTE #1: As this is an old rack from USA, all the rivets, bolts and nut sizes are imperial. So UNF is part of the "Unified Thread Standard" for Fine thread which has higher strength than UNC (Coarse) threaded bolts / nuts.
Using bolts / nuts for reassembly is very very fiddly as getting access to back side of sections is extremly difficult in many places (and impossible in some). I used a combination of: thick plastic straw, double sided tape with string / washer to position nut and spanner with blu-tack to hold the nuts in place for bolting. Tighting the bolt then required positioning either spanner, screwdriver or hex key to blindly push nut up against frame to stop it spinning, while tightening the bolt.
The end result:
The back of one of the few easily accessible open sections in rear of rack:
I was able to bolt the rack back together in situ in basement. However there are some closed box sections where the back is inaccessible. The only way to fully re-assemble these sections is by re-riveting. I used manual rivet tool as cost of battery tool was too much for number of rivets remaining.
NOTE #2: Thanks to "Irinikus" on IRIX Network for pointing me in right direction for rivets.
Rack Disassembly & Reassembly
Disassemly and reassembley is a lot of effort, as there are over 150 rivets to remove, optionally replace with bolts/nuts and/or re-rivet.
From pulling it to pieces:
To moving and reassembly in basement. Some example inaccessible box sections, with three holes each are visible in top corners:
Sliders and Rails for InfinteReality4 & O350 Modules
There seems to be a few different variations of rails / slider for the SGI rack:
- Sliding Rails - for O350 Chassis
- 2RU Steel Slider - for general use from SGI
- 4RU Steel Slider - for O3000 Bricks from SGI (not used with O350)
- 2RU Steel Sliders - from other suppliers.
The distance between rack mounting posts for SGI rack is 2 feet 1 2/4 inches (65.5 cm).
I also mounted Lenovo X3650 Server and HP D3700 Disk Array into the rack.
The Lenovo "snap install" sliding rails installed without any problem.
The HP D3700 rail was to long for the rack. Its rail is designed to go between the mounting posts with no allowance for the rail to go between chassis and mounting posts. So the length cannot be fixed by using an extension "adaptor backet". Instead I used a set of the SGI sliders. Using the SGI slider with the HP D3700 results in the HP box being just a fraction too high in the 2RU slot and it impedes the unit above from sliding. So I left 1RU gap between it and rest of SGI modules, otherwise the 1RU gap would just be at the top of rack.
To attach rails and sliders you will need to use rack screws/bolts which come in a variety of sizes.
Rack Screws and Nuts
The SGI rack has IBM style unthreaded round mounting holes. There are three holes per 1RU space. Each hole is around 7mm in diameter and can accomodate any of the "standard" rack screw sizes:
- 10-32 - UTS 3/16" x 32 TPI UNF Screw (Bolt) which is size that most of the SGI racking parts use. To mount equipment into rack you may need slide on rack "clip nut".
- 12-24 - this slighthly larger diameter with 24 TPI UNC thread, so it is courser thread than 10-32. These do not appear to be used for SGI racks.
- M6 - Metric 6 (as in 6mm) and used with "cage nuts" for square hole mounting posts
- M5 - Metric 5 (as in 5mm), I used these and 10-32 screws for front of sliding rail mounting
- 10-24 1/4" - these are UTS course thread (UNC) screw that are used to attach rails to the O350 chassis. This screw type is not used for general racking.
Here are some example SGI rails / sliders and mounting instuctions.
- General purpose 2RU slider, needs 12 x 10-32 screw. Slots into the mounting slot in mounting post and is secured by screws (3 in each mounting post)
- O350 sliding rail, needs 8 x mounting screw and 4 x 2 hole backing plates (barnut in SGI speak) to hold rail on mounting post.
The SGI sliding rail has 3 sections:
- Outer with mounting "claws" which are held onto mounting post within backing plate (barnut),
- Inner Sliding section and
- Rails to screw to O350 chasis (4 x 10-24 1/4 inch on one side and 5 x 1-24 1/4 inch screws on other side)
The lower mounting claws should be in hole two and three of allocated 2RU space. Installing the sliding rail is not easy, as there is no nice "snap in" mechanism. Instead the rail mount has very primitive and sharp "open claw" brackets that need to be attached with seperate bolts and mounting bracket. The rails are held in place by putting the claws between the mounting rail and the bracket (barnut) and tightening the bolts. You can see picture of mounting claw of rail in picture below in "Done" section.
As I did not have the required barnuts, I simply used combination of:
- M6 bolts and rack nuts for the rear installation,
- either M5 bolts with rack nuts or 10-32 bolts with clip nuts for front installation (as front bracket is narrower).
The 10-32 bolt with clip nut is shown here:
The front mount needs smaller diameter bolts than the rear mount.
I had rails with two different slider mechanisms, These are incompatible, with each other, so you need to be carefull not mis-match the rail installed on the O350 chassis.
The sliding rail offers little advantage over the "steel slider" and are substantially harder to mount both into rack and onto chassis. As I had five sets of sliding rails and only four of steel sliders I used combination of both. The Numalink Router needs steel slider.
NOTE #3: If you are in USA, you might be able to get "barnuts" from "Jonathan Engineered Solution" who appear to have a compitable item in their "Mounting Bracket" catalog, item SPO-630.
Mounting into Back of Rack
You would think what with 39RU rack that space would not be a problem. But with the lR4 chassis taking up 18RU, that does not leave a lot of space for other stuff.
As the rack is quite deep I got around this by adding another set of mounting posts into the back of the rack for:
- Top of Rack Switches
- Remote Power Controllers
- Video Breakout Box (VBOB)
- LUCID ADA Audio interface
This is a very simple construction in my case, where I just bolted together the mounting posts from my old rack with some horizontal steel internal bracing (again from old rack) and then slotted it into the back. This is secured with bolts to the rear of the rack (having flattened the steel horizontal brace section to allow the frame to be pulled closer to rear).
The rear mounting frame installed into the back of "Hour Glass" rack and with some kit added:
And the lower section for VBOB and LUCID:
Having back mounting was essential as the x86 servers have over eight ethernet ports each so there is a lot of ethernet cabling that needs to be managed:
NOTE: Sign of evolution is that you can run 10GbE over single twisted pair or optical pair, while "massive" Numalink 3 cables provides 1.6GB/sec uni-directional (around 12.8Gb/sec) or 3.2GB/sec bi-directional. So the "Top of Rack" Netgear 12 port 10GbE switch likely has similar total bandwidth to the 8 port Numalink 3 Router. The lastest current generation of Numalink 8 has around 850 GB/sec peak bandwidth.
Adding Panels and Door Back
Reattaching the side panels:
Power for Rack, InfiniteReality4 and L2
When supplied by SGI an Origin 3000 rack will typically have a Power Distribution Unit (PDU). The PDU is a high amp rate "power board" and in the case of SGI supplied ones come with industrial power plug rather then your typical domestic power plug.
My rack did not come with any PDU, so I did a search for units with the desired number of plugs and which looked like they could be mountable in back of rack. Most the PDU's come as either 19" rack horizontal mount or as virtual mount units.
These did not really have the right form factor to mount in back side of rack. After some search I found HP PDU (E7682) rated at 30 AMP (way more than I needed), which has 20 power outputs.
I got two of these (via eBay) as I wanted to distribute power load across multiple outlets. Using a home made mounting bracket they fit nicely into the back corners of rack:
Simple bracket to hold the PDU in place in back of rack:
The power plug on the HP PDU is IEC 60309 32A Blue variation. As I do not need 32A and do not have 32A compatible power supply, I changed plug to IEC 60309 16A Blue variety which has smaller diameter.
The installed PDU fits nicely in rear corner between the side of rack and extra rear post:
Power for G Brick (InfiniteReality4)
According to SGI specifications the G Brick (InfiniteReality4) has equivalent power consumption to the compute boxes and SGI provide clear instructions to have seperate and dedicated power for this.
Like the original SGI PDU the G Brick has a high amp industrial plug. Though unlike the HP units this is not IEC standard, but a "Hubbell" unit.
The plug is Hubbell HBL2621 (another industrial high ampage type plug):
Initially I thought I would use a Hubbell HBL2623 to IEC 60309 and then IEC 60309 to Australian domestic power cable to get power to the IR4. Simper approach was just be to replace the Hubbell plug with IEC 60309 16A Blue one. This can then be used with adaption/extention cable for Australian 15A plug / power supply. The IR4 should be on seperate power circuit to one that powers the rest of O35o machine, to avoid potential overload.
I also changed the HP PDU as well as IR4 to IEC 60309 Blue 16A plugs. Changing plugs just requires a screwdriver and being careful to re-wire Earth, Live and Neutral connections correctly. The IEC plugs has advantage of being much more readily available than the Hubbell type:
NOTE #4: My IR4 power cable was colour coded based on US "NEC" AC wiring conventions: Green or Geen/Yellow - Ground, Black - Active/Live, White - Netural. This may vary across IR4 chassis's so be sure to check carefully.
NOTE #5: Australian power is 240V AC with domestic power typically using 10A power sockets and optionally 15A power sockets (and corresponding wiring) and uses convention: Green (old) or Green/Yellow - Ground, Red (old) or Brown - Active/Line/Live, Black (old) or Blue - Neutral. The current convention is aligned with IEC. Again check up for latest information before changing anything.
The result is finally seeing signs of life at last from IR4:
Power for L2 Controller
The L2 Controller slots nicely into roof of rack. The O350 is an all AC powered setup (ie no SGI Power Bricks), so needs a power adaptor to provide 48V DC to the L2. This is available via D-Link DSA-0421S-50 Adaptor:
You can create your own L2 compatible plug for this using a HD50 plug and soldering power to pins 1 - 6. The details of plug and wiring are:
Looking face on to female DB50 have: 3 x Rows:
- Top Row: 17 <- 1 (right to left)
- Middle Row: 33 <- 18 (right to left)
- Botton Row: 50 <- 34 (right to left)
PIN ID, is based on stamped numbering on SGI original adaptor.
And measuring SGI original voltage:
- 48 Volts: PIN 1, 3, 5
- Ground: PIN 2, 4, 6
So for DIY wiring of plug:
- PINS 1, 3, 5 - 48V DC
- PINS 2, 3, 6 - Ground
Here is how I wired up my plug, using female socket to avoid cutting the male plug off the adaptor:
O350 and IR4 Special Cabling
In general setting up a large O350 systems once you have Numalink Router is relatively straight forward and well documented in the SGI documentation.
There are a couple of special cable that are not well documented:
- L2 Display - the oval shaped box that hangs from the door and
- IR4 USB Cable - which for some reason is non-standard
NOTE #7: See my other (sprawling) SGI Technical Tips blog for information on Numalink serial number setting.
On Numalink Router the lower four ports support USB via the Numalink cable, while the top four do not. In my setup with seven connected devices you can wire up the system via L2 without the need for extra USB Hub. My O350 has:
- Numalink Router - L1 USB port connected to L2 Controller USB
- 4 Hosts - connect via Numalink only (uses the existing Numalink USB connectivity) using lower four ports on Numalink Router
- 2 Hosts - have L1 USB directly connnected to L2 as they are on upper Numalink ports without USB
- IR4 L1 connected directly to L2 via special USB cable
That uses all four USB ports on the L2 Controller, so if you need more than this then the only answer is to use extra USB hub.
IR4 USB Cable
The IR4 chassis has USB port, but for some reason this has an DB9 plug rather than a standard USB one. It is an AMP cable. So you will need to either solder your own or source one:
L2 Display Cable
The L2 Display cable is another generally un-documented item. It turns out that this is just a standard MD50 - MD50 SCSI cable (also known as "micro HD50"). So finding a compatible cable is relatively easy, but finding one long enough to install into rack and connect the door mounted L2 display is not.
To connect the L2 Display to an L2 Controller in top of rack requires a cable that is 10 feet long. Before parallel SCSI became obselete, there appeared to be three standard lengths: 3 feet, 6 feet and 10 feet. Go for the longest one ;-) .
The cable has to run through from display to channel in door, along side and up to the L2 controller, so full 10 feet is needed:
There are channels and slots in the rack door mount, door and handle to allow you to thread the cable through:
With left panel re-attached and cable exiting via door slot, it is now possible to put the door back on an run cable down the side of door, which has special retaining clips:
Once threaded through the handle slot, the handle can be put back on and L2 Display attached via mounting bracket on the handle:
And the good old L2 Display... all this trouble for an "on" button:
L2 Installed - Check, PDUs installed - Check, Switching Installed - Check, Numalink Installed - Check, First O350 Chassis Added - Check
Not done yet but getting there ...
The L2 installed into roof of rack, illustrates the tightness of fit and reason why using an SGI rack which is specifically designed for the O3000 / O300 / O350 machines has its avantages:
I expect that the IR4 unit will be the last thing to go in, as it needs extra muscle to move to basement.
Now restacked with all of original setup bar the Lucid (in back of rack) and VBOB. Machine Numalink'ed and running:
Final step is to get this (below) into to that (above) and connect them up. Biggest hurdle is moving the thing...
I decided to put VBOB into back of rack and use final 2RU slot for another X86_64 server. I also have have a "classic" MacPro I would like squeeze into the back as well. So its going to be crowded in there.
Moving IR4 was a challenge, due to size and weight and COVID restrictions meant being unable to co-opt help. I decided that best approach would be to get "block and tackle" and use this to lift and lower the chassis into basement. To help reduce the weight I pulled all the boards and fan out of the chassis:
The KTOWN2 and GE14-4 board have compression connectors. So they should be installed with location screws first and only then should the compression connector be tighted. When removing boards loosen the compression connector first and then the location screws. The compression hex (7/64 inch) screws are beneath covers:
With the boards and fan removed the chassis is around 60 kg, so is much easer to move about. I went to tool shop and purchased geared winch and rope to get chassis into basement, without risk of dropping it:
Getting closer to the working configuration ...
Now with mounting shroud and my collection of IR boards installed: 2 x GE14-4, 2 x DG5-2/TVO, 1 x KTOWN2, 1 x RM11:
The IR4's rack sliders interfers with mounting shroud. To fix this I did some simple metal work:
The IR4 has KTOWN2 board to support the graphics interface to the compute module connection. These use "compression" connectors, as per SGI Octane.
The IR4 uses GE14-4 Geometry Engines board. These also have a compression connector.
The IR4 has seperate Raster Memory / Texture Memory Boards and can take four of these in first Pipe and 2 in the second pipe. I have single RM11. These are highly prone to failure. Each board weighs over 2 kg each and I think are the IR4's biggest power consumer:
To use Discreet Inferno, you need DG5-2/TVO (HD-GVO) display board, if you want to do HD quality video. The "TVO" has TDMS video output for VBOB. I will test to see if this can also drive TDMS monitor:
Empty slots should be filled with "baffle" boards:
NOTE: Should you first need to "un-rack" your G-Brick before racking it and need to get access to the front of the chassis, there are screws hidden behind the "InfiniteRealityX" logo, which is just a thin magnetic strip. So to get access to screw, just slide fingernail or blade behind the strip to take it off (thank to recondas on "irix network" for this tip):
And so onto finishing the rack and moving to powering up IR4
InfiniteReality4 now racked with rest of the O350 kit:
To power up the IR4 I changed the Hubbell plug to more standard IEC 60309 16A Blue one and connected this to 15A power via extension cable with IEC 60309 Female Socket. So now racked, stacked and powered (see below for hinx / gfxinfo details):
As I need a longer MD50 - MD50 SCSI cable for the L2 display I will delay putting on the door until cable arrives. To install this through the chassis will likely require removing the left side panel again.
The VBOB and LUCID ADA 8824 AD/DA Interface were mounted in the back mounting frame by using small cantilevered shelf mounted in additional back frame and the middle mounting post. I had to take mounting ears off VBOB and it now just sits across the two shelves. Both VBOB and LUCID are reconnected, with video cabling accessible (see picture above of rear installation).
In preparing to put door back on I popped off all the plastic panels and put them into dishwasher to remove years of accumalated dust. The door is looking much cleaner now:
And with arrival of 10 feet L2 Display cable the final item can be finished. To put the L2 cabling in meant taking the left panel back off. Then threading the SCSI cable wiring through sode of rack. Removing and reassembling the handle. So now I can power up the machine via the "on" button rather than having to telnet into the L2 and powering up from there. A small convenience but puts on nice final touch (literally ;-) ):
There is one small details left... and this is that the HP D3700 disk array sticks out a bit to far and so the door is ajar slightly. So maybe one last small bit of metal grinding... on the door frame, not the HP!
Links & References:
FAR (Bologna Italy) - K33F Hand Tool for Structural Rivets
Huck Fastenings - Manufactor of Magna-Lok Structural Rivets
SGI Onyx 350 Visualisation System User's Guide - The SGI documentation provides details on screw sizes and rack mounting of O35o modules
SGI Origin 350 User's Guide - Provides SGI documentation on rack mounting O350 modules
Rack Screw, Nut and Hole Sizes - Rackmount Solutions have good summary
Unified Thread Stardard - a standard for defining screw/bolt thread sizes
IEC Domestic Power Plugs/Outlets - are defined by "IEC 60320". The outlets on HP PDU are 8 x IEC-C14 and 2 x IEC-C20 Outlets
IEC Industrial Power Plugs/Outlets - are defined by "IEC 60309". The plug on the HP PDU is IEC-60309 (Blue P+N+E, 6h) plug. The colour of plug is part of standard.
Hubbell HBL2621 - is used by IR4/G Brick and original SGI PDU. The female socket for this is a HBL2623 (so search ebay for this type to get conversion cable)
NEC Wiring Convention - National Electrical Code provides US standards for wiring. The official version is "owned" by NFPA. A nice summary covering NEC & IEC and various countries wiring conventions is provided here.
SGI Technical Tips - my notes on running and fixing SGI / IRIX machines with strong "Chimera" machine bias
Sister O350 Unit - this is Australian sister unit to my setup and "SGI Enthausiast" (Andrew) provided many helpful tips in getting my setup sorted. I also sourced the IR4 G Brick from him, so many thanks for that !
Mounting Bracket / Barnut - possible supply from "Jonathan Engineered Solutions", item SPI-630. I submitted an enquiry via Web Form but got no response. Expect quick phone call would do the job ;-)
Numalink - the systems interconnect technology that SGI got from Cray aquisition and now owned by HPE, where it continues to evolve.
SGI Documentation & Datasheets - many of these are locked in servers that require account access. Please download and reshare to avoid loss: Origin 350 Datasheet, Onyx Family Overview Datasheet, InfiniteReality4 Datasheet, Numalink White Paper, Next Generation Graphics Hardware Architecture Presentation & InfiniteReality - A Real-Time Graphics System Paper
Web Archive - Onyx 350 Visualisation System begins (April 2003) and ends (Nov 2005), after which it went to "Remarketed" pages. The last MIPS/SGI built graphics super computer did outlast the Onyx4 (July 2003 - Sept 2005) which was replaced by Itanium/Linux Prism range.
HINV - Onyx 350 with InfiniteReality4
So having put all that effort into getting systems racked, stacked and powered up, it is good to finally have hinv and gfxinfo report. This means this machine is complete for now (I say for now, as it would be nice to have extra RM11 and get second graphics pipe running...).
Here is the gfxinfo report, which is what makes this an Onyx 350 InfiniteReality4 machine rather than just a plain O350:
Graphics board 0 is "KONAD" graphics. Managed (":0.0") 1920x1200 Display has 2 channels 4 GEs (of 4), occmask = 0x0f 4MB external BEF ram, 32bit path 1 RM11 board (of 1) 1/0/0/0 Texture Memory: 1024MB/-/-/- Extra-Large pixel depth 32K cmap TVO option detected brd: 80f61806 3061c04/-/-/- f9191002 ge: 0 14832057 24731057 14231057 rm0: 05032057 35431057 4631057 3/3/3/3 4d31057 3/3/3/3/3/3/3/3 4938057 5/5/5/5/5/5/5/5/5/5 dg: 05532057 5838057 1/1/1/1 5631057 1/1 GE: NIC #: 0000.0073.7831 (family: 0b) Serial #: NDR455 Part #: 030-1398-002 KT: NIC #: 0000.006c.5f9c (family: 0b) Serial #: MTL048 Part #: 030-1593-001 RM0: NIC #: 0000.006c.727a (family: 0b) Serial #: NAJ047 Part #: 030-1651-004 TM0: NIC #: 0000.006c.7058 (family: 0b) Serial #: NAJ072 Part #: 030-1650-004 RM1: No NIC serial number available. TM1: No NIC serial number available. RM2: No NIC serial number available. TM2: No NIC serial number available. RM3: No NIC serial number available. TM3: No NIC serial number available. BP: No NIC serial number available. DG: NIC #: 0000.0097.db2e (family: 0b) Serial #: NSV288 Part #: 030-1242-001 DGOPT:NIC #: 0000.0097.deef (family: 0b) Serial #: NRV947 Part #: 030-1511-004 Input Sync: Voltage - Video Level; Source - Internal; Genlocked - False Channel 0: Origin = (0,0) Video Output: 1920 pixels, 1200 lines, 60.00Hz (1920x1200_6000_DL.vfo) Video Format Flags: (none) Sync Output(s): Composite sync on Green Composite TTL sync on Aux 0 Using Gamma Map 0