Unconditioned Texas Garage Lab – 1 year later(blog.networkprofile.org)
blog.networkprofile.org
Unconditioned Texas Garage Lab – 1 year later
https://blog.networkprofile.org/unconditioned-garage-lab-1-year-later/
29 comments
I ran a bitcoin mining rig in my garage in Houston, Texas (6x Radeon cards across two workstations) for two years in 2010-2012. No issues, no loss of hardware. Ambient temperature went from 40F to 140F.
Wow those were the early years of mining, must have been quite profitable if you kept the yield for a couple of years till the price skyrocketed.
I sold coins along the way to cover initial investment and then exited the majority of my holding at $500-1500. I sold my final bitcoin in 2017 when prices were sky high. If I had sold everything at peak I would have been extremely wealthy. Maybe next time. :)
That is interesting. Do you know if the Radeon cards throttle automatically when they get too hot? Did you have any custom cooling? I can't imagine they would perform all that great at 140F if they throttles themselves, but them doing so may have kept them functional and error free at the expense of hiding poor performance, unless you have data showing otherwise (and you totally might, it seems like the person setting up that system might also check to ensure it's performing to spec :) )
140F/60C is not really out of band for a GPU, usually throttling starts around 75-80C.
Yes, but they said ambient temperature, and I assumed that to mean the temperature inside the garage (I grew up with a detached garage that was not insulated, it got very hot in the summer, especially when it's over 100F outside, so this doesn't seem outside the realm of possibility to me). It would be significantly harder to disperse heat from the CPU and GPU in that environment.
Open system with a box fan probably has better cooling than a mediocre, closed case. Cursory searching says 105-110F isn't too crazy for a closed case
This is unsurprising to me, but that is because I did a few years in Afghanistan servicing edge compute and communications stacks. About half of the hundred or so different sites I was responsible for had zero air conditioning. Dust was a major issue, every unit rotation (~18months) equipment would go to our depot for a full refurbishment.
That generally consisted of a new set of 30m cables/ends for the satellite portion, and a complete strip down and cleaning of everything else.
For field service(weight matters when you have to hike a few miles with gear), I adapted a stupid high velocity 120v leaf blower down to a 1 inch nozzle.
The vast, vast majority of damage was due to operator error or neglect. It's always fun to arrive on site for a malfunctioning terminal, notice the damage, and ask what happened. Then the answer is things like, "so we have this thing called the hundred days of wind, and the terminal on the roof wasn't anchored or using ballast..."
That generally consisted of a new set of 30m cables/ends for the satellite portion, and a complete strip down and cleaning of everything else.
For field service(weight matters when you have to hike a few miles with gear), I adapted a stupid high velocity 120v leaf blower down to a 1 inch nozzle.
The vast, vast majority of damage was due to operator error or neglect. It's always fun to arrive on site for a malfunctioning terminal, notice the damage, and ask what happened. Then the answer is things like, "so we have this thing called the hundred days of wind, and the terminal on the roof wasn't anchored or using ballast..."
I oversee a largish datacenter (>2MW) with only air cooling (a lot of it). Ambient temps range from -23c to +43c. Equipment gets dusty. We have some fan and psu failures, but as long as the air keeps moving, everything keeps working just fine. You'd be surprised at how well plain old airflow works.
Lots of surprised people in the comments. Folks, those parts inside your computer are rated for anywhere from 85C to 200C depending on the part. What this means is that those parts can work at this temperature indefinitely. Perhaps the only part whose lifetime depends on the temperature a lot are electrolytic capacitors, but even they are routinely rated at 105C. Your CPU is likely rated to 105C die temperature, and even then it will only throttle, not die outright. Moreover, letting energy dense parts warm up to a decent temperature improves the efficiency of cooling, because thermal gradient is much steeper. That's why Apple doesn't cool things until they have to. DIYers are obsessed with getting to the point where their CPU runs at ridiculously low temps, but the CPU itself is designed to tolerate much more. GPUs begin to throttle at something like 83C, and NVIDIA deliberately lets them to get to that temperature before fans kick in in earnest.
Electronics tend to be far more robust than people think. As long as it's not literally in an oven and there's nothing conductive getting involved (as in the steel mill example pointed out in another comment) most electronics will chug along just fine. Recognizing this is what led to me abandoning a case for my desktop PC. It's now just slapped on a piece of plywood on my wall because it literally doesn't matter.
This setup even fell (during wall mounting attempts) ~8 feet to hardwood floors...twice! And still going just fine running Half-Life: Alyx :)
This setup even fell (during wall mounting attempts) ~8 feet to hardwood floors...twice! And still going just fine running Half-Life: Alyx :)
This is the default last mile environment for at least a couple ISPs that I've worked with. I've seen switches and other networking equipment in non air conditioned closets in the Orlando heat. The equipment doesn't seem to mind the higher temp that much.
I'm no expert at this whatsoever but from what I've gleaned looking around, a good heuristic is that removing hot air is way better than pushing cold air in.
As a kind of hand-waivy argument, I think of the difference between averaging cool air with how hot it gets ("pushing" air in) vs. removing and replacing with ambient temperature air. In the case of 'pushing' air in, the average of something like 40deg F with 300deg F is 170deg F. Even if ambient is 70deg F, by replacing air with ambient, that's a 2x better cooling, not to mention the energy saving from just using a fan compared with cooling with an air conditioner or some other equipment.
As a kind of hand-waivy argument, I think of the difference between averaging cool air with how hot it gets ("pushing" air in) vs. removing and replacing with ambient temperature air. In the case of 'pushing' air in, the average of something like 40deg F with 300deg F is 170deg F. Even if ambient is 70deg F, by replacing air with ambient, that's a 2x better cooling, not to mention the energy saving from just using a fan compared with cooling with an air conditioner or some other equipment.
I manage a large datacenter of air cooled fanless (radiator) GPUs housed in cases where the case fans are on the hot side and pulling the air out. Hot air expands, which fills the space (more pressure) and thus prevents cooler air from getting in (you have to push harder to get it in). The faster you can get rid of hot air from the enclosure, the better.
One "fun" anecdote that I like to tell everyone. I was given a 1U server by a company I was working at. Some old ~2010 machine with 32GB of RAM and 2 CPUs. I anticipated noise but not quite that much, so I decided to move it to the exterior shed that had one outlet. This was in July in Canada so peeks of 33-34 celsius are expected but not much more than that.
The shed was in the sun and would get pretty hot for humans, but I thought it would be fun to try it and worst case scenario it wouldn't last long. Turns out we had some intense heat for the entire week and the server was dead about 8 days after putting it in the shed.
I opened it to see what had happened and several capacitors had popped.
TLDR: Intense heat isn't good for 1U servers, but I bet you knew that already.
The shed was in the sun and would get pretty hot for humans, but I thought it would be fun to try it and worst case scenario it wouldn't last long. Turns out we had some intense heat for the entire week and the server was dead about 8 days after putting it in the shed.
I opened it to see what had happened and several capacitors had popped.
TLDR: Intense heat isn't good for 1U servers, but I bet you knew that already.
Others point out these arent terrible conditions, all things considered. I'd like to see reports of this quality for stuff in wood shops, chicken barns, cement plants, etc.
I've seen a few examples of things like that and they were entertaining in a "wow it kept on going anyway" sense.
I've seen a few examples of things like that and they were entertaining in a "wow it kept on going anyway" sense.
I can tell you that in a steel mill, its a whole different story, largely because of the ferrous dust, the heat as you can imagine is also a problem, everything has to be sealed and fed through a filtered vents, or it will die
I live in similar conditions, not quite as humid. Tempted to move my mini rack out to the garage now. Very tempting to save on the air conditioning necessitated by the equipment in my rack. May even put my virtualization server out there for awhile and see what the temps get up to.
One year isn’t enough to test reduced MTBF through increased ambient temperatures. Dust and the consequence of insulating electronic components can take years to surface problems.
Sadly I can't give a longer timeframe, as I've only had this 1 year
But, everyone told me it wouldn't last a year
But, everyone told me it wouldn't last a year
What's the normal lifespan of this equipment? I'd imagine this environment, particularly with the dust, would cause failure in the 5-10 year range rather than the 1-2 year range.
My initial instinct was that the extreme temps of summer/winter would be the bigger lifespan shortener, but I suppose dust would start killing fans and clogging air intakes compounding the temperature problems.
This is pretty much it. Constant temps are more important than lower temps. Within reason of course.
Modern equipment is pretty tolerant to high intake temps, due to lots of very poor datacenter designs. It's pretty common to see servers in older co-location facilities at the top of racks seeing 95F intake temps if the on-site staff don't really care about rack blanks and the like. These can go unnoticed for years and simply chug away with their fans at high speed.
Modern equipment is pretty tolerant to high intake temps, due to lots of very poor datacenter designs. It's pretty common to see servers in older co-location facilities at the top of racks seeing 95F intake temps if the on-site staff don't really care about rack blanks and the like. These can go unnoticed for years and simply chug away with their fans at high speed.
The thing is though, there are no temperature problems
All equipment is within the operating specs
All equipment is within the operating specs
Start using a significant portion of the switching backplane capacity and see how well it does then though. That would be a much more interesting result, whether the device succeeds or fails.