My £60 ARM server

May 8, 2012 — 36 Comments

I’ve previously blogged about my belief that ARM servers are a big part of the future, and that server manufacturers are missing a trick by not testing the market with small, cheap, ARM servers, but I wondered if maybe they knew something I didn’t.

To find out what I was missing, I bought the closest thing I could find to a generic ARM server, the Beaglebone, an ARM development board from Texas Instruments, with a TI AM3358 ARM Cortex-A8-based microprocessor, 256MB of RAM, 100Mb ethernet, a couple of USB ports, a micro-SD card slot and not much else.

I’ve had the beaglebone for a few months now, and done various things like installed Ubuntu on it, put it in a Beaglebone custom case so it’s not a bare board anymore, run nodejs services, a VPN to get home network access from my phone and laptop, and generally used it as if it were a normal server, without any real issues.

So, now it’s time to take it to the next level – I’ve paid for the beaglebone to go into a colocation rack in Telehouse North, with a friendly colocation company called Jump Networks who were happy to help out with the experiment, and who only charge for £50 + VAT for installation and a very low monthly cost for hosting the equipment, as little as £5 per month – perfect for an ARM server.

And now, my tiny ARM server is live, running on the Internet, doing real stuff for my work, while we learn the limitations and issues of running an ARM server in the wild.

So far, I’ve found out the following things:

  • While USB is nice for a console at home, servers really need Ethernet or Serial consoles
  • 256MB of RAM is enough, but not plenty
  • Micro-SD card performance is highly variable, and generally rubbish
  • If you measure the power usage of a Beaglebone, you might not get a good answer – it’s too low for most meters!


    • Of these, only the third one has been a real issue for far – I’ve already had one SD card go bad while running the bone at home. If it happens again I’ll need to send a new pre-configured SD card to insert into the slot. The lack of a serial console hasn’t been a problem yet, but I expect it to be at the most awkward moment possible…

    All 3 problems are solvable, for a price, the Beagleboard XM for example has a serial console, and 512MB of RAM, but costs £125 as it comes with lots of other features a server doesn’t need, like DVI-D and S-Video ports, along with a slightly faster 1GHz CPU. While the price isn’t “high”, it’s twice the price of the Beaglebone.

    So for now, the Beaglebone will run day to day, doing real world things, and perhaps I’ll find out why server manufacturers aren’t making these things, and why hosting companies aren’t buying them in bulk yet. In the mean time, if you’ve got any questions about the Beaglebone, let me know and I’ll answer them the best I can.

    Update – I’ve now configured the Beaglebone to run WordPress with nginx caching and done some pretty successful load testing on it.

    In tech
    permalink
    • AsselinPaul

      Raspberry Pi, here we come. Not a lot of RAM on that too either :(
      Beowulf cluster would be a good idea (evil laugh). 

      • http://www.ewanleith.com/ Ewan Leith

        I know you’re half joking, but a custom case designed to hold 50 of these in a 2U space would be as dense as any virtualised system

    • Allan Pinto

      though i guess not arm, you might like the upcoming apc via board, also what about the raspberry pi in similar circumstance -> apc link
      http://apc.io/

      • http://www.ewanleith.com/ Ewan Leith

        Yeah I think the APC board looks really nice, it’s great to see people are showing in an interest in them. I do think VIA should’ve put it in a case though, bare boards are nice to look at, but not so nice for damage from static electricity..

    • essiccf37

      I’m looking forward to your conclusion !
      Nice experiments man

    • Edward Blackburn

      Microsoft is concentrating on an ARM build for Windows, presumably for tablets. An ARM build for Windows server might be mandatory if commodity servers and OSes flood the market. Exciting innovations.

    • nick a name

      Re issue #3

      Did you try upgrading to a class 4 or higher SD card? From what I know, the default speed for these devices is “class 2″. I hit the same problem with my smartphone and (mp3s skipping and such) and these were gone with a class 6 inserted.

      • http://www.ewanleith.com/ Ewan Leith

        I did yes, I had a Samsung 8GB Class 10 Micro SDHC which failed horribly shortly before I shipped it to the colocation, and replaced it with a class 6 card, which is currently running ok.

        The disk throughput is decent, but lots of little writes can cause it to crawl.

        • Phil

          IIRC The class 6 and up cards are optimised for streaming writes: they’re sold for use in cameras and camcorders. If your use case doesn’t include large streaming writes then there’s no point buying a card with a high class number.

          I’ve seen some reports that class 2/4 cards actually perform better on random read/write benchmarks but I’ve never put this to the test myself.

          • http://twitter.com/bestham Bjørn E. Hesthamar

            I think you should consider a SLC card, for a longer lifespan (not much fun to change card in a colocated server) and better performance.
            For instance: http://www.newit.co.uk/shop/proddetail.php?prod=Class_10_Integral

            • http://www.ewanleith.com/ Ewan Leith

              I looked for something exactly like this when I first bought the board, but couldn’t find one.

              Thanks!

    • lucidquiet

      Interesting post, I’ve been thinking about this as well. Instead of upgrading to a newer laptop, or workstation, simply adding in a new rack (so to speak). The trick, I feel, is creating a home station that is basically a mini-service-oriented-architecture. Additionally, a use might be to duplicate the “cloud” locally, since running it on a single machine doesn’t quite match the cloud deployment environment. For instance, instead of using Dokuen (https://github.com/peterkeen/dokuen) you might have a little rack of mini arm computers.

    • Stu

      Are you hosting this WP blog on the ARM device (as part of the normal server usage)? It might be interesting to see how it deals with the HN crowd :)

      Great post and good on you for putting your theory into action.

      • http://www.ewanleith.com/ Ewan Leith

        No, I’ve not dared “expose” it to unpredicatable users yet :)

        It’s currently running a few services that I control, but I will try flipping the blog traffic over to it sometime soon and see what happens..

    • http://www.kyleisom.net/ Kyle

      I run similar set ups using Sheeva plugs from Globalscale, and what I’ve found to be useful is to keep a bare root fs mounted read-only on an SD card (remounting rw for updates and to install software), and keeping /home and /var on a usb hard drive.

    • http://twitter.com/adriaan_pelzer Adriaan Pelzer

      We used to build Flash disk based encryption devices as far back as 1999, when writing to a Flash disk was still something you had to think about very carefully, since they had a very limited write cycle lifetime.

      We achieved very good performance by using picoBSD (comes with the FreeBSD source distribution), and running the entire system on a 32M ramdisk. (A lot like you would do when building a boot floppy).

      We would then only mount the flash disk when configuration changes or firmware upgrades took place. This took the flash disk out of the loop almost entirely.

      Sounds like something you could do with the SD card too.

    • Vinicius Alves

      Great experiment, I’m very excited to see the results.

    • RichB

      SheevaPlug. Why did Marvell drop the ball?
      My SheevaPlugs have been serving a 20K subscriber RSS feed 24/7 for 2.5 years

    • Danko

      “servers really need Ethernet or Serial consoles”… Serial consoles… The SERIAL CONSOLES called, they want their dodgy 1960s tech back. Why do you hate society and progress, you Luddite?

      • http://www.ewanleith.com/ Ewan Leith

        Because, rather surprisingly, that’s what most colocation providers offer as alternative access to your server in case the network fails…

        • JamesR

          It might not be so surprising why serial out-of-band (or ethernet, usually doing ssh to serial console anyway) is fantastic in the following situations :-) :

          1) Networking failed, perhaps you (or an initscript failing to do a networking restart fully) cause you to lose network access, the only way to fix it is over something other than your downed network port – serial console saves the day. (some servers do ethernet console shared with a normal networking port, but this is no help if you’ve downed the physical link to the switch – hence why serial or dedicated ethernet oob is valuable)

          2) There’s a kernel panic. What happened? You just go straight to what you’ve captured from your serial console and you’ve got the whole thing there for copy and paste to submit bug reports on, or search with.

          There’s also plenty of servers with IPKVM these days too, but this is less useful when you have a kernel panic as you can’t cut and paste the crashdump, and noone wants to transcribe a screenshot of register dumps into the plaintext it came from, so text based really wins here! Maybe when you can paste a screenshot of a panic into google the text based method will be less of a concern.

          Of course IPKVM is rather handy if you’re one of the few running windows as your server os.

          • http://www.ewanleith.com/ Ewan Leith

            Those are all good reasons, but the surprise is that something like 30 years after serial ports first became common, there’s still nothing better!

    • Ludovic Urbain

      Been looking into that a lot and the unfortunate truth is nobody is currently producing a cost-effective alternative to x86, that cortex a8 cpu is so weak it would take more than 2000 bucks of beaglebones to match a light bulldozer virtualization host that cost 500.

      The big reason the server makers and INTEL don’t rush into ARM is that it’s going to slaughter the margins, as you can source anything from anyone and the ARM license is cheap.

      In the end, most of the stuff will be ARM or another RISC that has much more open competition, but all the players will delay that as long as they can.

      • benjamincburns

        I build custom ARM hardware for a living. I promise, if you’re a company with high volume working at the component level rather than an individual buying off the shelf single-board-computers, ARM is much, much less expensive than x86.

        You hit the nail on the head about low margins/commoditization, though. However, if the value proposition is there, it might be a rain maker for a smaller player that’s willing to accept low margins for revenue growth. Dell probably isn’t going to start selling ARM servers right away because the margins on x86 are too fat, but with decent funding Joe Blow Electrical Engineer and his buddy Larry the Layout Guy can put together a very inexpensive Cortex board no problem.

        Annnd I’ve hit my buzzword limit for the day, folks.

    • Philip Trickett

      nice experiment.

      http://www.embeddedarm.com/ run their site on an arm board:
      http://www.embeddedarm.com/about/resource.php?item=257

      Nice board, I have one, but I have never tried to connect the SATA disk ports.

      Phil

    • Michael

      I’m guessing one reason mini-servers aren’t more popular in datacenters is that you’d have more hardware. Each box needs to be racked, have it’s own network cable, etc. Yoi also lose the flexibility thay comes with running on a VM, especially the ability to resize your.box as needed.

      • http://www.ewanleith.com/ Ewan Leith

        That’s certainly true on a small scale, but if a datacenter was hosting 100s or 1000s of these things, there’s nothing stopping them building custom chassis’ to hold them all, much like a blade center.

    • vitalie

      I’ve ordered a CuBox from Solid Run (99 EUR + shipping) and I’m happy so far: Ubuntu Lucid, 1 GB DDR3@800 RAM, Gigabit Ethernet, 2 x USB (I’m using an external HDD), more info here:
      http://www.solid-run.com/products/cubox

    • benjamincburns

      You probably know, but SD cards are essentially miniature SSDs, except they’re worse for reliability under moderate to high write load because all of the wear leveling is going into the same chip. I don’t remember if the bone has a USB host or not, but if you could plug in an external HDD and use that for any write-intensive operations, you’ll have much better luck.

      • benjamincburns

        Also, no matter what the controller on the SD card does, you’re still limited by the write speed of the NAND. For higher density MLC devices, this can get quite slow. SSDs get around this by interleaving writes across multiple cells/chips, but again, SD cards are limited in this way.

      • http://www.ewanleith.com/ Ewan Leith

        The problem with running an external HDD is power consumption will be massively higher for the HDD than the entire beaglebone, which runs at under 0.5W :(

    • Punit

      When running Ubuntu on an ARM board from SD Card I noticed that there is copious amounts of logging which effectively is killing the SD Card bit by bit. Have a look at /var/log/* to find out which files are seeing frequent updates and lessen the log levels for stuff you don’t care about. Might help extending the life of your cards…

    • http://twitter.com/rodedwards rodedwards
    • FiloSottile

      It may be a banal comment, but you really should try using RasPi!

    • Solar-Powered Supercomputing

      The OMAP hacking community (in academia mostly) has substantially addressed all of your issues to a fairly acceptable standard. The Sandia Strongbox uses both serial and usbnet to supplement a rather anemic ethernet. Some of these wouldn’t be appropriate for your usage but on our Pandaboard cluster we get decent WattsUp meter measurements because of our node count, and we boot from USB using a variant of https://groups.google.com/group/pandaboard/browse_thread/thread/f73e5e6c7127baab?pli=1

    • Pingback: Sysadmin Sunday 81 « Server Density Blog