It’s just about time for me to sort out the patchwork of devices on my home network. The real kicker for this has been all of the twitterings around the imminent release of Leopard, and some reading about what Time Machine is going to do for me. I’m excited.
Here’s the problem statement:
- a Netgear 54G wireless router with firewall on an ADSL connection
- a 64-bit AMD Linux workstation with RAID1 160Gb internal disk on which most of my photos are stored… but the power switch is broken and I rarely have the machine on as a) it has overheating issues and b) I have to touch pins on the motherboard to boot it. Half of my email archives are on here too.
- a Pentium 3 Linux server – a total workhorse. It fetches all my email from various accounts, filters it using SpamAssassin, and serves it up via SquirrelMail; it also presents our old black-and-white HP LaserJet to the other machines via Samba.
- Windows XP Thinkpad with two iTunes libraries, and some more photos (this used to be my primary photo processing machine prior to the MBP). The RAW photo library is split between an expired beta copy of Lightroom, and stuff from Rawshooter which hasn’t been imported to Lightroom. I used to back the photo directory to the Linux workstation over the network, since the disks were RAIDed.
- MacBook Pro which has my primary iTunes library (i.e. the one that syncs to my nano) and the majority of my photos… but the disk is getting pretty full, to the point where I’m no longer ripping CDs even though I want to… and I’ll soon have to move some of my photos off, but worry about breaking the Lightroom library.
- Windows XP Thinkpad which has yet more music on it.
- couple of external USB disks (300Gb and 250Gb I think) which have various backed-up things on them.
This all works, but it’s a bit of a mess really.
My questions / needs are:
- shared storage which will support a consistent and large photo library, a large music library, and Time Machine backup.
- access to the printer (which is parallel, so I guess I need to keep the old Linux server, which is OK… or do I ditch the printer and get something more modern)
- how to get all of the photos together into a single library… in fact, how to sort out my workflow in general. Essentially I don’t want Lightroom getting upset if I move photos off to external storage, and I guess I’d probably still import images to a local disk on the MBP when I connect the camera (as I may not be at home when I do that) and then move them off to the shared / external disk later. On a related note, what is the “right” workflow… do people export JPEGs from Lightroom into iPhoto when “done”? I currently don’t really use iPhoto and I’m not sure I’m making the most of the Mac.
- a centralised iTunes library but one which I can sync my iPod to, and which I can potentially have a subset of on the MBP disk for travelling (is that even possible?). I don’t expect the whole library of all of my ripped CDs to fit onto the MBP or the iPod, ever – I have a LOT of them.
- support for a mixed network. I guess I’ll continue to have a Windows box somewhere, although I’m (*gasp*) seriously considering the removal of the Linux things and replacement with an iMac or Mac Pro (at some point – contrary to the way this post must make me sound, I am not made of money)
I’m hearing good things (from Maria Langer, who I’ve come to trust on Mac issues) about the AirPort Extreme. I assume that I could get one, link it to the Netgear router, and attach disks to that. I’ve already read about the NSLU2, and had my eye on one for a couple of years. I also wonder about a proper NAS unit, and then I wonder about RAID options. Or, I could go for another external disk… I have no objection to that provided that I have the discipline to hook the MBP up to it for Time Machine usage when I’m at home. I’m thinking I’ll do a full backup of the Mac before I install Leopard anyway.
Then there’s just the small matter of actually combining all of the data in one place. I get the impression that this will be messy, and that’s why I haven’t done it yet. Combining iTunes libraries which may contain duplicates… not knowing how I can split music between the network and the laptops… trying to merge a full-version Lightroom library with a beta one and then a load of stuff from Rawshooter and my pre-DSLR pics… you can probably see why I’ve been putting all of this off.
Ugh. And here I am working in the IT industry and allegedly knowing about this stuff. I guess it’s like the old adage “physician, heal thyself”…
Advice gratefully received…
Oh my… that’s giving me a headache just reading about it. What is it with computer people collecting so much stuff? 🙂 I feel like a complete lightweight with my one Thinkpad and one MacBook. I do have a printer but that only comes out of its box on special occasions. Apart from that I have some webhosting and use AWS to backup everything.
Even with my puny setup I do have trouble synching my iTunes library. It’s a pain having to startup my Thinkpad to do it, I’d like to synch it at work, but I don’t have my MacBook then. I was considering using CVS (on my webhosting) to do it, but I was a little worried that all the CVS files that get added would cause iTunes some pain.
Hey Andy,
I would think the Airport Extreme/NSLU2 option to mount the USB hard drives which will contain all the music (don’t worry about duplicate because iTunes can show them so you can remove them). You have the iTunes libraries on the local disks on which ever machine you want but the actual Music Location will be the location that the USB disk is mounted on. Time machine can use the attached drives for backup.
Get rid of that printer, I got a new HP with a ethernet interface for £75. Plugs straight into router, and bonjour picks it up automatically on the home network.
As for the RAID array on the Linux box – why not run rsync (or Unison) scripts on the Windows/Mac/Linux machines to synchronize all of your data onto it (via IP and SSH tunnel if you like). This is what I do (I actually have all my machines [two windows laptops and one mac] wake up at 3AM and synchronize important data between them, and for the photos and music I sync to a attached USB drive once a week).
For on the road music you might need to either 1) use the ipod :), or 2) run another script to sync the stuff you might want on the road (i.e. maybe on file date if you want only new music, or just hard coded).
The photos are more interesting…. I will leave that for now (it is based on size, archiving needs, etc).
Also, the linux machine might come in useful for a media server one day…
I have these same sort of problems (my photo lib is 50GB and music is 35GB and I want to carry it all on my Thinkpad – together with IBM Ubuntu, RSA/WebSphere, VMWare images, ahhhhhhh!!!). Lucky 160GB laptop hard drives exist (320GB when using the hard drive bay as well).
Tell us how you get on 😉
Justin
Where to start?? (which seems to be your problem anyway!)
If I were looking to purchase a decent backup disk at the moment I wouldn’t look further than A Lacie 2big Triple ( http://www.lacie.com/uk/products/product.htm?pid=10967 ) 500GB of RAID 1 or 1TB of RAID 0. Expensive but worth it IMHO. I’m currently using a 500GB BigDisk Extreme d2 as the backup of my iMac, which is kind of insane as it is RAID 0 and therefore doubling my risk of failure. It *will* be replaced with the above solution at which point the BigDisk becomes a scratch disk for photoshop/video work. The Lacie triple interface stuff supports Firewire 800 giving you maximum speed when attached to your MBP.
I’d be tempted to get rid of the AMD Linux box and replace with NAS storage. I’m a happy NSLU2 user, and have it unslung to run TwonkyVision. However I suspect there are better NAS solutions around nowadays. I wouldn’t suggest trying to use the same disk NAS attached and then USB/Firewire attached to your Mac. Use separate ones.
I’m not qualified to suggest how you might replace the mail server box, but I’d be tempted to find a solution which didn’t require me to run a machine 24/7. I get along very well with a hosted IMAP mailbox
As for the printer, dump it! Ethernet enabled printers are ten-a-penny nowadays. If you can’t conveniently locate it near your router, then an Airport Express will be a good solution. I do use an Airport Extreme and love it. I don’t have a disk or printer attached though. 802.11n between it and the iMac is lovely.
I’ve not looked into TimeMachine enough to know if I’ll use it in favour of SuperDuper. I do like the fact that I have a bootable clone of my iMac disk if things went wrong.
I used to run my iTunes library from the disk attached to the NSLU2, which was then mounted on my old iBook. It worked, but iTunes got unhappy if the mount wasn’t there when it started up. What’s more it significantly slowed down iPod syncing and any activities which trawled the whole library (album art updates, new iTunes versions etc.) I don’t do it now I have the iMac as I have enough disk space and the only portable music mechanism I need is my iPod.
As for photos – I rather suspect the answer you need is to realise that you can run multiple catlogs with Lightroom, and that you don’t need to keep everything in one. That way you can keep catalogs in different physical locations. Think of catalogs as you would Eclipse workpaces ;-). For transferring the old ones from the Windows beta – did you configure it to save edits in the XMP sidecar files? If so then just copy all the RAWs and XMP files to your new location for them and import them into a new Lightroom catalog. If not and you don’t want to lose any edits, keywords etc. then your license key should allow you to install Lightroom 1.1 on Windows and then export catalogs which you can just copy over.
The NSLU2 is excellent. I have one which I run Debian on (http://www.nslu2-linux.org/wiki/Debian/HomePage), and use it serve up files (via Samba) and MP3s to my Roku Soundbridge (via Firefly), as well as do some overnight remote backups, etc. I’ve a sneaking feeling Firefly might also do the job for your iPod, but you’d need to double-check. The NSLU2 is cheap (60 quid + the price of a USB disk) and quiet – so much so I’m considering getting another to backup the first one onto. I suspect if you use the stock firmware, it’d still be fairly useful (although that limits the scope of usage – if you want to fiddle, Debian or Gentoo are probably good choices).
My other recommendation would be to ditch the local SpamAssassin installation. I did this for a long time, as I had an email address with insane amounts of spam, but in the end I just gave up, changed the address, and now my ISP hosts it on their IMAP server. Fun though SpamAssassin is to tweak, there are plenty of ISPs that do a good-enough job of spam killing.
Another point – I think Time Machine will only work with HFS+ formatted volumes as the backup volume according to :
http://www.apple.com/macosx/features/timemachine.html
“You can designate just about any HFS+ formatted FireWire or USB drive connected to a Mac as a Time Machine backup drive. Time Machine can also back up to another Mac running Leopard with Personal File Sharing, Leopard Server, or Xsan storage devices.”
So I doubt a NAS solution will work for you if you want to use Time Machine for backup. Either locally attach a drive or use SuperDuper or Carbon Copy Cloner. And even then you are probably going to be limited to creating a sparse disk image.
I’ve read some reports that say Time Machine will work with a journaled disk attached to an Airport Extreme base station. However based on my experience of write speeds to NAS devices I’d be worried about performance.
This is a very similar setup to what I had about six months ago. Then I got smart, sold three computers on eBay and consolidated from a MacBook Pro (OS X & Windows XP via Fusion), Sun Java Workstation (OpenSolaris), and PowerMac G5 (OS X) onto a new Mac Pro.
The Sun machine made a great ZFS file server with a 400GB mirrored ZFS data drives, email server, web hosting (Solaris zones), database (zoned), etc… at the MacBook Pro hard drive was very cramped.
Now I’ve only got the Mac Pro, but have the main boot disk in addition to moving the dual 400Gb SATA drives in as a mirror. All my current Windows, Solaris, BSD, and Linux needs are met running VMware Fusion on the same box. Sharing data and services is very easy. In fact, beefing the OS X firewall some, I’ve even ditched the Netgear router as one network feeds all of the virtual machines.
I don’t know specifically about Lightroom as I’m an Aperture user, but Aperture allows you to use “referenced” file locations. You can use either one big library or a hybrid of files all over the place…a very slick feature, but one I don’t use now that that I don’t need the MacBook Pro anymore.
So thanks everyone for the very detailed, very helpful comments and apologies for not having commented on this myself so far.
My thinking was along the lines of a NSLU2 but the issue is around the need for an HFS disk for Time Machine… so that pushes me towards an Airport Extreme. I’m not opposed to one of those… but it only offers the single USB port for disk or printer. So at that point, I’m limited to, say, a 500Gb disk on an Airport Extreme for iTunes and Time Machine… and that depends on whatever printer I might choose having an Ethernet port.
I’m relatively keen to get rid of one or both Linux boxes. It would be a shame in one sense, but the room would get cooler, quieter, and emptier. That means the disk mirroring goes away, but I can maybe setup an NSLU2 along with the Airport Extreme with some more storage… or is this all getting more complicated again?!
Printers… pretty much all of our printing is B&W but if I could get a nice quality photo printer that would be good too. Anyone got any experience of the multifunction Canon units?
On the disk front… Aidy, I’d looked at the Lacie disks but reviews I think I saw on the Apple store were less than nice about the disks with the 4-way interfaces, for instance, so I was a bit nervous about them. I will take a look at the RAIDed ones though as that would give me added confidence.
Phew. I can foresee a nice clean desk (although I’m appalled by the potential for junking all this kit), but I can see that I have the potential to build something complicated again 🙂
Oh, and on the mail front… I keep thinking of Gmail to aggregate my accounts together. I think I can import my old mail archives too, but I’m not certain. The annoying part is that, for example, the site I’m currently working on blocks access to Gmail, so I’m glad of my own server in that case.
Andy,
I don’t have a complete solution, but I can suggest one part of it.
You want to use a Linksys WRT54GL router, http://tinyurl.com/2stetu
It is easy to install Linux on it. I use the “White Russian” version. It provides a brower-interface for administration.
It is rock-solid and a delight to use. It’s also cheap.
thanks,dave
Re priorities, and especially since you are a fellow WordPress blogger, see
http://daveshields.wordpress.com/2007/10/23/dashboard-dashboard-on-the-panel/
thanks,dave
I was using an NSLU2 but kept getting annoyed by the hardware limitations, so I’ve switched to a D-Link DNS-323, basically the same idea but more powerful hardware, 2 disks inside the enclosure doing mirroring.
a tough one.. sounds like my setup a few years back.. which was a main music making PC in the music room (with some large drives) a linux based fileserver (again with large disk array) the old G4 mac, a tabletpc, a ‘normal’ laptop, a secondary laptop for ‘really important’ backups, and a little fileserver under the stairs.. (no really.. by the wine celler!)
backs were a werid kinda cross backup linky thing that meant data was always held in at least one other set of drives, and a FTP’ed off site backup to a friends house for security!)
the setup now is much cleaner.. G5 mac has 2 terabytes of attached storage (two firewire external drives), and there is the same for backup (thats to MUMnDAD Backup solutions, a single 1 terabyte drive is plugged in on a thursday night to copy the incremental backup of the attached 1tb, it moves to mum and dads place just down the road on a friday morning!) The rest of the machines then access the G5 and its storage.. so all itunes and iphotos system read from that drive, it’s also shared with the single lonely PC thinkpad. I have no idea how time machine will work with this… it sounds good, but how will it cope with backup drives that arent in place all the time (which to be honest… you dont really want your backup drive of all your photos sat on the desk next to your nice mac, nice and handy for the burgler to pick up on his way out..)
Via engadget –
Apple confirmed it: no backing up with Time Machine over the drives you have connected via USB to your Airport Extreme. Also, no Time Machine backups to SMB shares — AFP network shares only. Again, yes, Time Machine can back up over the network, but ONLY to AFP shares, ok?
[…] before the store opened, Justin commented on my blog that the feature to perform Time Machine backups to an Airport disk had been pulled at the last […]