home studios equipment staff/friends booking/rates for sale forum contact

archiving DAW sessions

All your geeky questions about electronics, microphones, tape machines, etc.

Moderators: kerble, Electrical-Staff

archiving DAW sessions

Postby jumpskins on Sun Feb 12, 2017 12:48 pm

hi

the focus of my university thesis is digital preservation for session-level audio and its related objects.

...stay with me.

i'm curious about current industry/pro-sumer/consumer practices and standards for archiving, cataloging and preserving audio projects, multitrack stems and related metadata.

i mean sure, tape is still an option. but i'm just concerned with the ones and zeros.

so how do you archive your projects? are you concerned about hard disk failures, future access, software/hardware obsolescence...? do you have a bunch of DAW projects from 10 years ago that won't load up anymore? i wanna know about your difficulties and concerns.

also if anyone has any contacts or papers on the area they could share, id be grateful. while this seems to be a narrow area of research as far as existing literature goes, i feel like its an underdocumented problem... or maybe theres nothing to be discussed and im barking up the wrong tree :|
jumpskins
fan
fan
 
Posts: 11
Joined: Wed May 26, 2010 6:31 am
Location: yorkshire, UK

Re: archiving DAW sessions

Postby aaronk on Sun Feb 12, 2017 9:19 pm

When a session is done, what I do is highlight all the tracks from a point that is just before the the beginning of the earliest starting clip to a point that is just beyond the ending of the track or whatever. From there (in Protools), I choose to consolidate all regions. This will make all the clips the same length. Then you can export all the clips into a separate folder as WAVs and they will all start and end at the same point. This ensures that if WAV survives as a format, you should at least be able to import the audio into any DAW and have all of the audio play from the correct point. I don't really care about preserving things like fader position or plug-in settings or anything like that (the batch export doesn't take those things into account). You wouldn't be able to do that on tape either unless you re-recorded every track. But this method is quick and easy to do, at least in ProTools, and should be effective as long as WAV survives as a format. Perhaps include a PDF of session notes if you've got the energy.

Cloud storage is getting pretty cheap. Keep your files in at least 3 separate locations and I think you'd be doing the best you can.
aaronk
ensign
ensign
 
Posts: 22
Joined: Wed Sep 21, 2016 11:04 pm
Location: Long Island

Re: archiving DAW sessions

Postby cak on Wed Feb 15, 2017 2:16 am

In November, I completed my first "professional" solo album after about 15 months of sporadic work, and while archiving is a fairly daunting task which I have yet to perform *completely* (everything is on Cloud and laptop hard disk, just not quite in a definitive state), I think it can broken down into original unprocessed stems, edited (but not processed) stems, the final processed stems, stereo mixdowns, and perhaps basic mixing notes like balance/pan adjustments. The last part is where it gets tricky; one cannot rely on the mere backing up of session files, unless you have a way of extracting balance/pan adjustments into a readable text format. One day the session files will be obsolete, or you will have lost the software needed to open them.

Aside from that, I also have some Yamaha AW16G files from years ago which are going to need to be migrated to a non-proprietary format. I'm pretty sure most people have gotten away from proprietary formats by now. So yeah, no session files, no proprietary stuff, and, as previously stated, have your stuff in at least 3 places including Cloud (I use Google Drive and currently pay $2 a month for 100gb). Also (and I am super bad about this) it is important to check data integrity.
User avatar
cak
philip guston
philip guston
 
Posts: 750
Joined: Thu Aug 02, 2007 10:34 pm
Location: Oregon

Re: archiving DAW sessions

Postby Mason on Wed Feb 15, 2017 12:26 pm

This is something that weighs heavily on my mind!

I've acknowledged that I'm just going to have to record digitally. It's fine—it's how I started/what I grew up with anyway.

I don't have a studio, and I record my own projects almost exclusively, so I'm not in a position where I can regularly afford tape or keep a professional tape machine properly maintained. I own an eight-track cassette recorder and have a six-track cassette recorder on loan for some reason, but anything else about them aside, those are fairly arcane formats in their own right, so hardly anything can be gained there.

Anyway, this is part of why I adopted Reaper as my DAW a while ago. I appreciate that they don't have obtuse licensing schemes that will hinder future third-party recovery. No dongles or whatever else other DAWs rely on. Anyone can run it, even off a thumb drive. (Will we have USB drives in the future, no. Carrying on:) Along the same lines, I try to use third-party plugins as rarely as possible. I hardly use any "outboard" besides EQ, compression, and the occasional heavily bandpassed reverb, so I can get away with that. The plugins Reaper bundles for those purposes are actually great, but primarily it's about minimizing how much of a session is going to go missing in the future.

And yeah, I keep a nicely organized collection of original stems and stems w/ effects printed, WAV format, on top of the two-track prints and, once they get back to me, the final masters. I also keep the files to run Reaper as a program where all my sessions are saved, and in the rare case that I record other people, I'll include Reaper with all the session files they get. I've been keeping very minimal session notes, but that's where I'll document settings for the rare third-party plugin I use. I store all my files on several drives—I never set up cloud storage, since I've felt that provides a false sense of security, but I should probably get on that because it clearly provides some security.

Digital archival is gonna give you problems, and all you can do is mitigate them. If it helps, take comfort in the fact that basically all present-day movies, TV broadcasts, photographs, and correspondence are also condemned to the ether? Since that's comforting?

For consideration: the ARSC Technical Committee's guidelines on the matter.
http://www.arsc-audio.org/pdf/ARSCTC_preservation.pdf
Marsupialized wrote:'It's very important for men in bands together to French kiss, every day, even if they don't feel like it'
User avatar
Mason
Master Of The Computer
Master Of The Computer
 
Posts: 4961
Joined: Sat Apr 26, 2008 10:42 pm

Re: archiving DAW sessions

Postby projectMalamute on Wed Feb 15, 2017 1:47 pm

With analog you have the master tape, which is durable and presumably something you will be able to play on some sort of system in the future, even if you have to do some jury rigging.

With digital it seems the expectation is that, in addition to being able to just play back the data, you should also be able to recall all the plugins and mix settings.

To me that's not the same thing at all. That would be like arguing that analog tape is a bad archival format because in 100 years you would need an SSL console and a bunch of fancy outboard gear in order to recover the session.

Just saving the stems in an openly documented file format gives you the same information as you get with an original analog tape.

With analog the physical media is sturdy, that's what it has going for it. With digital it is possible to make an unlimited number of perfect copies and store them in multiple locations, a different strength. I'm not convinced analog is the clear winner here.
bandcamp
youtube
web site

Endeavor to persevere.
User avatar
projectMalamute
Saint Who Rules w/ Extensive Magnanimity
Saint Who Rules w/ Extensive Magnanimity
 
Posts: 2014
Joined: Sun Nov 18, 2007 10:59 am
Location: Westfield MA

Re: archiving DAW sessions

Postby tmoneygetpaid on Wed Feb 15, 2017 2:06 pm

aaronk wrote:When a session is done, what I do is highlight all the tracks from a point that is just before the the beginning of the earliest starting clip to a point that is just beyond the ending of the track or whatever. From there (in Protools), I choose to consolidate all regions. This will make all the clips the same length. Then you can export all the clips into a separate folder as WAVs and they will all start and end at the same point. This ensures that if WAV survives as a format, you should at least be able to import the audio into any DAW and have all of the audio play from the correct point. I don't really care about preserving things like fader position or plug-in settings or anything like that (the batch export doesn't take those things into account). You wouldn't be able to do that on tape either unless you re-recorded every track. But this method is quick and easy to do, at least in ProTools, and should be effective as long as WAV survives as a format. Perhaps include a PDF of session notes if you've got the energy.

Cloud storage is getting pretty cheap. Keep your files in at least 3 separate locations and I think you'd be doing the best you can.


This is pretty much exactly the procedure one of my teachers showed us.

I do this sporadically, frankly too sporadically. As Steve has pointed out many times, this is a procedure you have to spend time on, whereas with tape once it's recorded the material is already archived. And because it requires time and doesn't yield a tangible, visceral reward (like it sounds cool), it's hard to stay in the habit of doing it.

I do migrate data between drives (ditch my old storage drive and replace with a new one) every ~3 years, or when issues with the drive manifest (bearing noise, SMART warnings, etc.) which corresponds to the life of the warranty. I don't schedule this or have some structure in place to force myself to do it regularly, I remember to do it informally.

As an aside, I have offered that the increasing ubiquity of the computer (and the tablet, phone, etc.), which is the storage and playback device for the actual audio when recorded and stored digitally, is digital audio's main benefit for archiving. Tape machines aren't made any more. Tape and supplies are barely being made. Keeping a tape machine operational and in good working order is almost impossible. So someone who has audio stored on tape will have to put considerable work and cost in to be able to listen to it, even if it's just finding people that do maintain tape machines, and that will presumably only increase with time. With digital, provided the data stays intact (which is not a given with digital audio, but is more of a given with tape) anyone can listen to the audio on a computing device they own, or one borrowed from a friend or neighbor. They could even play with the multitracks by downloading an open-source multitrack audio editor.

We should also note that the Wave format, although ubiquitous, is proprietary, and as such is not a great choice for archiving.
tmoneygetpaid
laney
laney
 
Posts: 159
Joined: Sat Apr 06, 2013 1:13 pm

Re: archiving DAW sessions

Postby 154 on Wed Feb 15, 2017 2:17 pm

Slightly off topic, but I find it surprising how hesitant people are to print effects to tracks now. It seems like raw tracks are seen as precious with this mentality that you are always going to want options for the future. If I want a slap sound added to a guitar part, I want it to become part of the art and not have to worry about what plug-ins (or whatever) were used and are available to recreate it to the future.
User avatar
154
Power Incarnate with Endless Creativity
Power Incarnate with Endless Creativity
 
Posts: 7813
Joined: Fri Nov 07, 2003 10:56 pm
Location: MKE

Re: archiving DAW sessions

Postby givemenoughrope on Wed Feb 15, 2017 2:24 pm

projectMalamute wrote:With digital it seems the expectation is that, in addition to being able to just play back the data, you should also be able to recall all the plugins and mix settings.


That's really the expectation? I'm sure that I'm in the minority but I'm always rendering tracks so whatever audio I playback in a different DAW (or just a different session) has been eq'ed, compressed, effected, etc. (Especially now that I'm using Acustica Audio's plugins on an older i7, I tend do that as I go since the plugins are CPU monsters and buggy as well.) And actually I will do this in stages before I do the final mixing levels; basically treating it like hardware. If there is to be some protocol is should at the very least require this. Plugins and OS's are a constant juggling act with compatibility rarely survives more than a few years. The only hope is to have mixed "stems" or rendered tracks, and even then...

What about mastering to analog? Not the same but something.
vockins wrote:So it's like the Finnegan's Wake of Super Bowl ads.
User avatar
givemenoughrope
Lode Star of the Twenty-First Century
Lode Star of the Twenty-First Century
 
Posts: 2376
Joined: Mon Jun 30, 2003 3:41 pm
Location: los angeles, a happy place

Re: archiving DAW sessions

Postby projectMalamute on Wed Feb 15, 2017 3:36 pm

tmoneygetpaid wrote:We should also note that the Wave format, although ubiquitous, is proprietary, and as such is not a great choice for archiving.


It's an open standard that was originally defined by Microsoft and IBM. You don't need any proprietary hardware, software, or licensing to use it and the specification is a matter of public record.

.wav files are fine.
bandcamp
youtube
web site

Endeavor to persevere.
User avatar
projectMalamute
Saint Who Rules w/ Extensive Magnanimity
Saint Who Rules w/ Extensive Magnanimity
 
Posts: 2014
Joined: Sun Nov 18, 2007 10:59 am
Location: Westfield MA

Re: archiving DAW sessions

Postby Adam P on Wed Feb 15, 2017 4:20 pm

givemenoughrope wrote:
projectMalamute wrote:With digital it seems the expectation is that, in addition to being able to just play back the data, you should also be able to recall all the plugins and mix settings.


That's really the expectation?


Obsolescence of DAW software and computer hardware always seems to be the primary argument against digital that I see, sonic quality notwithstanding. Setting every track to the same start point (a task that takes all of 30 seconds in Pro Tools) and redundant storage of the WAVs in a few locations does not take a lot of effort or babysitting.
User avatar
Adam P
meatball enthusiast
meatball enthusiast
 
Posts: 1258
Joined: Mon Dec 19, 2005 6:44 pm
Location: Cleveland, OH

Re: archiving DAW sessions

Postby eliya on Wed Feb 15, 2017 10:12 pm

givemenoughrope wrote:
projectMalamute wrote:With digital it seems the expectation is that, in addition to being able to just play back the data, you should also be able to recall all the plugins and mix settings.


That's really the expectation?


Seems to be if we judge by where this thread had gone, but projectMalamute nailed it. It's ridiculous to ask a software to recreate a giant control room worth of settings 20 years from now.

Exporting processed and unprocessed stems is more than enough. The processed stems are an overkill, if you compare it to tracks on a multitrack tape. Stick the session files in the cloud, put them on an SSD drive and make sure to keep moving the files whenever the drive starts showing signs of failure.
User avatar
eliya
Power Incarnate with Endless Creativity
Power Incarnate with Endless Creativity
 
Posts: 7618
Joined: Mon Apr 18, 2005 12:37 pm
Location: Chicago

Re: archiving DAW sessions

Postby japmn on Thu Feb 16, 2017 12:27 am

God, I remember backing up to SCSI Tape backups after each session back in ProTools 3-5 days. It took something like 6-8 hours to back up a 1 Gb session. These days, Just back up to a drive or 2, and 2-3 DVDR's. It's so fast to back up to those DVDr's, you can just make a bunch of copies and, probably, one will work. It's faster than doing an Analog tape work copy, these days. rely on numbers.
User avatar
japmn
Power Incarnate with Endless Creativity
Power Incarnate with Endless Creativity
 
Posts: 7865
Joined: Mon Dec 27, 2004 12:38 am
Location: Poutland

Re: archiving DAW sessions

Postby bishopdante on Tue Feb 21, 2017 11:07 am

154 wrote:Slightly off topic, but I find it surprising how hesitant people are to print effects to tracks now. It seems like raw tracks are seen as precious with this mentality that you are always going to want options for the future. If I want a slap sound added to a guitar part, I want it to become part of the art and not have to worry about what plug-ins (or whatever) were used and are available to recreate it to the future.


Should really back up several versions, both consolidated raw and consolidated processed, and non-consolidated raw + OMF, and have a written document detailing the processing involved, with screen-grabs / photographs of the various settings (whether in-computer or hardware outboard).

Proprietary session files more than a few years old are a nightmare to work with, often can't even be opened by newer versions of the same software.

File management and well-organised handover/archiving is something to take seriously. Very easy to make a mess / lose loads of stuff.
"One of the painful things about our time is that those who feel certainty are stupid, and those with any imagination and understanding are filled with doubt and indecision." - Bertrand Russell
User avatar
bishopdante
Humankind's Greatest Musical Genius
Humankind's Greatest Musical Genius
 
Posts: 4331
Joined: Tue Apr 05, 2011 11:25 am
Location: London

Re: archiving DAW sessions

Postby davidpye on Thu Feb 23, 2017 12:40 pm

For every project I work on:
I export stems of the final mix (Logic Pro); I normally run off coarsely grouped stems which will 100% represent the final mix when summed (Kit, Bass, Guitars 1, 2, 3 if there are loads, Vocal, Backing Vocals, extra stuff whatever makes most sense). Plus I also export all tracks as audio files as a super-extra-mega-backup, these files will not of course represent the mix I delivered, but they will represent the arrangement and recording. I think of those as a multitrack archive.

I also normally run instrumentals, instrumentals with backing vocals, and cappella lead vocals before the tracks are mastered as most bands want those for PR opportunities. That way they can be mastered at the same time as the main mix.

MOST of the time, from my recent experience, people only seem to want stems for the use of creative remixing, and almost all remixers only want grouped stems, a lot of the time with effects rendered into them. So the grouped stems work fine for that. I export all tracks as continuous audio files to put on a hard drive somewhere as an absolute last resort.

I honestly can't remember the last time anyone actually asked me for any multitrack of anything I've recorded or mixed. This is probably because I always send stems when I'm done mixing as a matter of course, but still, I don't know of anyone making any real use of them.

As for storing the archives, they're all on portable USB hard drives in a cupboard in my studio. I use standard off-the-shelf USB bus powered drives to increase the chances I'll be able to access them in future without hunting for the PSU (I WILL lose the PSU).

I know a few engineers and mixers in the UK who make a point of charging for stems to be run off, but I think that's a bit cheeky. It doesn't take long to do at all, and I'd never really want to leave a project without that sort of backup being made as a form of security. I think it's the responsibility of the engineer to protect the recording for future uses.

Sadly nowadays I always end up recording to digital, I'd love to be able to work with tape more like a few years back, but budgets, living costs, and people's expectations for the flexibility of digital forces my hand now really. I will record on Protools if the studio doesn't have Logic, but 90% of the time I'm only using Logic.
davidpye
berith
berith
 
Posts: 665
Joined: Wed Feb 02, 2005 5:46 am
Location: UK

Re: archiving DAW sessions

Postby bishopdante on Thu Feb 23, 2017 10:02 pm

davidpye wrote:As for storing the archives, they're all on portable USB hard drives in a cupboard in my studio. I use standard off-the-shelf USB bus powered drives to increase the chances I'll be able to access them in future without hunting for the PSU (I WILL lose the PSU).


Um... inside every external hard drive box is either a 2.5 inch drive or a 3.5 inch drive. If they have a USB plug on the box, chances are the drives natively talk SATA.

Image

They can be pulled out of the enclosure, and accessed with something like this USB to SATA adapter.

Image

Very handy little tools, those.

_____

Or you can get a desktop docking box such as these sorts of things:

Image

_____

Generally speaking it is much better to buy bare drives and box them yourself (or just run them bare). Much cheaper, and you know exactly what you are buying.

Very important to check the reliability stats, lots of independent companies are doing in depth analysis on test samples of thousands of drives per install, and they vary considerably.

_____

The best idea for bulk storage & backups is to keep the drives on all the time in a RAID rack, and employ automatic redundant backups, with the system flagging up a drive failure immediately, enabling replacement of the faulty drive and restoration of the data. (RAID systems can also be set up to do scheduled off-site backups to an FTP server, using something like Rsync). It is also not hugely expensive to have an uninterruptable power supply with a few lead-acid batteries to handle a power failure and power down gracefully if you blow a fuse / lose power.

Image

Also, that way you don't have loads of empty space sitting around across lots of separate half-full drives. It isn't initially the cheapest way to work, but it is cheaper in the long run.

It is a bit scary how few audio studios or video studios use the proper equipment, and instead just have shoeboxes or shelves with various portable USB drives on them gathering dust, half of which no longer work.

RAID racks are cheap as chips second hand, datacenters chuck them out regularly.

Using a datacenter type rack suatem also allows the use of SAS drives as well as SATA. SAS drives are more expensive but are built to much higher specs.
"One of the painful things about our time is that those who feel certainty are stupid, and those with any imagination and understanding are filled with doubt and indecision." - Bertrand Russell
User avatar
bishopdante
Humankind's Greatest Musical Genius
Humankind's Greatest Musical Genius
 
Posts: 4331
Joined: Tue Apr 05, 2011 11:25 am
Location: London

Re: archiving DAW sessions

Postby davidpye on Fri Feb 24, 2017 6:19 am

bishopdante wrote:Um... inside every external hard drive box is either a 2.5 inch drive or a 3.5 inch drive. If they have a USB plug on the box, chances are the drives natively talk SATA.


Yes I'm aware. A lot of the time now, at least in the UK USB drives are actually cheaper than standard bare SATA. And what do you get inside the box? An SATA drive, that's why I buy USB drives.
davidpye
berith
berith
 
Posts: 665
Joined: Wed Feb 02, 2005 5:46 am
Location: UK

Re: archiving DAW sessions

Postby bishopdante on Fri Feb 24, 2017 5:52 pm

It may be the case that in PC world or Maplin the cheapest USB drive will cost less than the cheapest bare drive, but that is likely due to the hard drives themselves being of substantially different quality. Usually the USB external drives have low quality drives in them (or are expensive, eg: lacie-seagate "rugged" w/mystery internals 1tb @ £100.)

Certainly if you buy online, I would certainly not recommend buying the cheapest drives, and it should be obvious that whether in a shop or online, the same part in shoebox-sized consumer packaging and a USB box compared to in an antistatic bag will inevitably be more expensive. Often twice the price.

The problem with buying boxed drives is that unless you open them up, pretty hard to know what drive is inside.

Here is a good article on the difference in reliability based on testing actual drives in a datacenter installation of tens of thousands of drives: https://www.backblaze.com/blog/what-har ... uld-i-buy/

And some graphs:

Image

Image

_____

Unfortunately since that article was written, Hitachi sold off their hard drive business.

The Hitachi platter drive division was sold to Western Digital, and is now called HGST, but the drives are still reportedly very reliable, and those are what I'd recommend. Have had very few Hitachi drives fail over the years, and IMO they are worth the extra.

Unfortunately the vast majority of hard drive buyers look at the price tag, not the reliability tests. Seagate sell a lot of drives and no doubt make plenty profit.

_____

https://www.backblaze.com/blog/hard-dri ... s-q1-2016/

Image

Of course if you're running a RAID system, it's a question of how long you get off a drive vs how much it costs, rather than losing data when a drive dies, but if you are actually relying on a drive completely, 1 in 100 odds of failure vs 1 in 10 or worse still 1 in 4 odds are *significant*.

As for the price difference buying one 1Tb 2.5" 5400rpm bare drive off the Internet?

Seagate: £50ish
HGST: £50ish

On the other hand, an HGST 8TB 3.5in SAS drive is £400ish. Price per Tb works out the same.
"One of the painful things about our time is that those who feel certainty are stupid, and those with any imagination and understanding are filled with doubt and indecision." - Bertrand Russell
User avatar
bishopdante
Humankind's Greatest Musical Genius
Humankind's Greatest Musical Genius
 
Posts: 4331
Joined: Tue Apr 05, 2011 11:25 am
Location: London

Re: archiving DAW sessions

Postby davidpye on Sun Feb 26, 2017 6:28 am

Interesting, thanks for the info on failure rates, I've definitely had a lot of useless Seagate drives in recent years!
HGST is on my shopping list.
davidpye
berith
berith
 
Posts: 665
Joined: Wed Feb 02, 2005 5:46 am
Location: UK

Re: archiving DAW sessions

Postby Stinky Pete on Mon Feb 27, 2017 5:10 am

Does anyone else offer the sessions to their clients? I don't really do it full time, so I do a 3 point backup (My pc's 2nd HD, my laptop and an external HD) but I also offer them the files. But it's also their music, they can buy hard drives just as easily and I feel the onus is more split now, I certainly don't intend to be migrating 10's of TB's of sessions in 3 years time.

fyi I don't render the processed stems. The audio are 44.1/48Khz BWF's (so they have timestamps), I have a folder of VSTs and IR's/samples in my drive and Reaper sessions files are readable as text.
User avatar
Stinky Pete
complete piece of shit
complete piece of shit
 
Posts: 143
Joined: Fri Sep 30, 2016 5:56 pm
Location: Off topic

Re: archiving DAW sessions

Postby jumpskins on Thu Mar 23, 2017 7:01 pm

thanks all for input on this. the stats about hdd failure rates are interesting for sure. ill throw my paper up here when im done. it will probably be a bit a little bit of a mess but i hope to convey what i can in the time ive been reading and writing.
jumpskins
fan
fan
 
Posts: 11
Joined: Wed May 26, 2010 6:31 am
Location: yorkshire, UK

Next

Return to Tech Room

Who is online

Users browsing this forum: tallchris, twelvepoint and 7 guests