You are here: Home » Articles » Archiving/Storage » Understanding Video Data Rates

Understanding Video Data Rates

[ This article was first published in the November, 2008, issue of
Larry's Final Cut Pro Newsletter. Click here to subscribe.
Updated: July, 2009 and May, 2010, to correct incorrect data rate entries. ]

As we move from the world of standard-definition to high-def, understanding hard disk speeds and the data requirements of video formats can prevent a lot of problems.

Here’s the key concept: hard disks are not infinite in speed, nor unlimited in storage.

 

The corollary to this rule is that dropped frame errors are most often caused by a hard disk that can’t keep up with your video format.

During my recent seminars, I spent a lot of time explaining hard drive speeds and video format requirements. This is a summary of what I talked about.

HARD DRIVE SPEEDS

While hard drive speed, that is, the speed with which a hard drive transfers data between the disks and the computer, is impacted by how fast it rotates, a much greater controlling factor is how the hard drive attaches to your computer. We call this a drive’s “data transfer rate.”

Here are several things to keep in mind:

1. All FireWire drives are hubbed. This means that when both fast and slow devices are connected to the built-in ports of your computer, the slower devices (cameras and decks) slow down faster devices (hard drives). For instance, connecting both a FireWire 800 drive and a FireWire 400 drive to the same computer, slows the speed of the FireWire 800 device 35% – 40% when compared to the speed of the FireWire 800 device connected alone. This speed drop is true whether the drives loop through each other, or both are connected to two different ports on the same computer. This is NOT true if you have added a plug-in card to provide additional FireWire ports.
2. FireWire does not operate at its rated speed. While a FireWire 400 drive has the potential to transfer data at up to 50 MB/second, it doesn’t. This is due to how the hard drive processes data internally using a FireWire bridge chip.
3. The more FireWire devices you add to your system, especially when you have more than five, the slower the data transfer rate is.
4. The more data you store on a hard drive the slower it goes. A drive is fastest when it is empty. When a drive is totally full, it neither plays back, nor records. I recommend keeping about 20% free space on your drive – though bar fights have broken out over what is the “best” proportion of free space to maintain.

While individual drives can vary, I’ve found that connection speeds generally group into the following speeds.

Data Transfer Rates By Hard Drive Connection Type

Connection	Data Transfer Speed
USB 1.0 or 2.0	10 – 15 MB / second
FireWire 400	20 – 25 MB / second
FireWire 800	40 – 50 MB / second
iSCSI (Ethernet)	75 – 95 MB / second
eSATA (single drive)	75 – 100 MB / second
RAIDs	100 – 600+ MB / second

NOTE: By the way, if terms like RAID 0 or RAID 1 confuse you, read this short explanation of how RAIDS are defined.

With this as a background, let’s see how we can apply that to video editing.

VIDEO FORMAT DATA TRANSFER REQUIREMENTS

Here’s how you read the table below.

Format indicates the video format as indicated by Easy Setup.

Store One Hour indicates how much hard disk space it takes to store an hour of material in a particular format. This allows you to estimate what your total storage needs are based upon the amount of material shot. Remember, to add 20% for general storage overhead and free space.

Comp indicates how the codec compresses the video. GOP compression compresses pictures in groups, I-frame compression compresses individual images. GOP compression requires an additional conforming process prior to output, which increases the time it takes to output your sequence.

Dupe indicates those video formats that Final Cut duplicates the media from its source format (i.e. P2 or AVCHD) into its transcoded format (i.e. QuickTime or ProRes 422). As a general rule of thumb, double all storage requirements for these formats. While not totally accurate (for instance, if you don’t transfer the footage into Final Cut Pro, no duplication occurs), its as reasonable a way to estimate file storage size as anything else.

Transfer Rate indicates how much data, on average per second, is required by that format. (The speed is measured in megabytes per second – MB/s.)

Keep in mind that a real-time dissolve doubles each of these transfer rates because during the dissolve two streams of video are playing. A four-camera multiclip increases each rate by four times because there are four streams of video playing.

Here’s the key point: Assuming your hard drives are healthy, a dropped frame error means that you are trying to read or write more data than your hard drive will support. This table helps you to match the data rate of your video format with the speed of your hard drive, indicated in the table above.

Note: HD-CAM and HD-CAM SR create massively big files and the exact file sizes vary widely by frame rate and image size. In all cases, these formats should be edited using a RAID with a data transfer rate in excess of 200 MB / second. Faster is absolutely better.

Data Transfer Rate by Video Format

Video Format	Store 1 Hour	Dupe	Comp	Transfer Rate
SD
DV NTSC / PAL	13 GB	No	I-frame	3.75 MB/second
DVCPRO-50	27 GB	No	I-frame	7.5 MB/second
Uncompressed 8-bit (Beta SP)	72 GB	No	I-frame	20.2 MB/second
Uncompressed 10-bit (DigiBeta)	96 GB	No	I-frame	26.7 MB/second
ProRes 422 (NTSC or PAL)	19.5 GB	No	I-frame	5.25 MB/second
ProRes 422 HQ	28.1 GB	No	I-frame	7.8 MB/second
HD
HDV (25 mbps) 60i	13 GB	No	GOP	3.75 MB/second
AVCHD (varies) *	Up to 10.8 GB	Yes	I-frame	1.5 – 3.0 MB/second
AVC-Intra (Panasonic) *	Up to 10.8 GB	Yes	I-frame	1.5 – 3.0 MB/second
AVCCAM (Sony & Pana.) *	Up to 10.8 GB	Yes	I-frame	1.5 – 3.0 MB/second
XDCAM HD (50 mbps)	28 GB	Yes	GOP	7.75 MB/second
XDCAM HD (35 mbps)	19 GB	Yes	GOP	5.2 MB/second
XDCAM EX	19 GB	Yes	GOP	5.2 MB/second
DVCPROHD	54 GB	Yes	I-frame	15 MB/second
ProRes 422 (Proxy)*	20 GB	No	I-frame	5.6 MB/second
ProRes 422 (LT)*	46 GB	No	I-frame	12.75 MB/second
ProRes 422*	66 GB	No	I-frame	18.1 MB/second
ProRes 422 (HQ)*	99 GB	No	I-frame	27.5 MB/second
ProRes 4444 (no alpha)*	148 GB	No	I-frrame	41.25 MB/second
R3D	137 GB	No	I-frame	28 or 38 MB/second
HDCAM 720p 60 fps	396 GB	No	I-frame	110 MB/second
HDCAM 1080 60 fps	834 GB	No	I-frame	237 MB/second

NOTE: All AVC video formats are converted (transcoded) to ProRes422 during ingest into Final Cut Pro. So, while the AVC source video uses GOP compression, ProRes uses I-frame compression. Also, when the AVC footage is duplicated, the ProRes data rate and file size applies to the converted video.

 

ProRes is a variable bit-rate encoder, so file sizes will vary depending upon format, image size, and frame rate. The HD specs for ProRes are based on shooting 1080i/60 and taken from Apple’s ProRes white paper. File sizes decrease for 720p files, or slower frame rates.

I found that understanding the relationship between hard disk speeds and video formats greatly simplified storage budgeting and solving dropped frame problems.

UPDATE – Nov. 17, 2008

Rich Roddman sent me the following note:

You have R3D files listed as not needing to create Pro Res files as they are imported into Final Cut. That is not quite true, while you can just drop the proxy file in the timeline to view or edit, if you use the Log & Transfer tool to import the R3D files they will transcode them into Pro Res (HQ). The 4K files are converted to 2K in in width, 3K import at 3K and 2K stays at 2K. Much like P2, it is not a fast process even with an 8 core tower.

Rob Naim adds:

There is a great widget from a UK company called Digital Heaven than can be helpful when working out video storage issues that you’ve discussed in your current newsletter.
Have a look at it here: http://www.digital-heaven.co.uk/videospace/

 

It doesn’t do data rates but does do storage needed for length of video and its free.

Larry replies: Thanks for both your thoughts.