Sony A7iii - Raw files - compressed VS uncompressed - there is no lossless compression available

[PRSS]

Hello all

I see that this topic has been discussed here before. But still, my query is quite a bit different.

I am new to Sony Alpha camera, switching over from Nikon systems. Just got the new system 4 days back. This is about RAW files - compressed and non compressed.
I have already googled on this topic. of compressed Vs non compressed RAW files with Sony while other manufacturers offer lossless compression files - which I too had always had this as default on my Nikons.

They say "some situations" demand non compressed files. I am yet to understand what exactly was meant by "some situations". Is it with too much of contrast in a scene, or a scene which contained lot of fine details. Fearing loss of quality, I have set my Sony A7iii as non compressed, even though the file size is double that of compressed. Not having lossless compression came to me as a surprise !

I would like to have the opinion of members in this forum on this. Should I just leave my setting as non compressed and put up with larger files requiring larger storage devises, since cost of storage devices are not that expensive now a days ? OR, would I get high quality pictures with compressed files itself ? It is not just a question of storage alone. Post processing a 45 MB file Vs a 24 MB file - that would make a difference ! Especially with not such a powerful computer. I am doing post processing in Luminar 4 software which is quite a slow software (IMHO) and on a Lenovo laptop Core-i7 2.5 Ghz, 24 GB RAM, Nvidia Geforce GTX 1650 graphics card with 4 GB VRAM. OS is Windows 10. I would not opt for Adobe Photoshop & Lightroom subscription even if that would make difference in speed of processing. Because I am not a professional photographer, but a passionate hobbyist only. And also because the number of photos I take in a month is quite low - does not justify an Adobe subscription.

I wish to continue with Luminar 4. OR with Affinity Photo, which I have too.

So, I am appealing in this forum to advise me - should I leave the setting as non compressed OR when to actually set it to non compressed and leave it as compressed for most of the pics. I have uploaded some of the photos taken by me which is typical of my type of photography. Thanking you in advance.
Best regards
PRSS

 

[Reciprocum]

Definitely leave it as uncompressed. With my former A7iii I even avoided shooting bursts in Hi+ mode as they would result in 12-bit deep RAW files, lacking the DR of the 14-bit deep RAWs obtained when shooting Hi (8fps) or slower. Doing the same now on A1.
Why spend money on good lenses and then throw away some of its potencial with a lossy setting ? Unless you shooting a sports event, taking thousands of images, lossy compressed makes no sense at all.
For archival I don't even keep the RAWs, just the 100% quality JPEG exports usually downsampled to 3240px on the short side.
For the rare 0.1% (top 50) portfolio images I might keep the RAWs for 1 year, then just a full res TIFF after a final re-edit.

 

[PRSS]

Thank you very much Alfonso and Tim ! Pixel peeping on a 27 inch monitor became my habit ! Yes, I should lose that habit ! And NO ! I don't print my photos. Online sharing only. And posting in photography forums for critique. Since I am yet to run my own tests, for now, Ill leave it as uncompressed. Thanking you once again.
Regards
PRSS

 

[pointreyes]

I remember the first time I experienced posterization artifacts on an image I took. I blamed my ability to hold the camera / lens steady. And then once I realized that was not it, I was concerned something was wrong with my camera and/or lens. And finally, these two links cleared up what happened.

Sony Raw Compression Artifacts: Real Examples

One of the curses of shooting with Sony cameras is their RAW file format. Currently they only offer two methods for saving RAW files: (1) a RAW file with lossy compression, and (2) an uncompressed RAW file. Did you notice what’s missing? Sony doesn’t offer a lossless compressed RAW file unlike...

stephenbayphotography.com

RawDigger: detecting posterization in SONY cRAW/ARW2 files

Lossy compression of raw data is currently the only option available in Sony cameras of series NEX, SLT, RX, ILCE, ILCA, and the recent DSLR-A. The first part of this article is showing how to detect artifacts caused by this compression. We will be discussing the technical details of this...

www.rawdigger.com

For months I was mad at Sony for this - took only uncompressed raw images for months and would compress the raw files into dng to save storage space.

For my cameras that don't have lossless compressed raw as an option, I just take the risk with the knowledge of knowing that for certain type of images with extreme contrast that I should switch to uncompressed raw.

 

[pointreyes]

Just understand that any kind of compression will result in loss of detail. Even 'lossless' compressed will have that effect. Again though, we come back to how much it matters? Most of the guys at my local shop shoot lossless compressed to save space and file size.

From 1 to 3: Uncompressed > Lossless Compressed > Lossy Compressed.

Which is one of the reasons why I went with an A7iii for infrared photography. Not only do I have to deal with diffraction being visible sooner than with visible light, but I am not willing to go with any form of compression out of risk of how a visible light algorithm will know what to do with IR. Working with 48 Meg is doable.

 

[PRSS]

I remember the first time I experienced posterization artifacts on an image I took. I blamed my ability to hold the camera / lens steady. And then once I realized that was not it, I was concerned something was wrong with my camera and/or lens. And finally, these two links cleared up what happened.

Sony Raw Compression Artifacts: Real Examples

One of the curses of shooting with Sony cameras is their RAW file format. Currently they only offer two methods for saving RAW files: (1) a RAW file with lossy compression, and (2) an uncompressed RAW file. Did you notice what’s missing? Sony doesn’t offer a lossless compressed RAW file unlike...

stephenbayphotography.com

RawDigger: detecting posterization in SONY cRAW/ARW2 files

Lossy compression of raw data is currently the only option available in Sony cameras of series NEX, SLT, RX, ILCE, ILCA, and the recent DSLR-A. The first part of this article is showing how to detect artifacts caused by this compression. We will be discussing the technical details of this...

www.rawdigger.com

For months I was mad at Sony for this - took only uncompressed raw images for months and would compress the raw files into dng to save storage space.

For my cameras that don't have lossless compressed raw as an option, I just take the risk with the knowledge of knowing that for certain type of images with extreme contrast that I should switch to uncompressed raw.

Thank you very much ! Sorry for my late response. I have left my setting as uncompressed RAW. I accidentally found out that converting to DNG made the file size into half (with original file extraction turned off). I gave this a try because I have Adobe Lightroom 6 (version 6.14 the last) perpetual license and it would not recognize Sony A7iii RAW files. Converted to DNG, I could now post process in Lightroom 6 as well as in Luminar or Affinity photo. Looks like I'd make this my routine process now, converting to DNG. Till now, I haven't seen any difference in post processing a converted DNG file which is about 23 to 25 MB Vs a 42 to 45 MB A7iii uncompressed RAW file. To me, it seems both yield the same results. Thanking you once again.
With best regards
PRSS

 

[AlphaWorld]

Just understand that any kind of compression will result in loss of detail. Even 'lossless' compressed will have that effect. Again though, we come back to how much it matters? Most of the guys at my local shop shoot lossless compressed to save space and file size.

From 1 to 3: Uncompressed > Lossless Compressed > Lossy Compressed.

Sorry, but I must dispute this. Lossless compressed does not result in loss of detail. It is lossless.

It is like saying, for example, that using Zip to compress a file results in loss of detail. Zip uses lossless compression, too - if you zip a file, then unzip it, you get back a file which is byte-for-byte identical to the original. That is what lossless means. If you zipped a file, unzipped it, and you got back something different, you would have cause for loud complaints.

Sadly, certain manufacturers (Nikon, for example) invented the term”visually lossless”, meaning “you can’t see any difference” - that confused matters, but that term is not applicable to Sony’s lossless compressed files (it could be used to describe Sony’s lossy compression, for the most part).

What you could argue you might lose with lossless compression is time - it takes time to compress the file (although that time is likely saved by having to wait less time to write the smaller file to the card), and it takes time to expand the file when processing the RAW file (although that may also be ameliorated by saving time reading the smaller file from disk).

 

[pointreyes]

Once it's uploaded and opened, do you end up with a compressed version and an uncompressed version for editing like a Zip file?

Just checked and no it does not do that. For years, I have even edited and saved files in a zip file; you don't have to unzip a file to use since the memory can do it. Using a third-party tool like 7-zip allows me to do this.

However, thanks to me doing a stupid thing I have discovered that Sony did not implement TIFF standards in their lossless compressed files. I forgot to change the time on my A1 to PST from PDT and I have a piece of utility software that will change the time in the file (not create a XMP file). That software had to un-compress each file to make the change. I contacted the vendor, and this is the response I got:

Our development department investigated the phenomenon that Sony ARWs files get doubled in size with Time Fix.

In the Sony, you can save uncompressed or lossless compressed RAWs (ARWs). Technically, however, under the hood these are TIFFs, though Sony’s lossless compressed TIFFs do not adhere to the TIFF standard.

However, the manufacturer of Exif tool (what we use to edit the metadata) does not (and will not) allow TIFFs to be saved non-standard. So a lossless compressed TIFF after using Time Fix is an uncompressed TIFF under the guise of ARW, which is then about twice as big. But then there is a standard compliant TIFF in the ARW

Unfortunately, there is nothing we can do about that.

 

[AlphaWorld]

What about the A1 and S, M, L? From what I can tell the S and M both drop pixels, and only the L is actually lossless. Is that the case?

In any event, I see no reason to shoot lossless compressed. The cameras write fast enough that it's irrelevant. Once it's uploaded and opened, do you end up with a compressed version and an uncompressed version for editing like a Zip file? Wouldn't that just increase storage needs or require me to deal with deleting the compressed version afterwards? Still not seeing the advantage here. The only time I can see the need would be if your card is getting full and you have to save space. I shot at the track yesterday, thousands of clicks. I had a 128G card in the #1 slot of both cameras, and a 64G in the #2 slot for backup. There's still space left on the first card in both cameras. I doubt they'd be filled even with a higher resolution sensor, but even if they were the second card would've barely been accessed.

I have never used the smaller fake "RAW" files on any camera that supported them - they (obviously) reduce the number of pixels, but more importantly, they are not true RAW files. I was only talking about what is now called L lossless compressed, but was simply called lossless compressed for the first year and a bit :-D

I've been using lossless compressed files for 15+ years now, starting with Canon (Canon did not support uncompressed - all their RAW files were lossless compressed), followed by Nikon (who do support uncompressed and lossy compressed as well). It's a well-established technology that Sony is only now starting to adopt - my theory is that they had to wait for the new CPU to make the lossless compression fast enough (their lossy compression was supported in hardware in the older CPUs), but it's only a theory.

No, the processing software always does the expansion of the compression in-flight - there is no uncompressed file left on the disk. That's true of all the compressed formats (lossless and lossy) that I have used, from all of the manufacturers. I'm not saying there isn't someone, somewhere, saving an uncompressed file, but I have not seen it.

I am mostly using Photoshop, and when I save an image as a PSD after RAW processing it's enormous - an uncropped A1 image produces a uncompressed PSD of almost 300MB (I thought there was an option for compressed PSD files? Must remember to look for it).

Why do I use lossless compressed? Yes, it's partly because I get more images on the card, and more images on the disk. I'm using an A1 with the massive temptation of large bursts - the nominal capacity of a 160GB CFeA card is over 2000 lossless compressed images, and I have on occasion filled both CFeA cards in the camera (once I came close to filling all four 160GB cards I had). That is uncommon, though. It's more that I can download the cards to the computer faster with the smaller files - when I get home I am always impatient to start working on the files :-D

Anyway, my goal was not to persuade you to join the ranks of the lossless compression users, but merely to correct the impression that lossless compressed files lose some image fidelity - that is just not true (for the full-size ones, I mean).