I do enjoy your physics posts more than your (also excellent) documentaries on the pervading insanities of our time. So I look forward to reading more of them. It’s a very long time since I personally did any relativistic and/or quantum stuff for myself, and these essays of yours reignite my lifelong interest and spark all those fading memories.
Also to mention my 3 all time fave real thinkers (Isaac, Albert and Dick) in a single essay. That’s a 5 ⭐️ thanks from me 😀
Thanks Drew - it's really hard to write this stuff in a way that is both accurate and intelligible. I'm not really satisfied with this latest piece - I don't think I drew out the 'mystery' there properly. And I also forgot a crucial element (as WW pointed out).
Feynman deserves mention in the same breath as Newton or Einstein who are (slightly) better known to the public. I never met him but have spoken to colleagues who did and the stories really aren't exaggerated - he was something else!
lost me on introducing a second beam splitter that clearly behaves differently than the first (2 inputs instead of 1 to start with) but without explanation of what that second one actually does.
Yes - my fault. In attempting to strip it to the "bare bones" I forgot about this important point. You're spot on - the beamsplitters each have 2 input arms and two output arms.
Thanks for pointing this out! I've edited the main article a bit and included another figure that hopefully makes it more clear.
I am flattered you credit me with "pointing out" something. I was genuinely unable to make the intellectual leap from 1 input to 2 without help. Having done that with your revision, I still am stuck in your reader Rickard's position of the whole thing being a bit too mysterious to have any sense of having grasped what any photon actually does. I was educated in engineering (as opposed to science) and learned that once someone had constructed a mathematical model that "worked" just use it. I had no real hope of knowing (in any fundamental sense) where the edges of the models' applicability lay. But if you fell off the edge of where the model worked, the real world would slap you soon enough.
"Scholars have estimated that only 1/6th of his [Newton's] output was actually scientific. I wonder what he could have achieved if he’d just stuck to what he was good at?" Another question is, given the admitted quality of his more comprehensible to us accomplishments, who are we -- in our limited hindsight and infectious case of 'scientific' confirmation bias -- to judge all of what he was good at?
Well it was intended to be a (vaguely) humorous comment - but your point is taken. I suppose, Newton being Newton, he was bloody amazing at everything 'intellectual' that he did - it's just that the science stuff has had by far and away the greatest impact on the world.
Human minds don't necessarily work in the straight lines favored by 'science stuff', though. Maybe Newton's forays into other fields and occupations benefited his productivity in science and mathematics. His scientific productivity may have, in fact, been at least partially dependent on those other influences.
So maybe I'm fick as two planks, but /why/ is it so strange that it is either, both and neither as circumstances dictate?
If all particles and the parts that make up the particles and so on were perfectly measurable and perfectly predictable, would existence exist at all then? Wouldn't it just be something singular, without dimensions?
Maybe that's what happened. One moment there's one thing without any 1,2,3,4+ dimensional representation, yet still existing in potentia. In somway, somehow, it is observed or observes itself as existing and thus all its potentialities collapses into a set form with an unknowable uncertainty at its core, so as to be ableto keep existing as total certainty would again set it in the same stasis it started out as.
(To quote Pratchett, this is hard to talk about using language initially developed to tell the other apes where the ripe fruits are.)
If matter and energy are interchangeable, maybe the photon simply is the ultimate end of one side of the scale, and there's some kind of energy-less matter (or state) on the other end?
I'm not very satisfied with the article - I don't think I did a good job of explaining why there is such a mystery surrounding these kinds of experiments. So you're struggling with 2 things - one being that this stuff is mysterious, the other being my crappy exposition!
Nah, your explanation is fine far as I can tell from my "seven letters, "baffles physicists"-level of knowledge of quantum.
The Why I'm circling is more to the tune of "What practical effect would it have if we understood it?" - base, utilitarian practical "Why?", sort of.
And my darker side ponders "what kind of weapon could that be turned into?". No doubt an artifact of a school-system that tried to teach "nuclear anything equals Hiroshima" as a lot of our teachers were alphabet-communists, DDR-trained or Maoists in that day.
It's probably something I picked up in the humanities, where the quest for the Holy Why is akin to an eternal golden braid: "Why do kids bully each other?", "Why does [category] [behaviour] [other category]?" and so on, it's so codified you can actually use brackets and fill-in-the-blanks-constructed essays and reports nowadays in sociology, psychology and so on (the Sokal Affair was just the tip of the Everest so to speak, and was widely scoffed at at the time).
To me, with my "feckless hooligan"-youth and lots of practical job-experience (it's amazing what theories of society and everything you can think of when mopping a subway platform at 03:30AM), the "why" only ever matters /after/, if ever, a problem's been handled - but with quantum it seems the "why" is the problem, then?
I mean, let's take bullying as a comparison. What's the problem? According to most teachers and people working in some kind of social capacity or another, the problem isn't the bullying but the bully, so to solve the situation you must understand the "why" of the bully.
According to me and I guess most people outside the social sciences/careers, the problem is the bullying as such, the act itself. So the solution is to forbid and stop the act, which logically then may mean suspending or even expelling the bully from school, or a restraining order plus a stint in Borstal/youth labour camp. Again and again until the lesson sticks or the bully does something warranting life in prison or short sharp shock.
I ramble on like this because I think I forgot to be clearer on what I meant, maybe. The heat (+22C) is making me thick like Sgt. Detritus in Klatch.
Here's a conundrum then, seeing as photons are weird: what is it that happens in photosynthesis then? Does the plant somehow "collapse thewaveform" of the photons?
I always just bought the simple version, "plants turns sunlight into nutrients using CO2 and water", and didn't think more of it.
If the photon is a wave (or partially a wave, or whatever) - when emitted from the Single Photon Source, the wave amplitude (or other parameter) is at a certain value but can that value be controlled or is it random? Apparently random, assuming that parameter is what influences the direction of the photon.
Once it hits beam splitter 1, perhaps this parameter (like a wave amplitude) gets adjusted to a certain value range (because of some property of beam splitters), which allows beam splitter 2 - with some adjusting - to direct it to SPD A or B.
These posts remind me of how intellectually lazy I can be. (When my head starts to hurt, I give up.) But that’s okay. The world needs all kinds and I am the kind that prefers to bake the bread and make the bed and help locate missing sets of keys and pairs of glasses.
If physicists had to make bread it would take months as they calibrated the mixing, the quantities, the oven, and worried about whether quantum effects were important.
There's a reason us weirdos get shunted off to places like universities. We can't (usually) do too much harm there, and are safely kept out of contact with the rest of the public 😆
In a universe of Trillions ( what we have seen so far) of Galaxies, I am certain that civilizations also in the trillions have solved all of this. Trillions of Einsteins and who knows how much smarter species there are. I like to think about a sentient program living in a holographic world made of pixels. The smallest thing it can possibly experience is the pixel. It has no idea what the pixel is , what the energy is that powers it and how it the sentient program came into being. ( because it’s creators are running an experiment on it) I think this is where we are now, not in a simulation ( I don’t buy that) but in a world who’s very nature we are so far clueless to fathom, and try as we may we will never figure it out until we recognize there are greater dimensions to this physical reality ( higher dimensions) that science as it exists today can not begin to fathom.
There is no wave-particle duality. My friend Eric Reiter has solved this problem. We’re just waiting for the science establishment to catch up.
Eric has been doing beam splitter experiments with a gamma source for over 20 years and has shown that you can often get a result where both detectors ‘click’. He’s also spent those 20 years trying to find errors in his set up and the results just keep coming. I’ll post a link to his work at the end of this comment.
Long story short: we never see photons. They don’t exist. They are just an idea. We invented photons due to our faulty inference about the photoelectric effect and due to Thompson’s earlier invention of the electron (which I think might also be an error).
When the light source in a quantum experiment emits, it does so in a quantised fashion. It releases quanta of light and, as you said, you can get it to do this one at a time. One quantum, not one photon.
So why do we think there are photons?
Because events in the detector are point-like, localised phenomena. And this is where we make the faulty inference.
The detector is made of individual atoms. Detection events consist of an individual atom changing its quantum state. It is dogma (from our mistaken ideas about the photoelectric effect) that this change requires that a whole quantum of energy be absorbed at one time by the detector atom.
But Eric has shown that a single quantum emitted from the source can produce two detection events. So either the law of conservation of energy is broken or something else happens. Of course, it’s something else.
Energy absorption by the atoms in the detector does not depend on the energy of the incoming electromagnetic radiation but on its frequency. The atoms load up due to resonance and they do so gradually. Planck posited this mechanism in what’s called his ‘Second Theory’, also known as loading theory.
When the electromagnetic radiation from the source arrives at the detector, it impinges on atoms which are already pre-loaded with some of the energy required to make the jump to the next quantum state. In fact, they make this jump not because they can but because they must. When an atom has absorbed enough energy to cross a threshold, it must make the change and produce a ‘click’ or a spot in the detector.
So while wave mechanics can give us the probability of a detection event occurring at a given point in the detector, it is the state of the individual atoms which determines which one clicks.
The light does not travel from the source to the detector as packets of energy. It propagates as a wave and only a fraction of a quantum of energy is required to produce a detection event.
Eric goes into this at far more length. His work is unknowingly corroborated by a Youtuber called Huygens Optics in an experiment he describes at length in a video titled “How big is a visible photon?”. He manages to create interference effects in his experimental setup which quantum dogma says should not happen at the energy levels he uses. In other words, interference should require multiple photons hitting the detector simultaneously but he has tuned his experimental setup so that the rate of photons hitting the detector is less than one at a time. He accepts a BS explanation from an expert at one of the big universities but has actually demonstrated, like Eric does, that there are no photons.
I'd largely agree with the statement "there is no wave-particle duality". This is really an artefact of trying to 'force' a picture (particle or wave or both) onto things.
What you say about detectors is also (partially) true. The photoelectric effect is usually given as "evidence" that photons exist, but as Scully and Lamb demonstrated (in the 60's I think - or maybe the 50's), it can be explained by using what's known as a 'semi-classical' approach in which the atoms in the detector/material are treated quantum mechanically and the EM field is treated classically (i.e. a wave phenomenon).
It was really the theoretical work of Roy Glauber that gave us the thing to look for if we wanted to demonstrate that the EM field needs to be treated quantum mechanically. It was in 1981 that Len Mandel did the experiments to confirm that the EM field could NOT be explained as a classical object. The thing you need to look for (at the time) are more subtle coherence properties - 2nd order coherences (or correlations). There's a purely **quantum** phenomenon known as photon anti-bunching which cannot be explained with a classical theory of the EM field - and it was this that Mandel experimentally confirmed.
The use of the word 'photon' is really used as a shorthand by physicists - more technically a 'photon' is an occupation of a quantised field mode. It does not imply either 'wave' or 'particle' - these are properties that depend on the experimental set up and which quantum 'operator' one is trying to measure.
Other quantum effects in light do exist - you can do some wacky stuff with polarisation, or you can construct 'squeezed' states of light, you can also play around with the quantum angular momentum of light to produce some really interesting 'helical' states which can be used to separate molecules by their chirality, for example.
Wave-particle duality is really just a very special subset of a much larger thing we might term "complementarity" - which comes about because the operators used to describe properties in QM do not always commute - and this means (amongst other things) that certain properties cannot be simultaneously measured with arbitrary accuracy - which most people might know as the phenomenon described by the Heisenberg uncertainty principle - which is really just wave-particle duality in another context.
Fundamentally it's all due to the algebraic structure of quantum mechanics - and I can't explain why we **have** to use this kind of algebra to describe nature. All I can say is that if you don't, you get your predictions wrong.
I’m not familiar with all of the work you cite as evidence of the quantum behaviour of light so I can’t argue against it. However, I will just put a thought out there that has been demonstrated by Eric’s work.
Atoms are demonstrably quantised and there are clear classical mechanisms to do with confined wave systems, resonance and harmonics (e.g. the “particle in a box”) which can account for this. However, there is nothing inherently quantised or granular about fields. The granularity appears when we solve the wave equation for a confined system.
What Eric’s critique of the double slit experiment does is point to two mistakes we might make habitually: inventing objects to account for events, and mistaking the behaviour of the instruments, which are made of quantised atoms, for the behaviour of light.
I'll have to look at Eric's stuff in detail later, but it sounds like he's kind of re-inventing the wheel. The question of whether the EM field **MUST** be treated quantum mechanically (in some situations) was a serious question for a long time. The brilliant work of Scully and Lamb showed that we really had to be careful - particularly when talking about things based on the photoelectric effect.
It took a long time before we finally had definitive proof that, yes, treating the EM field as a classical object was not sufficient to describe everything we can observe in nature. We had to wait 76 years from Einstein's first intuition that light is quantum in nature for the definitive experimental proof in 1981.
Since then, many more explicitly quantum features of light have been discovered and worked on.
The quantum nature of light doesn't disappear when we consider things that aren't "in a box" - it's not the 'confinement' that is responsible for the quantum nature. It's easier to use a confined system in analysis (as you say, you get discrete modes) but the quantum effects persist even when taking the theory to the full continuum - where there is a continuum of modes.
What Eric has done IMO is falsify the photon idea. So much about QM is built on this assumption: wave-particle duality, ‘collapse of the wave function’ and all of the philosophies related to it, non-local versus local and especially ‘entanglement’.
What you're describing here are attempts to come to terms with what the mathematics of QM describes. QM is NOT 'built' on any of these assumptions. Technically speaking the structure of QM is a non-commuting C* algebra. A **commuting** C* algebra is isomorphic to the phase space of classical mechanics.
It's just a matter of fact that a commuting C* algebra cannot explain all of the features we observe in nature.
There are various approaches ("interpretations") to QM - the so-called 'collapse of the wave function' is not a necessary component - and many (most perhaps) physicists reject such a notion. It arose early on from the Copenhagen interpretation of QM. I use it a lot when doing calculations, not because I think anything is really 'collapsing', but it's a way of thinking that facilitates an easier solution path (for me) in a lot of cases. I could get exactly the same answers using a different interpretation without having to consider 'collapse', but I (personally) don't find those approaches as easy to use.
Entanglement, for example, arises from the necessity of having to use this non-commuting algebra. Entanglement is just the quantum feature of interference playing out in a Hilbert space that is the tensor product of two sub-spaces. We don't get this interference in a theory based on a **commuting** C* algebra.
I have a vague recollection of someone writing, and I am obviously paraphrasing, ... If you can thoroughly describe Quantum Mechanics, then you certainly don't understand it... which seems to embrace quanta quite well.
Any recommendations of books to read to have a better understanding of QM?
One of my favourite expositions - which really does a magnificent job of describing the mysteries of the double-slit experiment is Johnjoe McFadden's "Quantum Evolution".
It's not just purely about explaining QM, though. The author's thesis is that QM has played a role in evolution - which is a more contentious claim. It's a very good read, though.
I read all the general/ popular/ non technical books on Quantum Theory and relativity I could find in my local public library as a youngster. One of the few on QM that I still remember 30-40 years on as an inspiring introduction is “In search of Schroedinger’s Cat” by John Gribbin. It’s available for cheap these days on kindle or paperback.
Really cool stuff Doc.
I do enjoy your physics posts more than your (also excellent) documentaries on the pervading insanities of our time. So I look forward to reading more of them. It’s a very long time since I personally did any relativistic and/or quantum stuff for myself, and these essays of yours reignite my lifelong interest and spark all those fading memories.
Also to mention my 3 all time fave real thinkers (Isaac, Albert and Dick) in a single essay. That’s a 5 ⭐️ thanks from me 😀
Thanks Drew - it's really hard to write this stuff in a way that is both accurate and intelligible. I'm not really satisfied with this latest piece - I don't think I drew out the 'mystery' there properly. And I also forgot a crucial element (as WW pointed out).
Feynman deserves mention in the same breath as Newton or Einstein who are (slightly) better known to the public. I never met him but have spoken to colleagues who did and the stories really aren't exaggerated - he was something else!
lost me on introducing a second beam splitter that clearly behaves differently than the first (2 inputs instead of 1 to start with) but without explanation of what that second one actually does.
Yes - my fault. In attempting to strip it to the "bare bones" I forgot about this important point. You're spot on - the beamsplitters each have 2 input arms and two output arms.
Thanks for pointing this out! I've edited the main article a bit and included another figure that hopefully makes it more clear.
I am flattered you credit me with "pointing out" something. I was genuinely unable to make the intellectual leap from 1 input to 2 without help. Having done that with your revision, I still am stuck in your reader Rickard's position of the whole thing being a bit too mysterious to have any sense of having grasped what any photon actually does. I was educated in engineering (as opposed to science) and learned that once someone had constructed a mathematical model that "worked" just use it. I had no real hope of knowing (in any fundamental sense) where the edges of the models' applicability lay. But if you fell off the edge of where the model worked, the real world would slap you soon enough.
"Scholars have estimated that only 1/6th of his [Newton's] output was actually scientific. I wonder what he could have achieved if he’d just stuck to what he was good at?" Another question is, given the admitted quality of his more comprehensible to us accomplishments, who are we -- in our limited hindsight and infectious case of 'scientific' confirmation bias -- to judge all of what he was good at?
Well it was intended to be a (vaguely) humorous comment - but your point is taken. I suppose, Newton being Newton, he was bloody amazing at everything 'intellectual' that he did - it's just that the science stuff has had by far and away the greatest impact on the world.
Human minds don't necessarily work in the straight lines favored by 'science stuff', though. Maybe Newton's forays into other fields and occupations benefited his productivity in science and mathematics. His scientific productivity may have, in fact, been at least partially dependent on those other influences.
So maybe I'm fick as two planks, but /why/ is it so strange that it is either, both and neither as circumstances dictate?
If all particles and the parts that make up the particles and so on were perfectly measurable and perfectly predictable, would existence exist at all then? Wouldn't it just be something singular, without dimensions?
Maybe that's what happened. One moment there's one thing without any 1,2,3,4+ dimensional representation, yet still existing in potentia. In somway, somehow, it is observed or observes itself as existing and thus all its potentialities collapses into a set form with an unknowable uncertainty at its core, so as to be ableto keep existing as total certainty would again set it in the same stasis it started out as.
(To quote Pratchett, this is hard to talk about using language initially developed to tell the other apes where the ripe fruits are.)
If matter and energy are interchangeable, maybe the photon simply is the ultimate end of one side of the scale, and there's some kind of energy-less matter (or state) on the other end?
"My 'ead 'urts"
Sorry Rikard.
I'm not very satisfied with the article - I don't think I did a good job of explaining why there is such a mystery surrounding these kinds of experiments. So you're struggling with 2 things - one being that this stuff is mysterious, the other being my crappy exposition!
Nah, your explanation is fine far as I can tell from my "seven letters, "baffles physicists"-level of knowledge of quantum.
The Why I'm circling is more to the tune of "What practical effect would it have if we understood it?" - base, utilitarian practical "Why?", sort of.
And my darker side ponders "what kind of weapon could that be turned into?". No doubt an artifact of a school-system that tried to teach "nuclear anything equals Hiroshima" as a lot of our teachers were alphabet-communists, DDR-trained or Maoists in that day.
It's probably something I picked up in the humanities, where the quest for the Holy Why is akin to an eternal golden braid: "Why do kids bully each other?", "Why does [category] [behaviour] [other category]?" and so on, it's so codified you can actually use brackets and fill-in-the-blanks-constructed essays and reports nowadays in sociology, psychology and so on (the Sokal Affair was just the tip of the Everest so to speak, and was widely scoffed at at the time).
To me, with my "feckless hooligan"-youth and lots of practical job-experience (it's amazing what theories of society and everything you can think of when mopping a subway platform at 03:30AM), the "why" only ever matters /after/, if ever, a problem's been handled - but with quantum it seems the "why" is the problem, then?
I mean, let's take bullying as a comparison. What's the problem? According to most teachers and people working in some kind of social capacity or another, the problem isn't the bullying but the bully, so to solve the situation you must understand the "why" of the bully.
According to me and I guess most people outside the social sciences/careers, the problem is the bullying as such, the act itself. So the solution is to forbid and stop the act, which logically then may mean suspending or even expelling the bully from school, or a restraining order plus a stint in Borstal/youth labour camp. Again and again until the lesson sticks or the bully does something warranting life in prison or short sharp shock.
I ramble on like this because I think I forgot to be clearer on what I meant, maybe. The heat (+22C) is making me thick like Sgt. Detritus in Klatch.
Here's a conundrum then, seeing as photons are weird: what is it that happens in photosynthesis then? Does the plant somehow "collapse thewaveform" of the photons?
I always just bought the simple version, "plants turns sunlight into nutrients using CO2 and water", and didn't think more of it.
If the photon is a wave (or partially a wave, or whatever) - when emitted from the Single Photon Source, the wave amplitude (or other parameter) is at a certain value but can that value be controlled or is it random? Apparently random, assuming that parameter is what influences the direction of the photon.
Once it hits beam splitter 1, perhaps this parameter (like a wave amplitude) gets adjusted to a certain value range (because of some property of beam splitters), which allows beam splitter 2 - with some adjusting - to direct it to SPD A or B.
These posts remind me of how intellectually lazy I can be. (When my head starts to hurt, I give up.) But that’s okay. The world needs all kinds and I am the kind that prefers to bake the bread and make the bed and help locate missing sets of keys and pairs of glasses.
If physicists had to make bread it would take months as they calibrated the mixing, the quantities, the oven, and worried about whether quantum effects were important.
There's a reason us weirdos get shunted off to places like universities. We can't (usually) do too much harm there, and are safely kept out of contact with the rest of the public 😆
In a universe of Trillions ( what we have seen so far) of Galaxies, I am certain that civilizations also in the trillions have solved all of this. Trillions of Einsteins and who knows how much smarter species there are. I like to think about a sentient program living in a holographic world made of pixels. The smallest thing it can possibly experience is the pixel. It has no idea what the pixel is , what the energy is that powers it and how it the sentient program came into being. ( because it’s creators are running an experiment on it) I think this is where we are now, not in a simulation ( I don’t buy that) but in a world who’s very nature we are so far clueless to fathom, and try as we may we will never figure it out until we recognize there are greater dimensions to this physical reality ( higher dimensions) that science as it exists today can not begin to fathom.
There is no wave-particle duality. My friend Eric Reiter has solved this problem. We’re just waiting for the science establishment to catch up.
Eric has been doing beam splitter experiments with a gamma source for over 20 years and has shown that you can often get a result where both detectors ‘click’. He’s also spent those 20 years trying to find errors in his set up and the results just keep coming. I’ll post a link to his work at the end of this comment.
Long story short: we never see photons. They don’t exist. They are just an idea. We invented photons due to our faulty inference about the photoelectric effect and due to Thompson’s earlier invention of the electron (which I think might also be an error).
When the light source in a quantum experiment emits, it does so in a quantised fashion. It releases quanta of light and, as you said, you can get it to do this one at a time. One quantum, not one photon.
So why do we think there are photons?
Because events in the detector are point-like, localised phenomena. And this is where we make the faulty inference.
The detector is made of individual atoms. Detection events consist of an individual atom changing its quantum state. It is dogma (from our mistaken ideas about the photoelectric effect) that this change requires that a whole quantum of energy be absorbed at one time by the detector atom.
But Eric has shown that a single quantum emitted from the source can produce two detection events. So either the law of conservation of energy is broken or something else happens. Of course, it’s something else.
Energy absorption by the atoms in the detector does not depend on the energy of the incoming electromagnetic radiation but on its frequency. The atoms load up due to resonance and they do so gradually. Planck posited this mechanism in what’s called his ‘Second Theory’, also known as loading theory.
When the electromagnetic radiation from the source arrives at the detector, it impinges on atoms which are already pre-loaded with some of the energy required to make the jump to the next quantum state. In fact, they make this jump not because they can but because they must. When an atom has absorbed enough energy to cross a threshold, it must make the change and produce a ‘click’ or a spot in the detector.
So while wave mechanics can give us the probability of a detection event occurring at a given point in the detector, it is the state of the individual atoms which determines which one clicks.
The light does not travel from the source to the detector as packets of energy. It propagates as a wave and only a fraction of a quantum of energy is required to produce a detection event.
Eric goes into this at far more length. His work is unknowingly corroborated by a Youtuber called Huygens Optics in an experiment he describes at length in a video titled “How big is a visible photon?”. He manages to create interference effects in his experimental setup which quantum dogma says should not happen at the energy levels he uses. In other words, interference should require multiple photons hitting the detector simultaneously but he has tuned his experimental setup so that the rate of photons hitting the detector is less than one at a time. He accepts a BS explanation from an expert at one of the big universities but has actually demonstrated, like Eric does, that there are no photons.
Here’s Eric’s website:
https://www.unquantum.net/
And here’s the video:
https://www.youtube.com/watch?v=SDtAh9IwG-I
I'd largely agree with the statement "there is no wave-particle duality". This is really an artefact of trying to 'force' a picture (particle or wave or both) onto things.
What you say about detectors is also (partially) true. The photoelectric effect is usually given as "evidence" that photons exist, but as Scully and Lamb demonstrated (in the 60's I think - or maybe the 50's), it can be explained by using what's known as a 'semi-classical' approach in which the atoms in the detector/material are treated quantum mechanically and the EM field is treated classically (i.e. a wave phenomenon).
It was really the theoretical work of Roy Glauber that gave us the thing to look for if we wanted to demonstrate that the EM field needs to be treated quantum mechanically. It was in 1981 that Len Mandel did the experiments to confirm that the EM field could NOT be explained as a classical object. The thing you need to look for (at the time) are more subtle coherence properties - 2nd order coherences (or correlations). There's a purely **quantum** phenomenon known as photon anti-bunching which cannot be explained with a classical theory of the EM field - and it was this that Mandel experimentally confirmed.
The use of the word 'photon' is really used as a shorthand by physicists - more technically a 'photon' is an occupation of a quantised field mode. It does not imply either 'wave' or 'particle' - these are properties that depend on the experimental set up and which quantum 'operator' one is trying to measure.
Other quantum effects in light do exist - you can do some wacky stuff with polarisation, or you can construct 'squeezed' states of light, you can also play around with the quantum angular momentum of light to produce some really interesting 'helical' states which can be used to separate molecules by their chirality, for example.
Wave-particle duality is really just a very special subset of a much larger thing we might term "complementarity" - which comes about because the operators used to describe properties in QM do not always commute - and this means (amongst other things) that certain properties cannot be simultaneously measured with arbitrary accuracy - which most people might know as the phenomenon described by the Heisenberg uncertainty principle - which is really just wave-particle duality in another context.
Fundamentally it's all due to the algebraic structure of quantum mechanics - and I can't explain why we **have** to use this kind of algebra to describe nature. All I can say is that if you don't, you get your predictions wrong.
I’m not familiar with all of the work you cite as evidence of the quantum behaviour of light so I can’t argue against it. However, I will just put a thought out there that has been demonstrated by Eric’s work.
Atoms are demonstrably quantised and there are clear classical mechanisms to do with confined wave systems, resonance and harmonics (e.g. the “particle in a box”) which can account for this. However, there is nothing inherently quantised or granular about fields. The granularity appears when we solve the wave equation for a confined system.
What Eric’s critique of the double slit experiment does is point to two mistakes we might make habitually: inventing objects to account for events, and mistaking the behaviour of the instruments, which are made of quantised atoms, for the behaviour of light.
I'll have to look at Eric's stuff in detail later, but it sounds like he's kind of re-inventing the wheel. The question of whether the EM field **MUST** be treated quantum mechanically (in some situations) was a serious question for a long time. The brilliant work of Scully and Lamb showed that we really had to be careful - particularly when talking about things based on the photoelectric effect.
It took a long time before we finally had definitive proof that, yes, treating the EM field as a classical object was not sufficient to describe everything we can observe in nature. We had to wait 76 years from Einstein's first intuition that light is quantum in nature for the definitive experimental proof in 1981.
Since then, many more explicitly quantum features of light have been discovered and worked on.
The quantum nature of light doesn't disappear when we consider things that aren't "in a box" - it's not the 'confinement' that is responsible for the quantum nature. It's easier to use a confined system in analysis (as you say, you get discrete modes) but the quantum effects persist even when taking the theory to the full continuum - where there is a continuum of modes.
What Eric has done IMO is falsify the photon idea. So much about QM is built on this assumption: wave-particle duality, ‘collapse of the wave function’ and all of the philosophies related to it, non-local versus local and especially ‘entanglement’.
What you're describing here are attempts to come to terms with what the mathematics of QM describes. QM is NOT 'built' on any of these assumptions. Technically speaking the structure of QM is a non-commuting C* algebra. A **commuting** C* algebra is isomorphic to the phase space of classical mechanics.
It's just a matter of fact that a commuting C* algebra cannot explain all of the features we observe in nature.
There are various approaches ("interpretations") to QM - the so-called 'collapse of the wave function' is not a necessary component - and many (most perhaps) physicists reject such a notion. It arose early on from the Copenhagen interpretation of QM. I use it a lot when doing calculations, not because I think anything is really 'collapsing', but it's a way of thinking that facilitates an easier solution path (for me) in a lot of cases. I could get exactly the same answers using a different interpretation without having to consider 'collapse', but I (personally) don't find those approaches as easy to use.
Entanglement, for example, arises from the necessity of having to use this non-commuting algebra. Entanglement is just the quantum feature of interference playing out in a Hilbert space that is the tensor product of two sub-spaces. We don't get this interference in a theory based on a **commuting** C* algebra.
I have a vague recollection of someone writing, and I am obviously paraphrasing, ... If you can thoroughly describe Quantum Mechanics, then you certainly don't understand it... which seems to embrace quanta quite well.
Any recommendations of books to read to have a better understanding of QM?
One of my favourite expositions - which really does a magnificent job of describing the mysteries of the double-slit experiment is Johnjoe McFadden's "Quantum Evolution".
It's not just purely about explaining QM, though. The author's thesis is that QM has played a role in evolution - which is a more contentious claim. It's a very good read, though.
I read all the general/ popular/ non technical books on Quantum Theory and relativity I could find in my local public library as a youngster. One of the few on QM that I still remember 30-40 years on as an inspiring introduction is “In search of Schroedinger’s Cat” by John Gribbin. It’s available for cheap these days on kindle or paperback.