Seeing this topic is still getting attention after two weeks, I'll be more specific about what is wrong with the question you posted. Here is my thought process as I read your question:
I have a circuit that generates random noise which I've measured with a 10 bit ADC. The following is the sample distribution:
OK so far. The graph looks plausible for this kind of noise source.
You will notice that it's a log normal distribution
No, I wouldn't. Maybe some time in the dark past a particular probability distribution was given the name "log normal" in some class, but if so, that has been long forgotten. The OP has now alienated all those who haven't just taken the class where this was mentioned, are working in the theoretical parts of EE, or happen to have bumped into this recently enough to remember what the implications of "log normal" are. Surely some people will know, probably the more academic types, but that will be quite a small subset of users here.
Since he didn't define it, link to a reference (probably wouldn't have followed it anyway since it's his job to put the salient points here), or explained whatever part of it being log normal is relevant here, he just lost most of the audience.
The writing style also gives a hint of the "research paper" attitude. That's being technically right and rigorous, but difficult to follow unless one already knows what is being talked about. Mabye it isn't, but it often seems this is done deliberatly: "I'm smarter than you are because my math it more impenetrable than yours."
I'll just let this slide for now because it isn't clear how important this distinction is, and the relevant aspects might become clear from context later.
You can see that approximately 14,000 samples were at a value of 1023 or above
No, I can't. For reference, here is the diagram being referred to:
I can see that the frequency of occurance at 1000 is well below 5000, probably less than 1000 from eyeballing the graph. I do see a box with a top that could be at 14,000, but that box is not mentioned anywhere, and in any case is below 1023, not above it. What the ...?
Clearly I'm not getting something here. Probably best not to touch this question because I'll just look stupid. If this was written more accessibly and less arrogantly, maybe we could work it out, but not with this question.
they are effectively a source of random entropy
I can see the randomness, and the plot showing the relative probabilities of any particular value occurring, but what's the relevance of "entropy"? From context, this seems like some specifically defined way of measuring something about the randomness of the stream of samples, but like "log normal", whatever definition I might have understood from that has been lost in time.
This guy either has no clue about his audience, or is deliberately being academically arrogant to feel superior.
Shannon entropy rate per sample
It's definitely been too long since this was covered in a class (if it ever was) for me to remember what that means. Again, it seems like some kind of randomness measure.
The next paragraph explains that the point of the question is to maximize the randomness of the resulting samples by deciding what section of the overall probability distribution histogram to pick off. Why couldn't he just say that in the first place? He mentions scaling (which is really what changing the reference voltage does), but has ignored shifting the window around (adding a offset). That seems a obvious thing to do, but why isn't it considered? It is easy enough to do electrically. I would engage and ask about this if I didn't think I'd just get more Shannon entropy log blah blah babble in return.
Also, it's not really clear why not scaling and offsetting to pick off a narrow area around the hump of the curve isn't good enough. That will cause the A/D to clip to 0 and 1023 more often. These are really "no data" and therefore don't count as useful readings? What about the usual approach of using only the low bit of the A/D? If the window was chosen carefully so that the probability of clipping at 0 and at 1023 were equal, then the low bit should still be random and usable from the clipped samples too.
However, engaging with this guy feels like it's going to be painful, so I'll go find a bunch of simpler questions to answer in the time it would take going back and forth with this ivory tower weenie.
Also, if this guy has a equation for the probability distribution (I'm assuming that's what "log normal" refers to), and knows how to evaluate this Shannon entropy stuff he's on about, why doesn't he just write the equation of Shannon entropy as a function of the scaling factor and maximize it? How is that not obvious, especially considering the high-falutin theoretical stance he's taken? Something doesn't add up. It feels like penny-wise and pound-foolish. But, I don't feel like spending a lot of time and then just looking foolish myself as a result, so screw this.